2024-08-19T09:44:58.919 INFO:root:teuthology version: 1.2.0 2024-08-19T09:44:58.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T09:44:59.034 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301 branch: main description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7859301' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request mon_bind_msgr2: false sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1858 sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b sleep_before_teardown: 0 subset: 29813/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgkZ07bNVkZstG3mqtkpYuRHGwod/ZAn+vKc4p9iLxbPj3RkCZBFsXIjN0pqtGAreUZmw7yBJOBuMftUT2W05E= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNm3VXhyZiJlzkEtoz19U+Qx82t001n0DX4IMYjA6/bVI/QoQTNashFB7cBRhG/JcFPe/1YZtCgoC0UUbJzLBUE= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_20:00:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-19T09:44:59.035 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa; will attempt to use it 2024-08-19T09:44:59.035 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks 2024-08-19T09:44:59.036 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-19T09:44:59.037 INFO:teuthology.task.internal:Checking packages... 2024-08-19T09:44:59.059 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '797669399b32d2915e64eb6d4b1257ae3c50f80b' 2024-08-19T09:44:59.060 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T09:44:59.060 INFO:teuthology.packaging:ref: None 2024-08-19T09:44:59.060 INFO:teuthology.packaging:tag: None 2024-08-19T09:44:59.060 INFO:teuthology.packaging:branch: main 2024-08-19T09:44:59.060 INFO:teuthology.packaging:sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T09:44:59.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-08-19T09:44:59.290 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5847.g79766939 2024-08-19T09:44:59.292 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-19T09:44:59.299 INFO:teuthology.task.internal:no buildpackages task found 2024-08-19T09:44:59.299 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-19T09:44:59.303 INFO:teuthology.task.internal:Saving configuration 2024-08-19T09:44:59.315 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-19T09:44:59.322 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-19T09:44:59.352 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 09:40:13.754831', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgkZ07bNVkZstG3mqtkpYuRHGwod/ZAn+vKc4p9iLxbPj3RkCZBFsXIjN0pqtGAreUZmw7yBJOBuMftUT2W05E='} 2024-08-19T09:44:59.381 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 09:40:13.753393', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNm3VXhyZiJlzkEtoz19U+Qx82t001n0DX4IMYjA6/bVI/QoQTNashFB7cBRhG/JcFPe/1YZtCgoC0UUbJzLBUE='} 2024-08-19T09:44:59.381 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-19T09:44:59.388 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-08-19T09:44:59.388 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-08-19T09:44:59.388 INFO:teuthology.run_tasks:Running task console_log... 2024-08-19T09:44:59.459 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feaa9763ac0>, signals=[15]) 2024-08-19T09:44:59.459 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-19T09:44:59.465 INFO:teuthology.task.internal:Opening connections... 2024-08-19T09:44:59.465 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T09:44:59.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T09:44:59.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T09:44:59.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T09:44:59.658 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-19T09:44:59.665 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-08-19T09:44:59.683 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-08-19T09:44:59.683 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-08-19T09:44:59.741 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-08-19T09:44:59.742 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T09:44:59.742 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-08-19T09:44:59.742 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T09:44:59.742 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T09:44:59.742 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T09:44:59.742 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-08-19T09:44:59.777 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-08-19T09:44:59.796 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-08-19T09:44:59.796 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-08-19T09:44:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-08-19T09:44:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-08-19T09:44:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-08-19T09:44:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-08-19T09:44:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-08-19T09:44:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T09:44:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T09:44:59.855 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-08-19T09:44:59.885 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-19T09:44:59.894 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-19T09:44:59.931 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-19T09:44:59.931 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-08-19T09:44:59.934 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-08-19T09:44:59.952 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-19T09:44:59.958 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-19T09:44:59.958 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-08-19T09:44:59.991 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-08-19T09:45:00.018 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-19T09:45:00.083 INFO:teuthology.run_tasks:Running task kernel... 2024-08-19T09:45:00.099 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-19T09:45:00.099 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-19T09:45:00.100 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T09:45:00.100 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T09:45:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T09:45:00.117 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-08-19T09:45:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T09:45:00.118 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-08-19T09:45:00.174 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-480.el9.x86_64 2024-08-19T09:45:00.174 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-480.el9.x86_64 2024-08-19T09:45:00.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-08-19T09:45:00.177 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-480.el9.x86_64 2024-08-19T09:45:00.177 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.14.0-480.el9.x86_64 2024-08-19T09:45:00.177 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2024-08-19T09:45:02.540 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T09:45:05.665 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-08-19T09:45:05.773 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.2 MB 00:03 2024-08-19T09:45:07.941 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-08-19T09:45:11.250 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.6 MB 00:01 2024-08-19T09:45:12.792 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 18 kB 00:00 2024-08-19T09:45:13.529 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 36 MB/s | 22 MB 00:00 2024-08-19T09:45:13.549 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.6 MB 00:01 2024-08-19T09:45:15.158 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 18 kB 00:00 2024-08-19T09:45:15.995 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 31 MB/s | 22 MB 00:00 2024-08-19T09:45:19.202 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-19T09:45:21.242 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T09:45:21.242 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T09:45:21.322 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:45:21.323 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:45:21.323 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:45:21.342 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-08-19T09:45:21.419 DEBUG:teuthology.orchestra.run.smithi001:> echo no | sudo yum reinstall kernel || true 2024-08-19T09:45:21.949 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:02 ago on Mon 19 Aug 2024 09:45:19 AM UTC. 2024-08-19T09:45:22.088 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:45:22.099 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout:Reinstalling: 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 1.5 M 2024-08-19T09:45:22.100 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 0 2024-08-19T09:45:22.101 INFO:teuthology.orchestra.run.smithi001.stderr:Operation aborted. 2024-08-19T09:45:22.164 INFO:teuthology.orchestra.run.smithi001.stdout:Is this ok [y/N]: 2024-08-19T09:45:22.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum reinstall -y kernel || true 2024-08-19T09:45:22.693 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 09:45:19 AM UTC. 2024-08-19T09:45:22.820 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:45:22.821 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:45:22.821 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-08-19T09:45:22.821 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:45:22.821 INFO:teuthology.orchestra.run.smithi001.stdout:Reinstalling: 2024-08-19T09:45:22.821 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 1.5 M 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 0 2024-08-19T09:45:22.822 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:45:23.257 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T09:45:23.257 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T09:45:23.270 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-480.el9.x86_64.rpm 3.8 MB/s | 1.5 MB 00:00 2024-08-19T09:45:23.271 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:45:23.271 INFO:teuthology.orchestra.run.smithi001.stdout:Total 3.4 MB/s | 1.5 MB 00:00 2024-08-19T09:45:23.308 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:45:23.332 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:45:23.333 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:45:23.333 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:45:23.442 DEBUG:teuthology.orchestra.run.smithi082:> echo no | sudo yum reinstall kernel || true 2024-08-19T09:45:23.560 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:45:23.560 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:45:23.586 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:45:23.586 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:45:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:45:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T09:45:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T09:45:23.959 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:02 ago on Mon 19 Aug 2024 09:45:21 AM UTC. 2024-08-19T09:45:24.117 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:45:24.127 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout:Reinstalling: 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:45:24.128 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 1.5 M 2024-08-19T09:45:24.129 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 0 2024-08-19T09:45:24.129 INFO:teuthology.orchestra.run.smithi082.stderr:Operation aborted. 2024-08-19T09:45:24.189 INFO:teuthology.orchestra.run.smithi082.stdout:Is this ok [y/N]: 2024-08-19T09:45:24.190 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum reinstall -y kernel || true 2024-08-19T09:45:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T09:45:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T09:45:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 09:45:21 AM UTC. 2024-08-19T09:45:24.832 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:45:24.833 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:45:24.833 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-08-19T09:45:24.833 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout:Reinstalling: 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 1.5 M 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 0 2024-08-19T09:45:24.834 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:45:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T09:45:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:45:25.309 INFO:teuthology.orchestra.run.smithi001.stdout:Reinstalled: 2024-08-19T09:45:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T09:45:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:45:25.309 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:45:25.486 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T09:45:25.577 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-480.el9.x86_64.rpm 3.1 MB/s | 1.5 MB 00:00 2024-08-19T09:45:25.577 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:45:25.577 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.0 MB/s | 1.5 MB 00:00 2024-08-19T09:45:25.581 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T09:45:25.582 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T09:45:25.582 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T09:45:25.583 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-19T09:45:25.583 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T09:45:25.610 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-08-19T09:45:25.612 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:45:25.634 DEBUG:teuthology.parallel:result is None 2024-08-19T09:45:25.839 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:45:25.839 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:45:25.864 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:45:25.864 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:45:26.005 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:45:26.096 INFO:teuthology.orchestra.run.smithi082.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T09:45:26.096 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T09:45:26.857 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T09:45:26.858 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T09:45:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T09:45:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:45:27.619 INFO:teuthology.orchestra.run.smithi082.stdout:Reinstalled: 2024-08-19T09:45:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T09:45:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:45:27.619 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:45:27.815 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T09:45:27.871 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T09:45:27.871 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T09:45:27.871 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T09:45:27.871 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-08-19T09:45:27.871 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T09:45:27.900 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-08-19T09:45:27.926 DEBUG:teuthology.parallel:result is None 2024-08-19T09:45:27.926 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-19T09:45:27.933 INFO:teuthology.task.internal:Creating test directory... 2024-08-19T09:45:27.934 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T09:45:27.936 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T09:45:27.985 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-19T09:45:27.992 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-19T09:45:28.295 INFO:teuthology.task.internal:Creating archive directory... 2024-08-19T09:45:28.296 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T09:45:28.299 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T09:45:28.333 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-19T09:45:28.340 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-19T09:45:28.341 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T09:45:28.364 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T09:45:28.404 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T09:45:28.415 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T09:45:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T09:45:28.427 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T09:45:28.429 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-19T09:45:28.436 INFO:teuthology.task.internal:Configuring sudo... 2024-08-19T09:45:28.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T09:45:28.439 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T09:45:28.483 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-19T09:45:28.492 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-19T09:45:28.492 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T09:45:28.507 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T09:45:28.538 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T09:45:28.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T09:45:28.653 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T09:45:28.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T09:45:28.778 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:45:28.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T09:45:28.845 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T09:45:28.896 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T09:45:28.944 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T09:45:29.035 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T09:45:29.068 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:45:29.068 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T09:45:29.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-08-19T09:45:29.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-08-19T09:45:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T09:45:29.216 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T09:45:29.688 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-19T09:45:29.696 INFO:teuthology.task.internal:Starting timer... 2024-08-19T09:45:29.696 INFO:teuthology.run_tasks:Running task pcp... 2024-08-19T09:45:29.706 INFO:teuthology.run_tasks:Running task selinux... 2024-08-19T09:45:29.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-08-19T09:45:29.800 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-08-19T09:45:29.802 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-08-19T09:45:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-08-19T09:45:29.858 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-19T09:45:29.859 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-08-19T09:45:29.890 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-08-19T09:45:29.890 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-08-19T09:45:29.922 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-08-19T09:45:29.922 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive'} 2024-08-19T09:45:29.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T09:45:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T09:45:29.958 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T09:45:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T09:45:29.988 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-19T09:45:29.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-08-19T09:45:30.034 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-08-19T09:45:30.059 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-19T09:45:30.069 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-19T09:45:30.072 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-19T09:45:30.213 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-19T09:45:30.227 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-19T09:45:30.228 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-19T09:47:14.806 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-08-19T09:47:14.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-08-19T09:47:14.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T09:47:14.897 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-08-19T09:47:14.978 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-08-19T09:47:14.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-08-19T09:47:14.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T09:47:15.061 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-08-19T09:47:15.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-08-19T09:47:15.146 INFO:teuthology.run_tasks:Running task clock... 2024-08-19T09:47:15.157 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-19T09:47:15.157 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T09:47:15.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T09:47:15.161 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T09:47:15.161 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T09:47:15.187 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T09:47:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T09:47:15.216 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T09:47:15.225 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-08-19T09:47:15.231 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T09:47:15.236 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-08-19T09:47:15.250 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T09:47:15.255 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-08-19T09:47:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T09:47:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-08-19T09:47:15.280 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T09:47:15.294 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T09:47:15.317 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-08-19T09:47:15.320 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T09:47:15.320 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-08-19T09:47:15.320 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.320 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.320 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.320 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.353 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-08-19T09:47:15.356 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T09:47:15.356 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-08-19T09:47:15.356 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.357 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.357 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.357 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T09:47:15.358 INFO:teuthology.run_tasks:Running task install... 2024-08-19T09:47:15.365 DEBUG:teuthology.task.install:project ceph 2024-08-19T09:47:15.365 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}} 2024-08-19T09:47:15.366 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'} 2024-08-19T09:47:15.366 INFO:teuthology.task.install:Using flavor: default 2024-08-19T09:47:15.371 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-19T09:47:15.371 INFO:teuthology.task.install:extra packages: [] 2024-08-19T09:47:15.372 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b', 'tag': None, 'wait_for_package': False} 2024-08-19T09:47:15.372 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T09:47:15.375 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b', 'tag': None, 'wait_for_package': False} 2024-08-19T09:47:15.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T09:47:15.532 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/ 2024-08-19T09:47:15.532 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5847.g79766939 2024-08-19T09:47:15.543 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/ 2024-08-19T09:47:15.543 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5847.g79766939 2024-08-19T09:47:15.658 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T09:47:15.658 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:47:15.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T09:47:15.661 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T09:47:15.661 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:47:15.661 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T09:47:15.687 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T09:47:15.687 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/797669399b32d2915e64eb6d4b1257ae3c50f80b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T09:47:15.688 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T09:47:15.688 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/797669399b32d2915e64eb6d4b1257ae3c50f80b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T09:47:15.753 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T09:47:15.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T09:47:15.831 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T09:47:15.832 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T09:47:15.895 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-08-19T09:47:15.896 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-08-19T09:47:15.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-08-19T09:47:15.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-08-19T09:47:16.119 INFO:teuthology.orchestra.run.smithi082.stdout:45 files removed 2024-08-19T09:47:16.127 INFO:teuthology.orchestra.run.smithi001.stdout:45 files removed 2024-08-19T09:47:16.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2024-08-19T09:47:16.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-08-19T09:47:16.685 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 314 kB/s | 84 kB 00:00 2024-08-19T09:47:16.700 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 314 kB/s | 84 kB 00:00 2024-08-19T09:47:16.960 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 104 kB/s | 24 kB 00:00 2024-08-19T09:47:16.992 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-08-19T09:47:17.228 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-08-19T09:47:17.267 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-08-19T09:47:17.961 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T09:47:17.984 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T09:47:21.862 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 20 MB 00:02 2024-08-19T09:47:22.385 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 20 MB 00:03 2024-08-19T09:47:26.062 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.6 MB 00:00 2024-08-19T09:47:27.345 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.6 MB 00:01 2024-08-19T09:47:27.513 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 18 kB 00:00 2024-08-19T09:47:28.283 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 34 MB/s | 22 MB 00:00 2024-08-19T09:47:29.562 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 18 kB 00:00 2024-08-19T09:47:30.299 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 37 MB/s | 22 MB 00:00 2024-08-19T09:47:33.681 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-08-19T09:47:35.855 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-19T09:47:35.974 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 11 M 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.5 M 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T09:47:35.976 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 22 M 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-5847.g79766939.el9 ceph 25 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 755 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T09:47:35.977 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 508 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:19.0.0-5847.g79766939.el9 ceph 45 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 138 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-5847.g79766939.el9 ceph 188 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-5847.g79766939.el9 ceph 371 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 356 k 2024-08-19T09:47:35.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 116 k 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout:Install 30 Packages 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2024-08-19T09:47:35.979 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:35.980 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 62 M 2024-08-19T09:47:35.980 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:47:37.015 INFO:teuthology.orchestra.run.smithi082.stdout:(1/32): ceph-base-19.0.0-5847.g79766939.el9.x86 8.4 MB/s | 5.5 MB 00:00 2024-08-19T09:47:37.073 INFO:teuthology.orchestra.run.smithi082.stdout:(2/32): ceph-selinux-19.0.0-5847.g79766939.el9. 432 kB/s | 25 kB 00:00 2024-08-19T09:47:37.190 INFO:teuthology.orchestra.run.smithi082.stdout:(3/32): libcephfs2-19.0.0-5847.g79766939.el9.x8 6.3 MB/s | 755 kB 00:00 2024-08-19T09:47:37.432 INFO:teuthology.orchestra.run.smithi082.stdout:(4/32): ceph-common-19.0.0-5847.g79766939.el9.x 20 MB/s | 22 MB 00:01 2024-08-19T09:47:37.457 INFO:teuthology.orchestra.run.smithi082.stdout:(5/32): libradosstriper1-19.0.0-5847.g79766939. 1.9 MB/s | 508 kB 00:00 2024-08-19T09:47:37.574 INFO:teuthology.orchestra.run.smithi082.stdout:(6/32): ceph-radosgw-19.0.0-5847.g79766939.el9. 9.0 MB/s | 11 MB 00:01 2024-08-19T09:47:37.616 INFO:teuthology.orchestra.run.smithi082.stdout:(7/32): python3-ceph-argparse-19.0.0-5847.g7976 285 kB/s | 45 kB 00:00 2024-08-19T09:47:37.658 INFO:teuthology.orchestra.run.smithi082.stdout:(8/32): python3-ceph-common-19.0.0-5847.g797669 1.6 MB/s | 138 kB 00:00 2024-08-19T09:47:37.683 INFO:teuthology.orchestra.run.smithi082.stdout:(9/32): python3-cephfs-19.0.0-5847.g79766939.el 2.8 MB/s | 188 kB 00:00 2024-08-19T09:47:37.708 INFO:teuthology.orchestra.run.smithi082.stdout:(10/32): python3-rados-19.0.0-5847.g79766939.el 7.2 MB/s | 371 kB 00:00 2024-08-19T09:47:37.742 INFO:teuthology.orchestra.run.smithi082.stdout:(11/32): python3-rbd-19.0.0-5847.g79766939.el9. 5.9 MB/s | 356 kB 00:00 2024-08-19T09:47:37.767 INFO:teuthology.orchestra.run.smithi082.stdout:(12/32): python3-rgw-19.0.0-5847.g79766939.el9. 1.9 MB/s | 116 kB 00:00 2024-08-19T09:47:37.859 INFO:teuthology.orchestra.run.smithi082.stdout:(13/32): librgw2-19.0.0-5847.g79766939.el9.x86_ 13 MB/s | 5.5 MB 00:00 2024-08-19T09:47:37.909 INFO:teuthology.orchestra.run.smithi082.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 641 kB/s | 107 kB 00:00 2024-08-19T09:47:37.968 INFO:teuthology.orchestra.run.smithi082.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-08-19T09:47:37.993 INFO:teuthology.orchestra.run.smithi082.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 711 kB/s | 160 kB 00:00 2024-08-19T09:47:38.043 INFO:teuthology.orchestra.run.smithi082.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-08-19T09:47:38.069 INFO:teuthology.orchestra.run.smithi082.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-08-19T09:47:38.094 INFO:teuthology.orchestra.run.smithi082.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-19T09:47:38.119 INFO:teuthology.orchestra.run.smithi082.stdout:(20/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-08-19T09:47:38.144 INFO:teuthology.orchestra.run.smithi082.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 477 kB/s | 36 kB 00:00 2024-08-19T09:47:38.151 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:47:38.153 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 11 M 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.5 M 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T09:47:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 22 M 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-5847.g79766939.el9 ceph 25 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 755 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 508 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T09:47:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.0.0-5847.g79766939.el9 ceph 45 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 138 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-5847.g79766939.el9 ceph 188 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-5847.g79766939.el9 ceph 371 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 356 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 116 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T09:47:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout:Install 30 Packages 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-08-19T09:47:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:47:38.158 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 62 M 2024-08-19T09:47:38.158 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:47:38.170 INFO:teuthology.orchestra.run.smithi082.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-08-19T09:47:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-08-19T09:47:38.246 INFO:teuthology.orchestra.run.smithi082.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-08-19T09:47:38.271 INFO:teuthology.orchestra.run.smithi082.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 300 kB/s | 26 kB 00:00 2024-08-19T09:47:38.296 INFO:teuthology.orchestra.run.smithi082.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-19T09:47:38.330 INFO:teuthology.orchestra.run.smithi082.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-19T09:47:38.372 INFO:teuthology.orchestra.run.smithi082.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-08-19T09:47:38.430 INFO:teuthology.orchestra.run.smithi082.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-19T09:47:38.505 INFO:teuthology.orchestra.run.smithi082.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-08-19T09:47:38.681 INFO:teuthology.orchestra.run.smithi082.stdout:(31/32): librados2-19.0.0-5847.g79766939.el9.x8 11 MB/s | 3.5 MB 00:00 2024-08-19T09:47:38.739 INFO:teuthology.orchestra.run.smithi082.stdout:(32/32): librbd1-19.0.0-5847.g79766939.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-08-19T09:47:38.744 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:47:38.744 INFO:teuthology.orchestra.run.smithi082.stdout:Total 22 MB/s | 62 MB 00:02 2024-08-19T09:47:38.848 INFO:teuthology.orchestra.run.smithi001.stdout:(1/32): ceph-base-19.0.0-5847.g79766939.el9.x86 11 MB/s | 5.5 MB 00:00 2024-08-19T09:47:38.849 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:47:38.883 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:47:38.883 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:47:38.898 INFO:teuthology.orchestra.run.smithi001.stdout:(2/32): ceph-selinux-19.0.0-5847.g79766939.el9. 504 kB/s | 25 kB 00:00 2024-08-19T09:47:38.974 INFO:teuthology.orchestra.run.smithi001.stdout:(3/32): libcephfs2-19.0.0-5847.g79766939.el9.x8 9.9 MB/s | 755 kB 00:00 2024-08-19T09:47:39.032 INFO:teuthology.orchestra.run.smithi001.stdout:(4/32): libradosstriper1-19.0.0-5847.g79766939. 8.5 MB/s | 508 kB 00:00 2024-08-19T09:47:39.357 INFO:teuthology.orchestra.run.smithi001.stdout:(5/32): librgw2-19.0.0-5847.g79766939.el9.x86_6 17 MB/s | 5.5 MB 00:00 2024-08-19T09:47:39.359 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:47:39.359 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:47:39.416 INFO:teuthology.orchestra.run.smithi001.stdout:(6/32): python3-ceph-argparse-19.0.0-5847.g7976 773 kB/s | 45 kB 00:00 2024-08-19T09:47:39.475 INFO:teuthology.orchestra.run.smithi001.stdout:(7/32): python3-ceph-common-19.0.0-5847.g797669 2.3 MB/s | 138 kB 00:00 2024-08-19T09:47:39.700 INFO:teuthology.orchestra.run.smithi001.stdout:(8/32): ceph-common-19.0.0-5847.g79766939.el9.x 16 MB/s | 22 MB 00:01 2024-08-19T09:47:39.725 INFO:teuthology.orchestra.run.smithi001.stdout:(9/32): python3-cephfs-19.0.0-5847.g79766939.el 753 kB/s | 188 kB 00:00 2024-08-19T09:47:39.759 INFO:teuthology.orchestra.run.smithi001.stdout:(10/32): python3-rados-19.0.0-5847.g79766939.el 6.2 MB/s | 371 kB 00:00 2024-08-19T09:47:39.784 INFO:teuthology.orchestra.run.smithi001.stdout:(11/32): python3-rbd-19.0.0-5847.g79766939.el9. 5.9 MB/s | 356 kB 00:00 2024-08-19T09:47:39.809 INFO:teuthology.orchestra.run.smithi001.stdout:(12/32): python3-rgw-19.0.0-5847.g79766939.el9. 2.3 MB/s | 116 kB 00:00 2024-08-19T09:47:39.926 INFO:teuthology.orchestra.run.smithi001.stdout:(13/32): ceph-radosgw-19.0.0-5847.g79766939.el9 6.9 MB/s | 11 MB 00:01 2024-08-19T09:47:40.076 INFO:teuthology.orchestra.run.smithi001.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-08-19T09:47:40.102 INFO:teuthology.orchestra.run.smithi001.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-08-19T09:47:40.127 INFO:teuthology.orchestra.run.smithi001.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 505 kB/s | 160 kB 00:00 2024-08-19T09:47:40.177 INFO:teuthology.orchestra.run.smithi001.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-08-19T09:47:40.219 INFO:teuthology.orchestra.run.smithi001.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-08-19T09:47:40.245 INFO:teuthology.orchestra.run.smithi001.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-08-19T09:47:40.261 INFO:teuthology.orchestra.run.smithi001.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-08-19T09:47:40.287 INFO:teuthology.orchestra.run.smithi001.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 533 kB/s | 36 kB 00:00 2024-08-19T09:47:40.312 INFO:teuthology.orchestra.run.smithi001.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 705 kB/s | 47 kB 00:00 2024-08-19T09:47:40.371 INFO:teuthology.orchestra.run.smithi001.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-08-19T09:47:40.404 INFO:teuthology.orchestra.run.smithi001.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-08-19T09:47:40.421 INFO:teuthology.orchestra.run.smithi001.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 276 kB/s | 26 kB 00:00 2024-08-19T09:47:40.447 INFO:teuthology.orchestra.run.smithi001.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-19T09:47:40.472 INFO:teuthology.orchestra.run.smithi001.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-08-19T09:47:40.531 INFO:teuthology.orchestra.run.smithi001.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-08-19T09:47:40.572 INFO:teuthology.orchestra.run.smithi001.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-19T09:47:40.648 INFO:teuthology.orchestra.run.smithi001.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-08-19T09:47:40.781 INFO:teuthology.orchestra.run.smithi001.stdout:(31/32): librados2-19.0.0-5847.g79766939.el9.x8 14 MB/s | 3.5 MB 00:00 2024-08-19T09:47:40.808 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:47:40.839 INFO:teuthology.orchestra.run.smithi001.stdout:(32/32): librbd1-19.0.0-5847.g79766939.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-08-19T09:47:40.844 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:47:40.845 INFO:teuthology.orchestra.run.smithi001.stdout:Total 23 MB/s | 62 MB 00:02 2024-08-19T09:47:40.889 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-19T09:47:40.942 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-19T09:47:40.946 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:47:40.968 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-19T09:47:40.986 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:47:40.986 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:47:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T09:47:41.014 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T09:47:41.304 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T09:47:41.355 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T09:47:41.485 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:47:41.485 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:47:41.737 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-19T09:47:41.806 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T09:47:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T09:47:41.997 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 8/34 2024-08-19T09:47:42.015 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T09:47:42.089 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T09:47:42.124 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-19T09:47:42.161 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 11/34 2024-08-19T09:47:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T09:47:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T09:47:42.315 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T09:47:42.349 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-19T09:47:42.460 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-19T09:47:42.491 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-19T09:47:42.532 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-19T09:47:42.557 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-19T09:47:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-19T09:47:42.627 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-19T09:47:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-19T09:47:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:47:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T09:47:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-19T09:47:43.093 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T09:47:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-19T09:47:43.120 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 23/34 2024-08-19T09:47:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-19T09:47:43.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-19T09:47:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T09:47:43.194 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T09:47:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T09:47:43.498 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T09:47:43.646 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-19T09:47:43.676 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T09:47:43.748 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T09:47:43.824 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 27/34 2024-08-19T09:47:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-19T09:47:43.853 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 28/34 2024-08-19T09:47:43.955 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T09:47:44.052 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T09:47:44.147 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 8/34 2024-08-19T09:47:44.182 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T09:47:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T09:47:44.282 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-19T09:47:44.328 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 11/34 2024-08-19T09:47:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T09:47:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T09:47:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T09:47:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-19T09:47:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-19T09:47:44.658 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-19T09:47:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-19T09:47:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-19T09:47:44.753 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-19T09:47:44.786 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-19T09:47:45.121 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-19T09:47:45.165 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T09:47:45.261 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T09:47:45.286 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 23/34 2024-08-19T09:47:45.322 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-19T09:47:45.833 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-19T09:47:45.867 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T09:47:45.940 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T09:47:46.019 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 27/34 2024-08-19T09:47:46.053 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 28/34 2024-08-19T09:47:46.131 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T09:47:46.216 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T09:47:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T09:47:46.802 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T09:47:46.882 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T09:47:46.882 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T09:47:46.883 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T09:47:46.883 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:46.912 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T09:47:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T09:47:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T09:47:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T09:47:49.155 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T09:47:49.249 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T09:47:49.250 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T09:47:49.250 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T09:47:49.250 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:47:49.281 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T09:47:56.411 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-08-19T09:47:56.412 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T09:47:57.454 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T09:47:57.454 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:47:57.454 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T09:47:57.454 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T09:47:57.454 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T09:47:57.454 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:57.456 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T09:47:57.548 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T09:47:57.548 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T09:47:58.763 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T09:47:58.763 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 1/34 2024-08-19T09:47:58.763 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2/34 2024-08-19T09:47:58.763 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 3/34 2024-08-19T09:47:58.763 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 4/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 5/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 6/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 8/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 9/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 10/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 11/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 13/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-19T09:47:58.764 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-19T09:47:58.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-19T09:47:58.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-19T09:47:58.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T09:47:58.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-19T09:47:58.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 33/34 2024-08-19T09:47:59.417 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T09:47:59.418 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-08-19T09:47:59.418 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-08-19T09:47:59.418 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-08-19T09:47:59.418 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-08-19T09:47:59.418 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-08-19T09:47:59.418 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-08-19T09:47:59.419 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-08-19T09:47:59.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:47:59.557 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.558 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T09:47:59.559 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T09:47:59.560 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:47:59.561 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:47:59.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2024-08-19T09:48:00.431 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:27 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T09:48:00.473 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T09:48:00.473 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:00.473 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T09:48:00.473 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T09:48:00.473 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T09:48:00.473 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:00.475 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T09:48:00.551 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T09:48:00.551 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T09:48:00.571 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:00.572 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:00.572 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:00.572 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:00.572 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:00.572 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-5847.g79766939.el9 ceph 50 M 2024-08-19T09:48:00.572 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-5847.g79766939.el9 ceph 163 k 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-08-19T09:48:00.573 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:00.574 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 51 M 2024-08-19T09:48:00.574 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 216 M 2024-08-19T09:48:00.574 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:01.041 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-08-19T09:48:01.079 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): libcephsqlite-19.0.0-5847.g79766939.el9. 653 kB/s | 163 kB 00:00 2024-08-19T09:48:01.108 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 948 kB/s | 64 kB 00:00 2024-08-19T09:48:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T09:48:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 1/34 2024-08-19T09:48:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2/34 2024-08-19T09:48:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 3/34 2024-08-19T09:48:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 4/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 5/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 6/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 8/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 9/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 10/34 2024-08-19T09:48:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 11/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 13/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-19T09:48:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-19T09:48:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-19T09:48:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T09:48:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-19T09:48:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 33/34 2024-08-19T09:48:02.452 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): ceph-test-19.0.0-5847.g79766939.el9.x86_ 31 MB/s | 50 MB 00:01 2024-08-19T09:48:02.454 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:02.454 INFO:teuthology.orchestra.run.smithi082.stdout:Total 27 MB/s | 51 MB 00:01 2024-08-19T09:48:02.481 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:02.502 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:02.502 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T09:48:02.784 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T09:48:02.785 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.786 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T09:48:02.786 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.786 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T09:48:02.786 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T09:48:02.786 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T09:48:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T09:48:02.788 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T09:48:02.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:02.788 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:02.882 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:02.882 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:03.038 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-08-19T09:48:03.409 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:03.485 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T09:48:03.525 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T09:48:03.554 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T09:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:28 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:03.766 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:03.767 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:03.767 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:48:03.767 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:03.767 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:03.767 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-5847.g79766939.el9 ceph 50 M 2024-08-19T09:48:03.768 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:48:03.768 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-5847.g79766939.el9 ceph 163 k 2024-08-19T09:48:03.768 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T09:48:03.768 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T09:48:03.768 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:03.768 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:03.769 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:03.769 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-08-19T09:48:03.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:03.770 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 51 M 2024-08-19T09:48:03.770 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 216 M 2024-08-19T09:48:03.770 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): libcephsqlite-19.0.0-5847.g79766939.el9. 741 kB/s | 163 kB 00:00 2024-08-19T09:48:04.598 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 180 kB/s | 64 kB 00:00 2024-08-19T09:48:04.640 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 494 kB/s | 304 kB 00:00 2024-08-19T09:48:05.654 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): ceph-test-19.0.0-5847.g79766939.el9.x86_ 31 MB/s | 50 MB 00:01 2024-08-19T09:48:05.656 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:05.656 INFO:teuthology.orchestra.run.smithi001.stdout:Total 27 MB/s | 51 MB 00:01 2024-08-19T09:48:05.684 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:05.705 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:05.705 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:06.101 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:06.102 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:06.675 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:06.764 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T09:48:06.812 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T09:48:06.839 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T09:48:08.274 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T09:48:08.349 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T09:48:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T09:48:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 1/4 2024-08-19T09:48:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2/4 2024-08-19T09:48:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T09:48:11.052 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:11.053 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:11.344 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2024-08-19T09:48:11.788 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T09:48:11.871 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T09:48:11.889 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:38 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:12.025 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:12.027 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:12.027 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-5847.g79766939.el9 ceph 6.6 k 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-5847.g79766939.el9 ceph 2.4 M 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-5847.g79766939.el9 ceph 1.5 M 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 284 k 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-5847.g79766939.el9 ceph 4.8 M 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-5847.g79766939.el9 ceph 17 M 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T09:48:12.028 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T09:48:12.029 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T09:48:12.030 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout:Install 37 Packages 2024-08-19T09:48:12.031 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:12.032 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 31 M 2024-08-19T09:48:12.032 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 109 M 2024-08-19T09:48:12.032 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:(1/37): ceph-19.0.0-5847.g79766939.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-08-19T09:48:12.761 INFO:teuthology.orchestra.run.smithi082.stdout:(2/37): ceph-mds-19.0.0-5847.g79766939.el9.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-08-19T09:48:12.803 INFO:teuthology.orchestra.run.smithi082.stdout:(3/37): ceph-mgr-19.0.0-5847.g79766939.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-08-19T09:48:12.895 INFO:teuthology.orchestra.run.smithi082.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5847.g7976 3.0 MB/s | 284 kB 00:00 2024-08-19T09:48:13.012 INFO:teuthology.orchestra.run.smithi082.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-08-19T09:48:13.087 INFO:teuthology.orchestra.run.smithi082.stdout:(6/37): ceph-mon-19.0.0-5847.g79766939.el9.x86_ 9.1 MB/s | 4.8 MB 00:00 2024-08-19T09:48:13.304 INFO:teuthology.orchestra.run.smithi082.stdout:(7/37): ceph-osd-19.0.0-5847.g79766939.el9.x86_ 32 MB/s | 17 MB 00:00 2024-08-19T09:48:13.329 INFO:teuthology.orchestra.run.smithi082.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 440 kB/s | 106 kB 00:00 2024-08-19T09:48:13.363 INFO:teuthology.orchestra.run.smithi082.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.5 MB/s | 1.2 MB 00:00 2024-08-19T09:48:13.388 INFO:teuthology.orchestra.run.smithi082.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-08-19T09:48:13.430 INFO:teuthology.orchestra.run.smithi082.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 135 kB 00:00 2024-08-19T09:48:13.455 INFO:teuthology.orchestra.run.smithi082.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-08-19T09:48:13.564 INFO:teuthology.orchestra.run.smithi082.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-08-19T09:48:13.589 INFO:teuthology.orchestra.run.smithi082.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 940 kB/s | 188 kB 00:00 2024-08-19T09:48:13.614 INFO:teuthology.orchestra.run.smithi082.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 935 kB/s | 172 kB 00:00 2024-08-19T09:48:13.631 INFO:teuthology.orchestra.run.smithi082.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-19T09:48:13.657 INFO:teuthology.orchestra.run.smithi082.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 991 kB/s | 42 kB 00:00 2024-08-19T09:48:13.682 INFO:teuthology.orchestra.run.smithi082.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 656 kB/s | 77 kB 00:00 2024-08-19T09:48:13.716 INFO:teuthology.orchestra.run.smithi082.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-08-19T09:48:13.741 INFO:teuthology.orchestra.run.smithi082.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 743 kB/s | 43 kB 00:00 2024-08-19T09:48:13.766 INFO:teuthology.orchestra.run.smithi082.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-08-19T09:48:13.783 INFO:teuthology.orchestra.run.smithi082.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 643 kB/s | 11 kB 00:00 2024-08-19T09:48:13.808 INFO:teuthology.orchestra.run.smithi082.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-08-19T09:48:13.825 INFO:teuthology.orchestra.run.smithi082.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-08-19T09:48:13.851 INFO:teuthology.orchestra.run.smithi082.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 551 kB/s | 23 kB 00:00 2024-08-19T09:48:13.876 INFO:teuthology.orchestra.run.smithi082.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-08-19T09:48:13.901 INFO:teuthology.orchestra.run.smithi082.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-08-19T09:48:13.926 INFO:teuthology.orchestra.run.smithi082.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-08-19T09:48:13.952 INFO:teuthology.orchestra.run.smithi082.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-08-19T09:48:13.969 INFO:teuthology.orchestra.run.smithi082.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-08-19T09:48:13.994 INFO:teuthology.orchestra.run.smithi082.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-08-19T09:48:14.019 INFO:teuthology.orchestra.run.smithi082.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-19T09:48:14.045 INFO:teuthology.orchestra.run.smithi082.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-19T09:48:14.070 INFO:teuthology.orchestra.run.smithi082.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-08-19T09:48:14.098 INFO:teuthology.orchestra.run.smithi082.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-08-19T09:48:14.120 INFO:teuthology.orchestra.run.smithi082.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-19T09:48:14.146 INFO:teuthology.orchestra.run.smithi082.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-08-19T09:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout:Total 14 MB/s | 31 MB 00:02 2024-08-19T09:48:14.207 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T09:48:14.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 1/4 2024-08-19T09:48:14.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2/4 2024-08-19T09:48:14.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T09:48:14.293 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:14.333 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:14.333 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:14.470 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:14.635 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:14.636 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:14.762 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-08-19T09:48:15.322 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:40 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:15.458 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:15.461 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:15.461 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:48:15.461 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:15.461 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:15.461 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-5847.g79766939.el9 ceph 6.6 k 2024-08-19T09:48:15.461 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-5847.g79766939.el9 ceph 2.4 M 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-5847.g79766939.el9 ceph 1.5 M 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 284 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-5847.g79766939.el9 ceph 4.8 M 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-5847.g79766939.el9 ceph 17 M 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T09:48:15.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T09:48:15.463 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:15.464 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:15.465 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:15.465 INFO:teuthology.orchestra.run.smithi001.stdout:Install 37 Packages 2024-08-19T09:48:15.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:15.465 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 31 M 2024-08-19T09:48:15.466 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 109 M 2024-08-19T09:48:15.466 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:16.090 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-19T09:48:16.123 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-19T09:48:16.151 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-19T09:48:16.156 INFO:teuthology.orchestra.run.smithi001.stdout:(1/37): ceph-19.0.0-5847.g79766939.el9.x86_64.r 43 kB/s | 6.6 kB 00:00 2024-08-19T09:48:16.222 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-19T09:48:16.258 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-19T09:48:16.291 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-19T09:48:16.323 INFO:teuthology.orchestra.run.smithi001.stdout:(2/37): ceph-mgr-19.0.0-5847.g79766939.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-08-19T09:48:16.373 INFO:teuthology.orchestra.run.smithi001.stdout:(3/37): ceph-mds-19.0.0-5847.g79766939.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-08-19T09:48:16.382 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-19T09:48:16.448 INFO:teuthology.orchestra.run.smithi001.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5847.g7976 3.7 MB/s | 284 kB 00:00 2024-08-19T09:48:16.461 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-19T09:48:16.502 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-19T09:48:16.537 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-19T09:48:16.567 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-19T09:48:16.601 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-19T09:48:16.635 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-19T09:48:16.674 INFO:teuthology.orchestra.run.smithi001.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-08-19T09:48:16.683 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-19T09:48:16.718 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-19T09:48:16.753 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-19T09:48:16.801 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-19T09:48:16.833 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-19T09:48:16.857 INFO:teuthology.orchestra.run.smithi001.stdout:(6/37): ceph-osd-19.0.0-5847.g79766939.el9.x86_ 32 MB/s | 17 MB 00:00 2024-08-19T09:48:16.906 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-19T09:48:16.966 INFO:teuthology.orchestra.run.smithi001.stdout:(7/37): ceph-mon-19.0.0-5847.g79766939.el9.x86_ 5.9 MB/s | 4.8 MB 00:00 2024-08-19T09:48:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 3.8 MB/s | 1.2 MB 00:00 2024-08-19T09:48:17.045 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-19T09:48:17.050 INFO:teuthology.orchestra.run.smithi001.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-08-19T09:48:17.083 INFO:teuthology.orchestra.run.smithi001.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-08-19T09:48:17.133 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-19T09:48:17.167 INFO:teuthology.orchestra.run.smithi001.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-08-19T09:48:17.186 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-19T09:48:17.192 INFO:teuthology.orchestra.run.smithi001.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 598 kB/s | 135 kB 00:00 2024-08-19T09:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-19T09:48:17.242 INFO:teuthology.orchestra.run.smithi001.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-08-19T09:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-19T09:48:17.284 INFO:teuthology.orchestra.run.smithi001.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 379 kB/s | 35 kB 00:00 2024-08-19T09:48:17.301 INFO:teuthology.orchestra.run.smithi001.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-08-19T09:48:17.326 INFO:teuthology.orchestra.run.smithi001.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-08-19T09:48:17.337 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-19T09:48:17.352 INFO:teuthology.orchestra.run.smithi001.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-08-19T09:48:17.377 INFO:teuthology.orchestra.run.smithi001.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 551 kB/s | 51 kB 00:00 2024-08-19T09:48:17.411 INFO:teuthology.orchestra.run.smithi001.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-08-19T09:48:17.436 INFO:teuthology.orchestra.run.smithi001.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-08-19T09:48:17.469 INFO:teuthology.orchestra.run.smithi001.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-08-19T09:48:17.495 INFO:teuthology.orchestra.run.smithi001.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 462 kB/s | 11 kB 00:00 2024-08-19T09:48:17.520 INFO:teuthology.orchestra.run.smithi001.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-08-19T09:48:17.532 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-19T09:48:17.545 INFO:teuthology.orchestra.run.smithi001.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-08-19T09:48:17.571 INFO:teuthology.orchestra.run.smithi001.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 927 kB/s | 23 kB 00:00 2024-08-19T09:48:17.579 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-19T09:48:17.596 INFO:teuthology.orchestra.run.smithi001.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-08-19T09:48:17.621 INFO:teuthology.orchestra.run.smithi001.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-08-19T09:48:17.646 INFO:teuthology.orchestra.run.smithi001.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-08-19T09:48:17.656 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-19T09:48:17.672 INFO:teuthology.orchestra.run.smithi001.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-19T09:48:17.697 INFO:teuthology.orchestra.run.smithi001.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-08-19T09:48:17.722 INFO:teuthology.orchestra.run.smithi001.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-08-19T09:48:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-19T09:48:17.747 INFO:teuthology.orchestra.run.smithi001.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-19T09:48:17.773 INFO:teuthology.orchestra.run.smithi001.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-19T09:48:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-19T09:48:17.806 INFO:teuthology.orchestra.run.smithi001.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-08-19T09:48:17.832 INFO:teuthology.orchestra.run.smithi001.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-08-19T09:48:17.857 INFO:teuthology.orchestra.run.smithi001.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-08-19T09:48:17.882 INFO:teuthology.orchestra.run.smithi001.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-08-19T09:48:17.886 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:17.886 INFO:teuthology.orchestra.run.smithi001.stdout:Total 13 MB/s | 31 MB 00:02 2024-08-19T09:48:17.916 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-19T09:48:18.040 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:18.083 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:18.083 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:18.085 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 32/37 2024-08-19T09:48:18.113 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T09:48:18.139 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T09:48:18.139 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:18.140 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T09:48:18.140 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T09:48:18.140 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T09:48:18.140 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:18.399 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:18.400 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:19.614 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T09:48:19.641 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T09:48:19.641 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:19.641 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T09:48:19.641 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T09:48:19.641 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T09:48:19.641 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:19.887 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:19.952 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-19T09:48:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-19T09:48:20.038 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-19T09:48:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T09:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T09:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T09:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T09:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T09:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:20.113 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-19T09:48:20.162 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-19T09:48:20.195 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-19T09:48:20.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-19T09:48:20.324 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T09:48:20.347 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T09:48:20.347 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:20.347 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T09:48:20.347 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T09:48:20.348 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T09:48:20.348 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:20.375 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-19T09:48:20.411 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T09:48:20.423 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-19T09:48:20.458 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-19T09:48:20.487 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-19T09:48:20.521 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-19T09:48:20.555 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-19T09:48:20.596 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-19T09:48:20.640 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-19T09:48:20.673 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-19T09:48:20.723 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-19T09:48:20.762 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-19T09:48:20.848 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-19T09:48:21.134 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-19T09:48:21.215 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-19T09:48:21.275 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-19T09:48:21.312 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-19T09:48:21.371 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-19T09:48:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-19T09:48:21.640 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-19T09:48:21.700 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-19T09:48:21.780 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-19T09:48:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-19T09:48:21.907 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-19T09:48:22.042 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-19T09:48:22.211 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 32/37 2024-08-19T09:48:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T09:48:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-5847.g79766939.el9.x86_64 1/37 2024-08-19T09:48:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2/37 2024-08-19T09:48:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 3/37 2024-08-19T09:48:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 4/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 5/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 6/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-19T09:48:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-19T09:48:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-19T09:48:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-19T09:48:22.241 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T09:48:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T09:48:22.268 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:22.268 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T09:48:22.268 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T09:48:22.268 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T09:48:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T09:48:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T09:48:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T09:48:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T09:48:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T09:48:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T09:48:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:23.321 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:23.661 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2024-08-19T09:48:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T09:48:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T09:48:23.836 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:23.836 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T09:48:23.836 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T09:48:23.836 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T09:48:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:24.204 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:51 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:48:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T09:48:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T09:48:24.325 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:24.325 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T09:48:24.325 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T09:48:24.325 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T09:48:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:48:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:24.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2024-08-19T09:48:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T09:48:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T09:48:24.584 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:24.584 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T09:48:24.584 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T09:48:24.584 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T09:48:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T09:48:24.929 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:51 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 772 k 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:25.054 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:48:25.055 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 772 k 2024-08-19T09:48:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 779 k 2024-08-19T09:48:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:25.331 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-19.0.0-5847.g79766939.el9.noarch.rpm 2.7 MB/s | 772 kB 00:00 2024-08-19T09:48:25.332 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:25.332 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.7 MB/s | 772 kB 00:00 2024-08-19T09:48:25.332 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:25.337 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:25.337 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:25.411 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:25.884 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:25.974 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:26.396 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-5847.g79766939.el9.x86_64 1/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 3/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 4/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 5/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 6/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-19T09:48:26.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-19T09:48:26.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-19T09:48:26.399 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-19T09:48:26.399 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-19T09:48:26.399 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-19T09:48:26.399 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-19T09:48:26.399 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-19T09:48:26.399 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-19T09:48:26.400 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-19T09:48:26.400 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-19T09:48:26.400 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-19T09:48:26.400 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-19T09:48:26.400 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-19T09:48:26.400 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-19T09:48:26.533 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:26.533 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:26.534 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:26.534 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:26.534 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:26.534 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:26.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T09:48:27.278 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-19T09:48:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:27.332 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T09:48:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T09:48:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:27.335 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:27.406 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5847.g79766939.el9 ceph 145 k 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:27.407 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:27.408 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:48:27.408 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:27.408 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 145 k 2024-08-19T09:48:27.408 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 436 k 2024-08-19T09:48:27.408 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:27.628 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-19.0.0-5847.g797669 658 kB/s | 145 kB 00:00 2024-08-19T09:48:27.629 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:27.629 INFO:teuthology.orchestra.run.smithi082.stdout:Total 652 kB/s | 145 kB 00:00 2024-08-19T09:48:27.629 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:27.637 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:27.637 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:27.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-08-19T09:48:27.693 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:27.693 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:27.846 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:27.921 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T09:48:27.946 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T09:48:27.946 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:27.946 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T09:48:27.946 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:28.216 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:28.303 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:48:28.347 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:28.348 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:48:28.348 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:28.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-08-19T09:48:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T09:48:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:28.915 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:28.915 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:28.950 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:29.081 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:29.082 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:29.082 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:48:29.082 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 772 k 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 772 k 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 779 k 2024-08-19T09:48:29.083 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:29.147 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2024-08-19T09:48:29.360 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-19.0.0-5847.g79766939.el9.noarch.rpm 2.7 MB/s | 772 kB 00:00 2024-08-19T09:48:29.360 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:29.360 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.7 MB/s | 772 kB 00:00 2024-08-19T09:48:29.360 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:29.366 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:29.366 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:29.372 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:29.372 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:29.446 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:29.694 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:29.780 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:48:29.821 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:29.822 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:48:29.822 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:29.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-dashboard 2024-08-19T09:48:29.929 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:30.309 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:30.463 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:57 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:30.536 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:48:30.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:30.536 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:30.537 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:30.537 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:30.537 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:30.590 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:30.591 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 14 M 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 34 k 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 16 k 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T09:48:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-08-19T09:48:30.593 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:30.594 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 19 M 2024-08-19T09:48:30.594 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 158 M 2024-08-19T09:48:30.594 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:30.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T09:48:30.956 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): ceph-grafana-dashboards-19.0.0-5847.g79 235 kB/s | 34 kB 00:00 2024-08-19T09:48:30.982 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): ceph-prometheus-alerts-19.0.0-5847.g797 96 kB/s | 16 kB 00:00 2024-08-19T09:48:31.281 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:31.413 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5847.g79766939.el9 ceph 145 k 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:31.414 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:31.415 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:48:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:31.415 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 145 k 2024-08-19T09:48:31.415 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 436 k 2024-08-19T09:48:31.415 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:31.465 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): ceph-mgr-dashboard-19.0.0-5847.g7976693 22 MB/s | 14 MB 00:00 2024-08-19T09:48:31.615 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-19.0.0-5847.g797669 724 kB/s | 145 kB 00:00 2024-08-19T09:48:31.616 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:31.616 INFO:teuthology.orchestra.run.smithi001.stdout:Total 718 kB/s | 145 kB 00:00 2024-08-19T09:48:31.616 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:31.624 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:31.624 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:31.684 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:31.684 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:31.835 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T09:48:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T09:48:31.958 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:48:31.958 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T09:48:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:32.040 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 254 kB/s | 269 kB 00:01 2024-08-19T09:48:32.090 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-08-19T09:48:32.116 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.3 MB/s | 863 kB 00:00 2024-08-19T09:48:32.141 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 386 kB/s | 19 kB 00:00 2024-08-19T09:48:32.175 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.4 MB/s | 144 kB 00:00 2024-08-19T09:48:32.200 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-08-19T09:48:32.225 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-08-19T09:48:32.292 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-08-19T09:48:32.334 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 750 kB/s | 1.0 MB 00:01 2024-08-19T09:48:32.339 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:32.339 INFO:teuthology.orchestra.run.smithi082.stdout:Total 11 MB/s | 19 MB 00:01 2024-08-19T09:48:32.390 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:32.410 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:32.410 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:32.630 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:32.631 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:32.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T09:48:32.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:32.851 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:32.851 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:48:32.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:32.851 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:33.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-08-19T09:48:33.218 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:33.284 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-19T09:48:33.320 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-19T09:48:33.367 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-19T09:48:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-19T09:48:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-19T09:48:33.607 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-19T09:48:33.622 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:33.702 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-19T09:48:33.711 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:48:33.756 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:33.757 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:48:33.757 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-19T09:48:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-19T09:48:33.820 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 10/12 2024-08-19T09:48:33.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-08-19T09:48:34.386 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:59 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:34.520 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 14 M 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 34 k 2024-08-19T09:48:34.521 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 16 k 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T09:48:34.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T09:48:34.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-08-19T09:48:34.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:34.524 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 19 M 2024-08-19T09:48:34.524 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 158 M 2024-08-19T09:48:34.524 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:35.310 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): ceph-grafana-dashboards-19.0.0-5847.g79 225 kB/s | 34 kB 00:00 2024-08-19T09:48:35.337 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): ceph-prometheus-alerts-19.0.0-5847.g797 92 kB/s | 16 kB 00:00 2024-08-19T09:48:35.561 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-08-19T09:48:35.653 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-08-19T09:48:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-08-19T09:48:35.719 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-08-19T09:48:35.878 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): ceph-mgr-dashboard-19.0.0-5847.g7976693 20 MB/s | 14 MB 00:00 2024-08-19T09:48:35.928 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-08-19T09:48:35.954 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-19T09:48:36.004 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.2 MB/s | 2.0 MB 00:00 2024-08-19T09:48:36.029 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-08-19T09:48:36.563 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 862 kB/s | 863 kB 00:01 2024-08-19T09:48:36.567 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:36.567 INFO:teuthology.orchestra.run.smithi001.stdout:Total 9.4 MB/s | 19 MB 00:02 2024-08-19T09:48:36.619 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:36.639 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:36.640 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:36.871 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:36.871 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:37.059 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 11/12 2024-08-19T09:48:37.116 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T09:48:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:37.539 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-19T09:48:37.574 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-19T09:48:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-19T09:48:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-19T09:48:37.730 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-19T09:48:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-19T09:48:37.974 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-19T09:48:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-19T09:48:38.068 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-19T09:48:38.100 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 10/12 2024-08-19T09:48:39.108 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T09:48:39.108 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 1/12 2024-08-19T09:48:39.108 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 2/12 2024-08-19T09:48:39.108 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 3/12 2024-08-19T09:48:39.108 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-19T09:48:39.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-19T09:48:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-19T09:48:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:39.430 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T09:48:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:39.432 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:39.683 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-19T09:48:40.218 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:07 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:40.345 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:40.346 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-08-19T09:48:40.346 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 7.4 M 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T09:48:40.347 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-08-19T09:48:40.348 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:40.349 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 38 M 2024-08-19T09:48:40.349 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 209 M 2024-08-19T09:48:40.349 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:40.882 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 839 kB/s | 195 kB 00:00 2024-08-19T09:48:40.991 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-08-19T09:48:41.024 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-08-19T09:48:41.133 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-58 15 MB/s | 7.4 MB 00:00 2024-08-19T09:48:41.183 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 115 kB/s | 18 kB 00:00 2024-08-19T09:48:41.267 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 264 kB/s | 35 kB 00:00 2024-08-19T09:48:41.417 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): python3-devel-3.9.19-4.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-08-19T09:48:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 11/12 2024-08-19T09:48:41.537 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T09:48:41.809 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-08-19T09:48:41.917 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-08-19T09:48:42.467 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.7 MB/s | 4.7 MB 00:01 2024-08-19T09:48:42.734 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.9 MB/s | 5.1 MB 00:01 2024-08-19T09:48:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 1/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 2/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 3/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-19T09:48:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-19T09:48:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-19T09:48:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-19T09:48:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:43.767 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:43.768 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:44.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-19T09:48:44.571 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:44.617 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-08-19T09:48:44.618 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:44.618 INFO:teuthology.orchestra.run.smithi082.stdout:Total 8.9 MB/s | 38 MB 00:04 2024-08-19T09:48:44.709 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:44.710 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-08-19T09:48:44.710 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 7.4 M 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T09:48:44.711 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:44.712 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 38 M 2024-08-19T09:48:44.713 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 209 M 2024-08-19T09:48:44.713 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:44.971 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:44.996 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:44.996 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:45.004 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-08-19T09:48:45.046 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-08-19T09:48:45.071 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-08-19T09:48:45.113 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 434 kB/s | 18 kB 00:00 2024-08-19T09:48:45.163 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 25 MB/s | 3.0 MB 00:00 2024-08-19T09:48:45.188 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-08-19T09:48:45.289 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 38 MB/s | 4.7 MB 00:00 2024-08-19T09:48:45.314 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-devel-3.9.19-4.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-08-19T09:48:45.402 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:45.403 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:45.423 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-58 13 MB/s | 7.4 MB 00:00 2024-08-19T09:48:45.448 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-08-19T09:48:45.523 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-08-19T09:48:45.915 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-08-19T09:48:45.916 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:45.916 INFO:teuthology.orchestra.run.smithi001.stdout:Total 32 MB/s | 38 MB 00:01 2024-08-19T09:48:46.161 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:46.213 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-19T09:48:46.248 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-19T09:48:46.278 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:46.306 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:46.306 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:46.346 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-19T09:48:46.383 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 4/12 2024-08-19T09:48:46.558 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-19T09:48:46.590 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-19T09:48:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-19T09:48:46.742 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:46.743 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:47.301 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-19T09:48:47.410 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-19T09:48:47.520 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-19T09:48:47.656 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-19T09:48:47.758 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-19T09:48:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 4/12 2024-08-19T09:48:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-19T09:48:48.006 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-19T09:48:48.084 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-19T09:48:48.618 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T09:48:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-19T09:48:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-19T09:48:49.638 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-19T09:48:49.664 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T09:48:50.167 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T09:48:51.252 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-19T09:48:51.288 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 1/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-19T09:48:52.209 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-19T09:48:52.210 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-19T09:48:52.210 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 9/12 2024-08-19T09:48:52.210 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T09:48:52.210 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-19T09:48:52.645 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T09:48:52.646 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T09:48:52.647 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T09:48:52.647 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T09:48:52.647 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T09:48:52.647 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:52.647 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:53.019 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-rook 2024-08-19T09:48:53.564 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:20 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 1/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 9/12 2024-08-19T09:48:53.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T09:48:53.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-19T09:48:53.691 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 49 k 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T09:48:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2024-08-19T09:48:53.694 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:53.695 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.2 M 2024-08-19T09:48:53.695 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-08-19T09:48:53.695 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:54.016 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): ceph-mgr-rook-19.0.0-5847.g79766939.el9 303 kB/s | 49 kB 00:00 2024-08-19T09:48:54.065 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T09:48:54.066 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T09:48:54.067 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T09:48:54.067 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T09:48:54.067 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T09:48:54.067 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:54.067 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:54.349 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 39 kB/s | 19 kB 00:00 2024-08-19T09:48:54.366 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 50 kB/s | 26 kB 00:00 2024-08-19T09:48:54.416 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 555 kB/s | 222 kB 00:00 2024-08-19T09:48:54.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-08-19T09:48:54.475 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-19T09:48:54.500 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 649 kB/s | 54 kB 00:00 2024-08-19T09:48:54.525 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 643 kB/s | 32 kB 00:00 2024-08-19T09:48:54.551 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-08-19T09:48:54.576 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 191 kB/s | 14 kB 00:00 2024-08-19T09:48:54.609 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-08-19T09:48:54.635 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.7 MB/s | 90 kB 00:00 2024-08-19T09:48:54.660 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-08-19T09:48:54.693 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-08-19T09:48:54.697 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:54.697 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-08-19T09:48:54.761 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:54.775 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:54.776 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:54.862 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:48:54.994 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:19 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:48:55.132 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:48:55.133 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:55.133 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:48:55.133 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:55.133 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:48:55.133 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 49 k 2024-08-19T09:48:55.133 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T09:48:55.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:55.135 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:48:55.135 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:48:55.135 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-08-19T09:48:55.135 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:55.135 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.2 M 2024-08-19T09:48:55.135 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-08-19T09:48:55.136 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:48:55.402 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:48:55.566 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-19T09:48:55.620 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-19T09:48:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 289 kB/s | 26 kB 00:00 2024-08-19T09:48:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 168 kB/s | 19 kB 00:00 2024-08-19T09:48:55.685 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-19T09:48:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): ceph-mgr-rook-19.0.0-5847.g79766939.el9 285 kB/s | 49 kB 00:00 2024-08-19T09:48:55.723 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-19T09:48:55.746 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-08-19T09:48:55.764 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-19T09:48:55.780 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-08-19T09:48:55.813 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 798 kB/s | 54 kB 00:00 2024-08-19T09:48:55.847 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-08-19T09:48:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 291 kB/s | 14 kB 00:00 2024-08-19T09:48:55.867 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-19T09:48:55.889 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-08-19T09:48:55.923 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-08-19T09:48:55.948 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-08-19T09:48:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-08-19T09:48:55.975 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-19T09:48:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-08-19T09:48:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-08-19T09:48:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-19T09:48:56.074 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:48:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:48:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:48:56.181 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:48:56.182 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:48:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:48:56.758 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-19T09:48:56.813 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-19T09:48:56.855 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-19T09:48:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-19T09:48:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-19T09:48:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T09:48:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-19T09:48:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-19T09:48:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-19T09:48:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-19T09:48:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-19T09:48:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-19T09:48:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-19T09:48:57.706 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 1/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-19T09:48:57.707 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-19T09:48:58.035 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T09:48:58.036 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T09:48:58.037 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T09:48:58.037 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T09:48:58.037 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T09:48:58.037 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T09:48:58.037 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:58.037 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:48:58.119 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-19T09:48:58.167 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-19T09:48:58.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-19T09:48:58.249 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-19T09:48:58.278 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T09:48:58.317 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-cephadm 2024-08-19T09:48:58.854 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:25 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:48:58.981 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 150 k 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T09:48:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T09:48:58.983 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:58.983 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:48:58.983 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:48:58.983 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-08-19T09:48:58.983 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:48:58.983 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 7.0 M 2024-08-19T09:48:58.984 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 33 M 2024-08-19T09:48:58.984 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:48:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T09:48:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 1/13 2024-08-19T09:48:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-19T09:48:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-19T09:48:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-19T09:48:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-19T09:48:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-19T09:48:59.272 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-08-19T09:48:59.331 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-08-19T09:48:59.356 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): ceph-mgr-cephadm-19.0.0-5847.g79766939.e 613 kB/s | 150 kB 00:00 2024-08-19T09:48:59.381 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-19T09:48:59.391 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-19T09:48:59.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:59.391 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:48:59.391 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:48:59.391 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T09:48:59.391 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:48:59.392 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:48:59.423 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-08-19T09:48:59.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-08-19T09:48:59.765 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.1 MB/s | 6.0 MB 00:00 2024-08-19T09:48:59.768 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:48:59.768 INFO:teuthology.orchestra.run.smithi082.stdout:Total 8.9 MB/s | 7.0 MB 00:00 2024-08-19T09:48:59.839 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:48:59.851 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:48:59.851 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:48:59.950 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:48:59.951 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:00.177 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:25 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:00.309 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:00.310 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:49:00.310 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 150 k 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:00.311 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:00.312 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:00.312 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-08-19T09:49:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:00.312 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 7.0 M 2024-08-19T09:49:00.312 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-08-19T09:49:00.312 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:00.428 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-19T09:49:00.492 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-19T09:49:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-08-19T09:49:00.760 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-08-19T09:49:00.785 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): ceph-mgr-cephadm-19.0.0-5847.g79766939.e 646 kB/s | 150 kB 00:00 2024-08-19T09:49:00.801 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-08-19T09:49:00.844 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-08-19T09:49:00.935 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-08-19T09:49:00.938 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:00.938 INFO:teuthology.orchestra.run.smithi001.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-08-19T09:49:01.011 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:01.018 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-19T09:49:01.022 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:01.023 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-19T09:49:01.125 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:01.127 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-19T09:49:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T09:49:01.433 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:01.617 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-19T09:49:01.672 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-19T09:49:01.964 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T09:49:01.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/6 2024-08-19T09:49:01.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-19T09:49:01.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-19T09:49:01.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-19T09:49:01.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-19T09:49:02.230 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T09:49:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T09:49:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T09:49:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T09:49:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:02.234 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:02.314 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-19T09:49:02.386 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-19T09:49:02.410 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T09:49:02.494 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2024-08-19T09:49:03.037 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T09:49:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/6 2024-08-19T09:49:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-19T09:49:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-19T09:49:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-19T09:49:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-19T09:49:03.164 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:03.165 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:03.165 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:49:03.165 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:03.165 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:03.165 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 901 k 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-08-19T09:49:03.166 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:03.167 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 981 k 2024-08-19T09:49:03.167 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 2.9 M 2024-08-19T09:49:03.167 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-19T09:49:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T09:49:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T09:49:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:03.417 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:03.550 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 622 kB/s | 80 kB 00:00 2024-08-19T09:49:03.668 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-08-19T09:49:03.683 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-fuse-19.0.0-5847.g79766939.el9.x86_ 3.4 MB/s | 901 kB 00:00 2024-08-19T09:49:03.684 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:03.684 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.8 MB/s | 981 kB 00:00 2024-08-19T09:49:03.695 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:03.717 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:03.717 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:03.797 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:03.797 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:03.990 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:04.129 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T09:49:04.189 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T09:49:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:04.351 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:04.355 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:49:04.355 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:04.355 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 901 k 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 981 k 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.9 M 2024-08-19T09:49:04.356 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:04.614 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 502 kB/s | 80 kB 00:00 2024-08-19T09:49:04.706 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-fuse-19.0.0-5847.g79766939.el9.x86_ 3.5 MB/s | 901 kB 00:00 2024-08-19T09:49:04.706 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:04.706 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.7 MB/s | 981 kB 00:00 2024-08-19T09:49:04.718 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:04.741 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:04.741 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:04.823 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:04.823 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:04.868 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T09:49:04.868 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/2 2024-08-19T09:49:05.027 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:05.140 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T09:49:05.140 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:05.140 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:05.141 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T09:49:05.141 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:05.141 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T09:49:05.248 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T09:49:05.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-volume 2024-08-19T09:49:05.884 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T09:49:05.885 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/2 2024-08-19T09:49:06.005 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:06.137 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:06.138 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:06.138 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-08-19T09:49:06.138 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 285 k 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 285 k 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 1.3 M 2024-08-19T09:49:06.139 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:06.172 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T09:49:06.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:06.172 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:06.172 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T09:49:06.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:06.172 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:06.371 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-volume-19.0.0-5847.g79766939.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-08-19T09:49:06.372 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:06.372 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-08-19T09:49:06.372 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:06.380 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:06.380 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:06.400 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:06.400 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:06.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-volume 2024-08-19T09:49:06.615 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:06.682 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:49:06.699 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:49:06.699 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:49:06.699 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T09:49:06.699 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:07.033 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:07.165 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:07.166 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:07.166 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-08-19T09:49:07.166 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:07.166 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 285 k 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 285 k 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.3 M 2024-08-19T09:49:07.167 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:07.376 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-volume-19.0.0-5847.g79766939.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-08-19T09:49:07.377 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:07.377 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-08-19T09:49:07.377 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:07.386 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:07.386 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:07.407 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:07.407 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:07.680 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:49:07.681 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:07.681 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:07.681 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:49:07.681 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:07.681 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:07.703 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:49:07.719 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:49:07.720 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:49:07.720 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T09:49:07.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:07.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2024-08-19T09:49:08.508 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:08.639 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 127 k 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:08.640 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 127 k 2024-08-19T09:49:08.641 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 456 k 2024-08-19T09:49:08.641 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:08.680 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T09:49:08.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:08.680 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:08.680 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T09:49:08.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:08.680 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:08.834 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-19.0.0-5847.g79766939.el9.x86_64 653 kB/s | 127 kB 00:00 2024-08-19T09:49:08.835 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:08.835 INFO:teuthology.orchestra.run.smithi082.stdout:Total 649 kB/s | 127 kB 00:00 2024-08-19T09:49:08.835 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:08.841 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:08.841 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:08.904 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:08.904 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:08.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-08-19T09:49:09.063 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:09.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:09.524 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:09.565 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:09.660 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 127 k 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 127 k 2024-08-19T09:49:09.661 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 456 k 2024-08-19T09:49:09.662 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:09.798 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:09.798 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:09.799 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:09.799 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:09.799 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:09.799 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:09.873 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-19.0.0-5847.g79766939.el9.x86_64 601 kB/s | 127 kB 00:00 2024-08-19T09:49:09.874 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:09.874 INFO:teuthology.orchestra.run.smithi001.stdout:Total 596 kB/s | 127 kB 00:00 2024-08-19T09:49:09.874 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:09.880 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:09.880 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:09.947 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:09.947 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:10.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2024-08-19T09:49:10.100 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:10.198 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:10.578 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:10.623 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:10.708 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:10.751 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:10.752 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:10.752 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:10.829 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2024-08-19T09:49:10.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:10.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:10.840 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:10.840 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:10.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:10.840 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:11.100 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-08-19T09:49:11.388 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:11.515 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:11.516 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 32 k 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 32 k 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 142 k 2024-08-19T09:49:11.517 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:11.655 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:36 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:11.662 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-19.0.0-5847.g79766939.el9.x86_6 219 kB/s | 32 kB 00:00 2024-08-19T09:49:11.662 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:11.663 INFO:teuthology.orchestra.run.smithi082.stdout:Total 217 kB/s | 32 kB 00:00 2024-08-19T09:49:11.663 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:11.666 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:11.667 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:11.670 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:11.670 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:11.743 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:11.763 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:11.791 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:11.792 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:11.792 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:11.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-08-19T09:49:11.875 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:12.230 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:12.419 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:12.479 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:12.479 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:12.479 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:12.479 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:12.479 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:12.479 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:12.551 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 32 k 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:12.552 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:12.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:12.553 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:49:12.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:12.553 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 32 k 2024-08-19T09:49:12.553 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 142 k 2024-08-19T09:49:12.553 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:12.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2024-08-19T09:49:12.701 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-19.0.0-5847.g79766939.el9.x86_6 214 kB/s | 32 kB 00:00 2024-08-19T09:49:12.701 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:12.702 INFO:teuthology.orchestra.run.smithi001.stdout:Total 212 kB/s | 32 kB 00:00 2024-08-19T09:49:12.702 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:12.706 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:12.706 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:12.709 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:12.710 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:12.802 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:12.916 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:13.201 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:13.269 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:13.284 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:13.327 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:13.328 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:13.328 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:13.406 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2024-08-19T09:49:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:13.478 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:13.479 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:13.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:13.479 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:13.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-08-19T09:49:13.941 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:14.025 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:14.068 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:14.069 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:14.069 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:14.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2024-08-19T09:49:14.227 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:39 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:14.314 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:14.359 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:14.360 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:14.360 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:14.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-08-19T09:49:14.675 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:14.759 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:14.802 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:14.803 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:14.803 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:14.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2024-08-19T09:49:14.998 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:39 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:15.086 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:15.132 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:15.133 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:15.133 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:15.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-08-19T09:49:15.412 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:42 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:15.496 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:15.539 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:15.540 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:15.540 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:15.610 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2024-08-19T09:49:15.771 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:15.859 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:15.904 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:15.905 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:15.906 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:15.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-08-19T09:49:16.149 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:16.232 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:16.274 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:16.275 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:16.275 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:16.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2024-08-19T09:49:16.546 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:16.635 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:16.681 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:16.682 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:16.682 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:16.760 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-08-19T09:49:16.892 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:16.977 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:17.019 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:17.020 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-08-19T09:49:17.020 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:17.096 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2024-08-19T09:49:17.319 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:42 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:17.407 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:17.452 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:17.453 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:17.453 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:17.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-08-19T09:49:17.631 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:44 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:17.758 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 85 k 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:17.759 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:17.760 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:49:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:17.760 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 85 k 2024-08-19T09:49:17.760 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 231 k 2024-08-19T09:49:17.760 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:17.940 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-19.0.0-5847.g79766939.el9.x86_64.rpm 473 kB/s | 85 kB 00:00 2024-08-19T09:49:17.941 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:17.941 INFO:teuthology.orchestra.run.smithi082.stdout:Total 470 kB/s | 85 kB 00:00 2024-08-19T09:49:17.941 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:17.947 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:17.948 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:18.010 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:18.010 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:18.089 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:18.148 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:18.176 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T09:49:18.223 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:18.224 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-08-19T09:49:18.224 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:18.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-08-19T09:49:18.608 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:18.850 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:18.879 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:18.879 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:18.879 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:18.879 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:18.880 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:18.880 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:18.982 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:18.982 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 85 k 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 85 k 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 231 k 2024-08-19T09:49:18.983 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:19.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2024-08-19T09:49:19.180 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-19.0.0-5847.g79766939.el9.x86_64.rpm 435 kB/s | 85 kB 00:00 2024-08-19T09:49:19.180 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:19.181 INFO:teuthology.orchestra.run.smithi001.stdout:Total 431 kB/s | 85 kB 00:00 2024-08-19T09:49:19.181 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:19.188 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:19.189 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:19.254 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:19.255 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:19.401 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:19.512 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:19.712 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:46 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:19.843 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:19.843 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:19.843 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.2 M 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 14 M 2024-08-19T09:49:19.844 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:19.874 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:20.138 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:20.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:20.138 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:20.139 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:20.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:20.139 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:20.168 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-19.0.0-5847.g79766939.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-08-19T09:49:20.168 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:20.169 INFO:teuthology.orchestra.run.smithi082.stdout:Total 9.7 MB/s | 3.2 MB 00:00 2024-08-19T09:49:20.169 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:20.178 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:20.178 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:20.262 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:20.263 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:20.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-08-19T09:49:20.727 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:20.794 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:20.820 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:20.821 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:49:20.821 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T09:49:20.821 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T09:49:20.821 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T09:49:20.821 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:20.980 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:45 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:21.112 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:21.112 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:21.112 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.2 M 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 14 M 2024-08-19T09:49:21.113 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:21.458 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-19.0.0-5847.g79766939.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-08-19T09:49:21.459 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:21.459 INFO:teuthology.orchestra.run.smithi001.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-08-19T09:49:21.459 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:21.469 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:21.469 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:21.554 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:21.554 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:21.978 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:21.978 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:21.978 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:21.978 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:21.979 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:21.979 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:22.011 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:22.092 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:22.092 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T09:49:22.092 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T09:49:22.093 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T09:49:22.093 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T09:49:22.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:22.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2024-08-19T09:49:22.812 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:49 ago on Mon 19 Aug 2024 09:47:33 AM UTC. 2024-08-19T09:49:22.939 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 171 k 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-08-19T09:49:22.940 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-08-19T09:49:22.941 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-08-19T09:49:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:22.941 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 171 k 2024-08-19T09:49:22.941 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 490 k 2024-08-19T09:49:22.941 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-08-19T09:49:23.103 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:23.103 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:23.103 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:23.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:23.164 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-19.0.0-5847.g79766939.el9.x86_64.rpm 769 kB/s | 171 kB 00:00 2024-08-19T09:49:23.164 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:23.164 INFO:teuthology.orchestra.run.smithi082.stdout:Total 763 kB/s | 171 kB 00:00 2024-08-19T09:49:23.165 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-08-19T09:49:23.172 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-08-19T09:49:23.172 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-08-19T09:49:23.236 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-08-19T09:49:23.236 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-08-19T09:49:23.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-08-19T09:49:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-08-19T09:49:23.493 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:23.940 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:48 ago on Mon 19 Aug 2024 09:47:35 AM UTC. 2024-08-19T09:49:24.073 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-08-19T09:49:24.073 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 171 k 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 171 k 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 490 k 2024-08-19T09:49:24.074 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-08-19T09:49:24.136 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:24.137 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:24.137 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-08-19T09:49:24.137 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:24.137 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:24.137 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-08-19T09:49:24.286 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-19.0.0-5847.g79766939.el9.x86_64.rpm 808 kB/s | 171 kB 00:00 2024-08-19T09:49:24.286 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-08-19T09:49:24.287 INFO:teuthology.orchestra.run.smithi001.stdout:Total 803 kB/s | 171 kB 00:00 2024-08-19T09:49:24.287 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-08-19T09:49:24.295 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-08-19T09:49:24.295 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-08-19T09:49:24.362 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-08-19T09:49:24.362 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-08-19T09:49:24.437 DEBUG:teuthology.parallel:result is None 2024-08-19T09:49:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-08-19T09:49:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:25.021 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T09:49:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:25.279 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-08-19T09:49:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T09:49:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:25.279 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-08-19T09:49:25.507 DEBUG:teuthology.parallel:result is None 2024-08-19T09:49:25.508 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T09:49:25.665 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T09:49:25.686 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-5847.g79766939.el9 2024-08-19T09:49:25.686 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5847.g79766939.el9 2024-08-19T09:49:25.687 INFO:teuthology.task.install:The correct ceph version 19.0.0-5847.g79766939 is installed. 2024-08-19T09:49:25.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T09:49:25.836 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T09:49:25.862 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-5847.g79766939.el9 2024-08-19T09:49:25.862 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5847.g79766939.el9 2024-08-19T09:49:25.862 INFO:teuthology.task.install:The correct ceph version 19.0.0-5847.g79766939 is installed. 2024-08-19T09:49:25.864 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-19T09:49:25.864 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:25.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T09:49:25.889 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:25.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T09:49:25.931 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-19T09:49:25.931 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:25.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T09:49:25.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T09:49:26.020 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:26.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T09:49:26.048 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T09:49:26.116 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-19T09:49:26.116 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:26.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T09:49:26.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T09:49:26.207 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:26.207 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T09:49:26.234 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T09:49:26.300 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-19T09:49:26.300 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:26.300 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T09:49:26.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T09:49:26.389 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:26.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T09:49:26.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T09:49:26.480 INFO:teuthology.run_tasks:Running task ceph... 2024-08-19T09:49:26.560 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-19T09:49:26.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-08-19T09:49:26.563 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2024-08-19T09:49:26.588 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-19T09:49:26.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-19T09:49:26.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-19T09:49:26.654 INFO:tasks.ceph:Creating extra log directories... 2024-08-19T09:49:26.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-19T09:49:26.692 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-19T09:49:26.728 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-19T09:49:26.728 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-08-19T09:49:26.728 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301', 'branch': 'main', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7859301', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '964adda23da9f7b057cb34fdc0b82aeb7ed65324'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1858, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b', 'sleep_before_teardown': 0, 'subset': '29813/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '964adda23da9f7b057cb34fdc0b82aeb7ed65324', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgkZ07bNVkZstG3mqtkpYuRHGwod/ZAn+vKc4p9iLxbPj3RkCZBFsXIjN0pqtGAreUZmw7yBJOBuMftUT2W05E=', 'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNm3VXhyZiJlzkEtoz19U+Qx82t001n0DX4IMYjA6/bVI/QoQTNashFB7cBRhG/JcFPe/1YZtCgoC0UUbJzLBUE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-18_20:00:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-19T09:49:26.729 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-19T09:49:26.756 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-19T09:49:26.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-08-19T09:49:26.811 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2024-08-19T09:49:26.838 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:26.838 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T09:49:26.888 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T09:49:26.888 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-08-19T09:49:26.943 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T09:49:26.943 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-08-19 09:49:22.349803618 +0000 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-08-19 09:46:53.793801510 +0000 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-08-19 09:46:53.793801510 +0000 2024-08-19T09:49:26.944 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-08-19 09:46:53.793801510 +0000 2024-08-19T09:49:26.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T09:49:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-08-19T09:49:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-08-19T09:49:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.4729e-05 s, 5.4 MB/s 2024-08-19T09:49:27.007 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T09:49:27.063 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-08-19T09:49:27.118 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-08-19 09:49:22.349803618 +0000 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-08-19 09:46:54.114802059 +0000 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-08-19 09:46:54.114802059 +0000 2024-08-19T09:49:27.119 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-08-19 09:46:54.114802059 +0000 2024-08-19T09:49:27.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T09:49:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-08-19T09:49:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-08-19T09:49:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.5529e-05 s, 5.4 MB/s 2024-08-19T09:49:27.183 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T09:49:27.238 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 709 Links: 1 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-08-19 09:49:22.349803618 +0000 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-08-19 09:46:54.430802599 +0000 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-08-19 09:46:54.430802599 +0000 2024-08-19T09:49:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-08-19 09:46:54.430802599 +0000 2024-08-19T09:49:27.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T09:49:27.357 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-08-19T09:49:27.357 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-08-19T09:49:27.357 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.4584e-05 s, 5.4 MB/s 2024-08-19T09:49:27.358 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T09:49:27.414 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 727 Links: 1 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-08-19 09:49:22.349803618 +0000 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-08-19 09:46:54.746803139 +0000 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-08-19 09:46:54.746803139 +0000 2024-08-19T09:49:27.470 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-08-19 09:46:54.746803139 +0000 2024-08-19T09:49:27.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T09:49:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-08-19T09:49:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-08-19T09:49:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.9257e-05 s, 5.7 MB/s 2024-08-19T09:49:27.533 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T09:49:27.589 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:27.589 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:27.589 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T09:49:27.606 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T09:49:27.606 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-08-19 09:49:21.085857707 +0000 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-08-19 09:46:53.781952599 +0000 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-08-19 09:46:53.781952599 +0000 2024-08-19T09:49:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-08-19 09:46:53.781952599 +0000 2024-08-19T09:49:27.664 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T09:49:27.727 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-08-19T09:49:27.727 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-08-19T09:49:27.727 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000216125 s, 2.4 MB/s 2024-08-19T09:49:27.728 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T09:49:27.786 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-08-19T09:49:27.841 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-08-19 09:49:21.085857707 +0000 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-08-19 09:46:54.116947563 +0000 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-08-19 09:46:54.116947563 +0000 2024-08-19T09:49:27.842 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-08-19 09:46:54.116947563 +0000 2024-08-19T09:49:27.842 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T09:49:27.905 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-08-19T09:49:27.905 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-08-19T09:49:27.906 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000213797 s, 2.4 MB/s 2024-08-19T09:49:27.907 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T09:49:27.963 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-08-19 09:49:21.086857686 +0000 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-08-19 09:46:54.450942541 +0000 2024-08-19T09:49:28.019 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-08-19 09:46:54.450942541 +0000 2024-08-19T09:49:28.020 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-08-19 09:46:54.450942541 +0000 2024-08-19T09:49:28.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T09:49:28.082 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-08-19T09:49:28.082 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-08-19T09:49:28.082 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000189953 s, 2.7 MB/s 2024-08-19T09:49:28.083 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T09:49:28.140 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-08-19 09:49:21.086857686 +0000 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-08-19 09:46:54.777937626 +0000 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-08-19 09:46:54.777937626 +0000 2024-08-19T09:49:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-08-19 09:46:54.777937626 +0000 2024-08-19T09:49:28.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T09:49:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-08-19T09:49:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-08-19T09:49:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.00024205 s, 2.1 MB/s 2024-08-19T09:49:28.259 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T09:49:28.316 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:28.316 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-08-19T09:49:28.316 INFO:tasks.ceph:Generating config... 2024-08-19T09:49:28.317 INFO:tasks.ceph:[client] debug monc = 20 2024-08-19T09:49:28.317 INFO:tasks.ceph:[client] debug ms = 1 2024-08-19T09:49:28.317 INFO:tasks.ceph:[global] debug ms = 1 2024-08-19T09:49:28.317 INFO:tasks.ceph:[global] debug objecter = 20 2024-08-19T09:49:28.317 INFO:tasks.ceph:[global] debug rados = 20 2024-08-19T09:49:28.317 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-08-19T09:49:28.317 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-08-19T09:49:28.318 INFO:tasks.ceph:[global] ms type = async 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mgr] debug monc = 10 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-19T09:49:28.318 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] paxos min = 10 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-08-19T09:49:28.319 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-08-19T09:49:28.319 INFO:tasks.ceph:[osd] bdev async discard = True 2024-08-19T09:49:28.319 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-08-19T09:49:28.319 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-19T09:49:28.319 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-19T09:49:28.319 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-08-19T09:49:28.319 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] osd class default list = * 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] osd class load list = * 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-19T09:49:28.320 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-08-19T09:49:28.321 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-08-19T09:49:28.321 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-08-19T09:49:28.321 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-19T09:49:28.321 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-08-19T09:49:28.321 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-08-19T09:49:28.321 INFO:tasks.ceph:Setting up mon.a... 2024-08-19T09:49:28.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:28.378 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-08-19T09:49:28.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-19T09:49:28.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-19T09:49:28.451 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.1:6789]'), ('mon.b', '[v1:172.21.15.1:6790]'), ('mon.c', '[v1:172.21.15.1:6791]'), ('mon.d', '[v1:172.21.15.1:6792]'), ('mon.e', '[v1:172.21.15.1:6793]'), ('mon.f', '[v1:172.21.15.82:6789]'), ('mon.g', '[v1:172.21.15.82:6790]'), ('mon.h', '[v1:172.21.15.82:6791]'), ('mon.i', '[v1:172.21.15.82:6792]')] 2024-08-19T09:49:28.452 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.1:6789],[v1:172.21.15.1:6790],[v1:172.21.15.1:6791],[v1:172.21.15.1:6792],[v1:172.21.15.1:6793],[v1:172.21.15.82:6789],[v1:172.21.15.82:6790],[v1:172.21.15.82:6791],[v1:172.21.15.82:6792]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-08-19T09:49:28.452 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:28.453 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-19T09:49:28.508 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.1:6789]' --addv b '[v1:172.21.15.1:6790]' --addv c '[v1:172.21.15.1:6791]' --addv d '[v1:172.21.15.1:6792]' --addv e '[v1:172.21.15.1:6793]' --addv f '[v1:172.21.15.82:6789]' --addv g '[v1:172.21.15.82:6790]' --addv h '[v1:172.21.15.82:6791]' --addv i '[v1:172.21.15.82:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:28.615 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:28.615 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:1: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:2: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:3: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:4: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:5: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:28.616 INFO:teuthology.orchestra.run.smithi001.stdout:6: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:28.617 INFO:teuthology.orchestra.run.smithi001.stdout:7: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:28.617 INFO:teuthology.orchestra.run.smithi001.stdout:8: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:28.617 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-08-19T09:49:28.618 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-19T09:49:28.632 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0e9f827b-9129-41b7-89ad-24f744d49b77... 2024-08-19T09:49:28.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-19T09:49:28.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-19T09:49:28.714 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-08-19T09:49:28.714 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-08-19T09:49:28.715 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-08-19T09:49:28.716 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.717 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.1:6790],[v1:172.21.15.1:6791],[v1:172.21.15.1:6792],[v1:172.21.15.1:6793],[v1:172.21.15.82:6789],[v1:172.21.15.82:6790],[v1:172.21.15.82:6791],[v1:172.21.15.82:6792]" 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: debug objecter = 20 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: debug rados = 20 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: mon election default strategy = 1 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: ms bind msgr2 = False 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject delay max = 1 2024-08-19T09:49:28.718 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject delay probability = 0.005 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject delay type = mon 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject internal delays = 0.002 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 2500 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: ms type = async 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-08-19T09:49:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-08-19T09:49:28.721 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore write v2 = False 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore zero block detection = True 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 20 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 20 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 10 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: osd class default list = * 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: osd class load list = * 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify cached snaps = True 2024-08-19T09:49:28.722 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify missing on start = True 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock override recovery settings = True 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd memory target = 1610612736 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue = debug_random 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue cut off = debug_random 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-08-19T09:49:28.723 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: debug monc = 10 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-08-19T09:49:28.724 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-08-19T09:49:28.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon min osdmap epochs = 25 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune interval = 2 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune min = 15 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune txsize = 2 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon scrub inject crc mismatch = 0.01 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon scrub inject missing keys = 0.05 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon scrub interval = 300 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on pool no app = False 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: paxos min = 10 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: paxos service trim min = 5 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: paxos trim min = 10 2024-08-19T09:49:28.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug monc = 20 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-08-19T09:49:28.727 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.d] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.e] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.f] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.g] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.h] 2024-08-19T09:49:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.i] 2024-08-19T09:49:28.729 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-19T09:49:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.1:6790],[v1:172.21.15.1:6791],[v1:172.21.15.1:6792],[v1:172.21.15.1:6793],[v1:172.21.15.82:6789],[v1:172.21.15.82:6790],[v1:172.21.15.82:6791],[v1:172.21.15.82:6792]" 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: debug objecter = 20 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: debug rados = 20 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: mon client directed command retry = 5 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: mon election default strategy = 1 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: ms bind msgr2 = False 2024-08-19T09:49:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject delay max = 1 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject delay probability = 0.005 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject delay type = mon 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject internal delays = 0.002 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject socket failures = 2500 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: ms type = async 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2024-08-19T09:49:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug trim objects = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bdev async discard = True 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bdev enable discard = True 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore block size = 96636764160 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore fsck on mount = True 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore write v2 = False 2024-08-19T09:49:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore zero block detection = True 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluefs = 20 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluestore = 20 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: debug rocksdb = 10 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd backfillfull_ratio = 0.85 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd full ratio = 0.9 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd nearfull ratio = 0.8 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: osd class default list = * 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: osd class load list = * 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify cached snaps = True 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify missing on start = True 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: osd failsafe full ratio = 0.95 2024-08-19T09:49:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock override recovery settings = True 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: osd memory target = 1610612736 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: osd objectstore = bluestore 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue = debug_random 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue cut off = debug_random 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-08-19T09:49:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 10 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2024-08-19T09:49:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2024-08-19T09:49:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: mon min osdmap epochs = 25 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune interval = 2 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune min = 15 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune txsize = 2 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub inject crc mismatch = 0.01 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub inject missing keys = 0.05 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub interval = 300 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on pool no app = False 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: paxos min = 10 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: paxos service trim min = 5 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: paxos trim min = 10 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2024-08-19T09:49:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 20 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.b] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.c] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.d] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.e] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.f] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.g] 2024-08-19T09:49:28.742 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.h] 2024-08-19T09:49:28.743 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.i] 2024-08-19T09:49:28.743 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-19T09:49:28.743 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-19T09:49:28.824 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-19T09:49:28.824 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:28.824 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-19T09:49:28.839 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:28.839 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-19T09:49:28.895 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T09:49:28.895 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:28.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-19T09:49:28.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-19T09:49:28.968 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:28.969 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:29.024 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T09:49:29.024 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:29.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-19T09:49:29.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-19T09:49:29.060 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:29.060 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:29.115 INFO:tasks.ceph:Setting up mon nodes... 2024-08-19T09:49:29.115 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-19T09:49:29.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-19T09:49:29.224 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-19T09:49:29.227 INFO:tasks.ceph:Setting up mds nodes... 2024-08-19T09:49:29.227 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-19T09:49:29.227 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-19T09:49:29.267 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-19T09:49:29.280 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-19T09:49:29.280 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-08-19T09:49:29.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-19T09:49:29.305 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:29.305 INFO:tasks.ceph:role: osd.0 2024-08-19T09:49:29.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T09:49:29.305 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-19T09:49:29.370 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T09:49:29.370 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T09:49:29.371 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T09:49:29.403 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-08-19T09:49:29.405 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-08-19T09:49:29.405 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-19T09:49:29.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-19T09:49:29.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-19T09:49:29.607 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:29.608 INFO:tasks.ceph:role: osd.1 2024-08-19T09:49:29.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T09:49:29.608 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-19T09:49:29.672 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T09:49:29.672 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T09:49:29.673 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T09:49:29.705 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-08-19T09:49:29.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-08-19T09:49:29.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-19T09:49:29.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-19T09:49:29.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-19T09:49:29.909 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:29.910 INFO:tasks.ceph:role: osd.2 2024-08-19T09:49:29.910 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T09:49:29.910 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-19T09:49:29.974 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T09:49:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T09:49:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T09:49:29.975 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T09:49:30.007 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-08-19T09:49:30.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-08-19T09:49:30.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-19T09:49:30.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-19T09:49:30.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:30.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:30.240+0000 7fd908dcb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-08-19T09:49:30.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:30.240+0000 7fd908dcb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-08-19T09:49:30.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:30.240+0000 7fd908dcb640 -1 bdev(0x55674432d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-08-19T09:49:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:30.240+0000 7fd908dcb640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-08-19T09:49:33.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-19T09:49:33.186 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:33.276+0000 7f6ba04ec640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-08-19T09:49:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:33.276+0000 7f6ba04ec640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-08-19T09:49:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:33.276+0000 7f6ba04ec640 -1 bdev(0x5584a097b800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-08-19T09:49:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:33.276+0000 7f6ba04ec640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-08-19T09:49:36.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-19T09:49:36.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:36.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:36.302+0000 7fcfce2df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-08-19T09:49:36.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:36.302+0000 7fcfce2df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-08-19T09:49:36.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:36.302+0000 7fcfce2df640 -1 bdev(0x56307827b800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-08-19T09:49:36.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:36.302+0000 7fcfce2df640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-08-19T09:49:39.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-19T09:49:39.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-19T09:49:39.281 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:39.282 INFO:tasks.ceph:role: osd.3 2024-08-19T09:49:39.282 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T09:49:39.282 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-19T09:49:39.357 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T09:49:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T09:49:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T09:49:39.358 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T09:49:39.866 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-08-19T09:49:39.868 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-08-19T09:49:39.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-08-19T09:49:39.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-19T09:49:40.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-08-19T09:49:40.082 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:40.082 INFO:tasks.ceph:role: osd.4 2024-08-19T09:49:40.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T09:49:40.082 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-19T09:49:40.156 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T09:49:40.156 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T09:49:40.156 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T09:49:40.156 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T09:49:40.157 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T09:49:40.157 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-08-19T09:49:40.157 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T09:49:40.157 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T09:49:40.157 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T09:49:40.157 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T09:49:40.624 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-08-19T09:49:40.625 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-08-19T09:49:40.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-08-19T09:49:40.719 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-08-19T09:49:40.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-08-19T09:49:40.812 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T09:49:40.812 INFO:tasks.ceph:role: osd.5 2024-08-19T09:49:40.813 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T09:49:40.813 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T09:49:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T09:49:40.886 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T09:49:40.886 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T09:49:41.372 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-08-19T09:49:41.374 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-08-19T09:49:41.374 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-08-19T09:49:41.516 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-08-19T09:49:41.545 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:41.639 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:41.638+0000 7f2fb7817640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-08-19T09:49:41.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:41.643+0000 7f2fb7817640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-08-19T09:49:41.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:41.644+0000 7f2fb7817640 -1 bdev(0x557473515800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-08-19T09:49:41.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:41.644+0000 7f2fb7817640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-08-19T09:49:44.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-19T09:49:44.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:44.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:44.677+0000 7fef24a86640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-08-19T09:49:44.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:44.677+0000 7fef24a86640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-08-19T09:49:44.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:44.677+0000 7fef24a86640 -1 bdev(0x55a0b79f7800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-08-19T09:49:44.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:44.677+0000 7fef24a86640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-08-19T09:49:47.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-08-19T09:49:47.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:47.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:47.713+0000 7fc41ffbc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-08-19T09:49:47.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:47.714+0000 7fc41ffbc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-08-19T09:49:47.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:47.714+0000 7fc41ffbc640 -1 bdev(0x5649f8bfb800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-08-19T09:49:47.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:47.714+0000 7fc41ffbc640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-08-19T09:49:50.886 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-19T09:49:50.913 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-19T09:49:50.914 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:50.914 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-19T09:49:50.944 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:50.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-19T09:49:51.013 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:51.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-19T09:49:51.080 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:51.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-19T09:49:51.108 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:51.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-19T09:49:51.174 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:51.175 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-08-19T09:49:51.241 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:51.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-08-19T09:49:51.306 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:51.306 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-19T09:49:51.363 INFO:tasks.ceph:Adding keys to all mons... 2024-08-19T09:49:51.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-19T09:49:51.365 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-19T09:49:51.406 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-08-19T09:49:51.406 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAqFcNmxupkDhAAoJETXu5wu2fdPdkl6zHxpA== 2024-08-19T09:49:51.406 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAtFcNmh1eDEBAAzlE+GF0qkubxXXDmTAAAww== 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAwFcNmW5sXEhAAtxT2cLedUDWNc28thweKaA== 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQApFcNm9AJhDRAA5RIa+zLcjoKnEoUq0DqAwA== 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQA1FcNmst8bJhAAoZrdOOG8fnfAZ7jY8fQk3g== 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQA4FcNmWptvKBAACNCedh3isumR6PehLOrm9w== 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQA7FcNmfVydKhAAEwqo7fFYNrWjhdAJ3Ar1tA== 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-08-19T09:49:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQApFcNmqoD3DxAAR01Sa8zsQRjAcSi8S6wKuA== 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAqFcNmxupkDhAAoJETXu5wu2fdPdkl6zHxpA== 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAtFcNmh1eDEBAAzlE+GF0qkubxXXDmTAAAww== 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAwFcNmW5sXEhAAtxT2cLedUDWNc28thweKaA== 2024-08-19T09:49:51.426 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQApFcNm9AJhDRAA5RIa+zLcjoKnEoUq0DqAwA== 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQA1FcNmst8bJhAAoZrdOOG8fnfAZ7jY8fQk3g== 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQA4FcNmWptvKBAACNCedh3isumR6PehLOrm9w== 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQA7FcNmfVydKhAAEwqo7fFYNrWjhdAJ3Ar1tA== 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-08-19T09:49:51.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQApFcNmqoD3DxAAR01Sa8zsQRjAcSi8S6wKuA== 2024-08-19T09:49:51.428 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.575 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-19T09:49:51.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-19T09:49:51.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.731 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.791 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T09:49:51.896 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-19T09:49:51.899 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-19T09:49:51.952 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-19T09:49:51.953 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-19T09:49:51.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:52.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-19T09:49:52.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-08-19T09:49:52.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:52.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-08-19T09:49:52.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-08-19T09:49:52.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:52.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-08-19T09:49:52.465 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-08-19T09:49:52.530 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:52.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-08-19T09:49:52.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-08-19T09:49:52.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:52.809 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-08-19T09:49:52.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-08-19T09:49:52.864 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:52.967 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-08-19T09:49:52.994 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-08-19T09:49:53.059 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:53.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-08-19T09:49:53.182 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-08-19T09:49:53.246 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:53.343 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-08-19T09:49:53.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-08-19T09:49:53.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T09:49:53.528 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-08-19T09:49:53.553 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:53.556 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-19T09:49:53.608 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-19T09:49:53.609 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T09:49:53.609 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T09:49:53.612 INFO:tasks.ceph.mon.a:Started 2024-08-19T09:49:53.612 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-19T09:49:53.612 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-19T09:49:53.614 INFO:tasks.ceph.mon.b:Started 2024-08-19T09:49:53.615 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-19T09:49:53.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-19T09:49:53.617 INFO:tasks.ceph.mon.c:Started 2024-08-19T09:49:53.617 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T09:49:53.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T09:49:53.619 INFO:tasks.ceph.mon.d:Started 2024-08-19T09:49:53.619 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T09:49:53.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T09:49:53.621 INFO:tasks.ceph.mon.e:Started 2024-08-19T09:49:53.622 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T09:49:53.622 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T09:49:53.651 INFO:tasks.ceph.mon.f:Started 2024-08-19T09:49:53.651 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T09:49:53.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T09:49:53.653 INFO:tasks.ceph.mon.g:Started 2024-08-19T09:49:53.653 INFO:tasks.ceph.mon.h:Restarting daemon 2024-08-19T09:49:53.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-08-19T09:49:53.655 INFO:tasks.ceph.mon.h:Started 2024-08-19T09:49:53.656 INFO:tasks.ceph.mon.i:Restarting daemon 2024-08-19T09:49:53.656 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-08-19T09:49:53.658 INFO:tasks.ceph.mon.i:Started 2024-08-19T09:49:53.658 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-19T09:49:53.658 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-19T09:49:53.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-19T09:49:53.660 INFO:tasks.ceph.mgr.x:Started 2024-08-19T09:49:53.660 DEBUG:tasks.ceph:set 0 configs 2024-08-19T09:49:53.661 INFO:tasks.ceph:Setting crush tunables to default 2024-08-19T09:49:53.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.792+0000 7fdfffc15640 10 monclient: get_monmap_and_config 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.792+0000 7fdfffc15640 10 monclient: build_initial_monmap 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.792+0000 7fdfffc15640 10 monclient: monmap: 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:53.793043+0000 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:53.793043+0000 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:53.794 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.792+0000 7fdfffc15640 1 Processor -- start 2024-08-19T09:49:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.792+0000 7fdfffc15640 1 -- start start 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.792+0000 7fdfffc15640 10 monclient: init 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient: _add_conns ranks=[1,6,8,3,4,2,7,0,5] 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient(hunting): picked mon.noname-f con 0x7fdff80a1a90 addr v1:172.21.15.82:6789/0 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient(hunting): picked mon.noname-d con 0x7fdff80a1e70 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient(hunting): picked mon.noname-e con 0x7fdff80a2280 addr v1:172.21.15.1:6793/0 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff809f6e0 con 0x7fdff80a1a90 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff809fb90 con 0x7fdff80a1e70 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff809fe70 con 0x7fdff80a2280 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdfffc15640 10 monclient(hunting): authenticate will time out at 649.869478s 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdffd1b0640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdff80a1e70 0x7fdff813b250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:55868/0 (socket says 172.21.15.1:55868) 2024-08-19T09:49:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdffe1b2640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fdff80a2280 0x7fdff813e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43002/0 (socket says 172.21.15.1:43002) 2024-08-19T09:49:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.793+0000 7fdffd1b0640 1 -- 172.21.15.1:0/1221604391 learned_addr learned my addr 172.21.15.1:0/1221604391 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:49:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.795+0000 7fdffd1b0640 1 -- 172.21.15.1:0/1221604391 >> v1:172.21.15.1:6792/0 conn(0x7fdff80a1e70 legacy=0x7fdff813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.795+0000 7fdffe1b2640 1 -- 172.21.15.1:0/1221604391 >> v1:172.21.15.1:6793/0 conn(0x7fdff80a2280 legacy=0x7fdff813e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:53.795+0000 7fdffd9b1640 1 -- 172.21.15.1:0/1221604391 >> v1:172.21.15.82:6789/0 conn(0x7fdff80a1a90 legacy=0x7fdff80a3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:55.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430461635 0 0) 0x7fdff809f6e0 con 0x7fdff80a1a90 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 10 monclient(hunting): _negotiate 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 10 monclient(hunting): my global_id is 4105 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfe0003520 con 0x7fdff80a1a90 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.754+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1241962686 0 0) 0x7fdff809fe70 con 0x7fdff80a2280 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): _negotiate 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:55.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): my global_id is 4112 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdff809f6e0 con 0x7fdff80a2280 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428499548 0 0) 0x7fdff809fb90 con 0x7fdff80a1e70 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): _negotiate 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient(hunting): my global_id is 4110 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdff809fe70 con 0x7fdff80a1e70 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3474274028 0 0) 0x7fdfe0003520 con 0x7fdff80a1a90 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 >> v1:172.21.15.1:6793/0 conn(0x7fdff80a2280 legacy=0x7fdff813e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 >> v1:172.21.15.1:6792/0 conn(0x7fdff80a1e70 legacy=0x7fdff813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: _finish_hunting 0 2024-08-19T09:49:55.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 monclient: found mon.noname-f 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff809ee60 con 0x7fdff80a1a90 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: _finish_auth 0 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: _check_auth_tickets 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdff0001b30 con 0x7fdff80a1a90 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: dump: 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:55.760 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdfffc15640 5 monclient: authenticate success, global_id 4105 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdfffc15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdff0003610 con 0x7fdff80a1a90 2024-08-19T09:49:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 1 -- 172.21.15.1:0/1221604391 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdff0003b00 con 0x7fdff80a1a90 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdffc9af640 10 monclient: dump: 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:55.763 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdfffc15640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.755+0000 7fdfffc15640 4 set_mon_vals no callback set 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.756+0000 7fdfffc15640 10 monclient: shutdown 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.756+0000 7fdfffc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.756+0000 7fdfffc15640 1 -- 172.21.15.1:0/1221604391 >> v1:172.21.15.82:6789/0 conn(0x7fdff80a1a90 legacy=0x7fdff80a3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.756+0000 7fdfffc15640 1 -- 172.21.15.1:0/1221604391 shutdown_connections 2024-08-19T09:49:55.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.756+0000 7fdfffc15640 1 -- 172.21.15.1:0/1221604391 >> 172.21.15.1:0/1221604391 conn(0x7fdff80a8430 msgr2=0x7fdff80a8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:55.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.757+0000 7fdfffc15640 1 -- 172.21.15.1:0/1221604391 shutdown_connections 2024-08-19T09:49:55.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.757+0000 7fdfffc15640 1 -- 172.21.15.1:0/1221604391 wait complete. 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.758+0000 7fdfffc15640 10 monclient: build_initial_monmap 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.758+0000 7fdfffc15640 10 monclient: monmap: 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:55.769 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.758+0000 7fdfffc15640 1 librados: starting msgr at 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.758+0000 7fdfffc15640 1 librados: starting objecter 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.758+0000 7fdfffc15640 1 Processor -- start 2024-08-19T09:49:55.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 1 -- start start 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 1 librados: setting wanted keys 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 1 librados: calling monclient init 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 10 monclient: init 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 10 monclient: _add_conns ranks=[0,2,6,3,8,4,7,1,5] 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 10 monclient(hunting): picked mon.noname-a con 0x7fdff80a1a90 addr v1:172.21.15.1:6789/0 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 10 monclient(hunting): picked mon.noname-b con 0x7fdff80a1e70 addr v1:172.21.15.1:6790/0 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.759+0000 7fdfffc15640 10 monclient(hunting): picked mon.noname-d con 0x7fdff80a2280 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdfffc15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff8076e40 con 0x7fdff80a1a90 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdfffc15640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff81241c0 con 0x7fdff80a1e70 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdfffc15640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff8128010 con 0x7fdff80a2280 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdfffc15640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdfffc15640 10 monclient(hunting): authenticate will time out at 651.836056s 2024-08-19T09:49:55.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdffe1b2640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdff80a2280 0x7fdff81d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:55888/0 (socket says 172.21.15.1:55888) 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdffd9b1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdff80a1a90 0x7fdff8142290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33814/0 (socket says 172.21.15.1:33814) 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdffd1b0640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fdff80a1e70 0x7fdff81d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33794/0 (socket says 172.21.15.1:33794) 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.760+0000 7fdffe1b2640 1 -- 172.21.15.1:0/4032862276 learned_addr learned my addr 172.21.15.1:0/4032862276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdffe1b2640 1 -- 172.21.15.1:0/4032862276 >> v1:172.21.15.1:6792/0 conn(0x7fdff80a2280 legacy=0x7fdff81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdffd9b1640 1 -- 172.21.15.1:0/4032862276 >> v1:172.21.15.1:6789/0 conn(0x7fdff80a1a90 legacy=0x7fdff8142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 client.?.objecter ms_handle_connect 0x7fdff80a2280 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 client.?.objecter ms_handle_connect 0x7fdff80a1e70 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 client.?.objecter ms_handle_connect 0x7fdff80a1a90 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdffd1b0640 1 -- 172.21.15.1:0/4032862276 >> v1:172.21.15.1:6790/0 conn(0x7fdff80a1e70 legacy=0x7fdff81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198703495 0 0) 0x7fdff809ee60 con 0x7fdff80a1a90 2024-08-19T09:49:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 monclient(hunting): my global_id is 4104 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfe4004eb0 con 0x7fdff80a1a90 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 341455035 0 0) 0x7fdfe4004eb0 con 0x7fdff80a1a90 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdff809ee60 con 0x7fdff80a1a90 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdff0003b00 con 0x7fdff80a1a90 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.762+0000 7fdff67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient(hunting): dump: 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2016809224 0 0) 0x7fdff809ee60 con 0x7fdff80a1a90 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 >> v1:172.21.15.1:6792/0 conn(0x7fdff80a2280 legacy=0x7fdff81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 >> v1:172.21.15.1:6790/0 conn(0x7fdff80a1e70 legacy=0x7fdff81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: _finish_hunting 0 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 monclient: found mon.a 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff8123ee0 con 0x7fdff80a1a90 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: _finish_auth 0 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: _check_auth_tickets 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 5 monclient: authenticate success, global_id 4104 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 10 monclient: _renew_subs 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:49:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdff81235e0 con 0x7fdff80a1a90 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 10 client.4104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 10 monclient: _renew_subs 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdff81442c0 con 0x7fdff80a1a90 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdfffc15640 1 librados: init done 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdff0003610 con 0x7fdff80a1a90 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdff00015d0 con 0x7fdff80a1a90 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff7fff640 4 set_mon_vals no callback set 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 monclient: dump: 2024-08-19T09:49:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdff0005b50 con 0x7fdff80a1a90 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3010313334 0 0) 0x7fdff0006da0 con 0x7fdff80a1a90 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 10 client.4104.objecter ms_dispatch 0x7fdff8000b90 osd_map(1..1 src has 1..1) 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 3 client.4104.objecter handle_osd_map got epochs [1,1] > 0 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 3 client.4104.objecter handle_osd_map decoding full epoch 1 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.763+0000 7fdff67fc640 20 client.4104.objecter dump_active .. 0 homeless 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.764+0000 7fdfffc15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.764+0000 7fdfffc15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.764+0000 7fdfffc15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.764+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd8005080 con 0x7fdff80a1a90 2024-08-19T09:49:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.766+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fdff8076e40 con 0x7fdff80a1a90 2024-08-19T09:49:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.766+0000 7fdff67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.766+0000 7fdff67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.834+0000 7fdfffc15640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-19T09:49:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.834+0000 7fdfffc15640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-19T09:49:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.834+0000 7fdfffc15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:49:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:55.834+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fdfd8005370 con 0x7fdff80a1a90 2024-08-19T09:49:56.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.293+0000 7fdff67fc640 1 -- 172.21.15.1:0/4032862276 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7fdff0005e40 con 0x7fdff80a1a90 2024-08-19T09:49:56.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.293+0000 7fdff67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-19T09:49:56.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.293+0000 7fdff67fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-08-19T09:49:56.294 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-08-19T09:49:56.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.294+0000 7fdfffc15640 10 librados: watch_flush enter 2024-08-19T09:49:56.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 10 librados: watch_flush exit 2024-08-19T09:49:56.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 20 client.4104.objecter shutdown clearing up homeless session... 2024-08-19T09:49:56.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 10 client.4104.objecter successfully canceled tick 2024-08-19T09:49:56.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 10 monclient: shutdown 2024-08-19T09:49:56.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 >> v1:172.21.15.1:6789/0 conn(0x7fdff80a1a90 legacy=0x7fdff8142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 shutdown_connections 2024-08-19T09:49:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 >> 172.21.15.1:0/4032862276 conn(0x7fdff80a8430 msgr2=0x7fdff8133250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 shutdown_connections 2024-08-19T09:49:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 1 -- 172.21.15.1:0/4032862276 wait complete. 2024-08-19T09:49:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:56.295+0000 7fdfffc15640 1 librados: shutdown 2024-08-19T09:49:56.309 INFO:tasks.ceph:check_enable_crimson: False 2024-08-19T09:49:56.309 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-19T09:49:56.309 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:56.309 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-19T09:49:56.336 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:56.336 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-19T09:49:56.404 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-08-19T09:49:56.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-19T09:49:56.471 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:56.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-19T09:49:56.496 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:56.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-08-19T09:49:56.562 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:49:56.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-08-19T09:49:56.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 70148d02-f1a0-47ce-987a-5539fce9ac64 0 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.766+0000 7f77b29aa640 10 monclient: get_monmap_and_config 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.766+0000 7f77b29aa640 10 monclient: build_initial_monmap 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.766+0000 7f77b29aa640 10 monclient: monmap: 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:56.767076+0000 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:56.767076+0000 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:56.767 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:56.768 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.766+0000 7f77b29aa640 1 Processor -- start 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.766+0000 7f77b29aa640 1 -- start start 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.766+0000 7f77b29aa640 10 monclient: init 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient: _add_conns ranks=[5,0,6,3,7,1,2,8,4] 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient(hunting): picked mon.noname-h con 0x7f77ac0a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient(hunting): picked mon.noname-a con 0x7f77ac0694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient(hunting): picked mon.noname-d con 0x7f77ac0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac0a3c70 con 0x7f77ac0694b0 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac0a33d0 con 0x7f77ac0a6830 2024-08-19T09:49:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac0a36b0 con 0x7f77ac0a46a0 2024-08-19T09:49:56.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:56.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77b29aa640 10 monclient(hunting): authenticate will time out at 647.795344s 2024-08-19T09:49:56.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77abfff640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f77ac0a6830 0x7f77ac0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:34476/0 (socket says 172.21.15.82:34476) 2024-08-19T09:49:56.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.767+0000 7f77abfff640 1 -- 172.21.15.82:0/1242430316 learned_addr learned my addr 172.21.15.82:0/1242430316 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:56.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77abfff640 1 -- 172.21.15.82:0/1242430316 >> v1:172.21.15.82:6791/0 conn(0x7f77ac0a6830 legacy=0x7f77ac0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:56.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77b0f47640 1 -- 172.21.15.82:0/1242430316 >> v1:172.21.15.1:6792/0 conn(0x7f77ac0a46a0 legacy=0x7f77ac141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:56.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77ab7fe640 1 -- 172.21.15.82:0/1242430316 >> v1:172.21.15.1:6789/0 conn(0x7f77ac0694b0 legacy=0x7f77ac0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690150093 0 0) 0x7f77ac0a3c70 con 0x7f77ac0694b0 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77aaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77aaffd640 10 monclient(hunting): my global_id is 4113 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.769+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7798003520 con 0x7f77ac0694b0 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1780815356 0 0) 0x7f7798003520 con 0x7f77ac0694b0 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 >> v1:172.21.15.1:6792/0 conn(0x7f77ac0a46a0 legacy=0x7f77ac141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 >> v1:172.21.15.82:6791/0 conn(0x7f77ac0a6830 legacy=0x7f77ac0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 monclient: found mon.noname-a 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ac06dbb0 con 0x7f77ac0694b0 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: _finish_auth 0 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f779c001bf0 con 0x7f77ac0694b0 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: dump: 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:56.777 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:56.778 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 5 monclient: authenticate success, global_id 4113 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f779c003610 con 0x7f77ac0694b0 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 1 -- 172.21.15.82:0/1242430316 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f779c003b40 con 0x7f77ac0694b0 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77aaffd640 10 monclient: dump: 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:56.779 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:56.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 4 set_mon_vals no callback set 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 10 monclient: shutdown 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 1 -- 172.21.15.82:0/1242430316 >> v1:172.21.15.1:6789/0 conn(0x7f77ac0694b0 legacy=0x7f77ac0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 1 -- 172.21.15.82:0/1242430316 shutdown_connections 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 1 -- 172.21.15.82:0/1242430316 >> 172.21.15.82:0/1242430316 conn(0x7f77ac0a9020 msgr2=0x7f77ac068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 1 -- 172.21.15.82:0/1242430316 shutdown_connections 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.770+0000 7f77b29aa640 1 -- 172.21.15.82:0/1242430316 wait complete. 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.771+0000 7f77b29aa640 10 monclient: build_initial_monmap 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.771+0000 7f77b29aa640 10 monclient: monmap: 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:56.781 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:56.782 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.771+0000 7f77b29aa640 1 librados: starting msgr at 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.771+0000 7f77b29aa640 1 librados: starting objecter 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.771+0000 7f77b29aa640 1 Processor -- start 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 1 -- start start 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 1 librados: setting wanted keys 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 1 librados: calling monclient init 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient: init 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient: _add_conns ranks=[3,7,2,0,1,4,5,6,8] 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient(hunting): picked mon.noname-g con 0x7f77ac0694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient(hunting): picked mon.noname-i con 0x7f77ac0a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient(hunting): picked mon.noname-b con 0x7f77ac0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:49:56.783 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac138d30 con 0x7f77ac0a6830 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac139010 con 0x7f77ac0694b0 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac142570 con 0x7f77ac0a46a0 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77b29aa640 10 monclient(hunting): authenticate will time out at 647.800545s 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77abfff640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f77ac0694b0 0x7f77ac1307d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.82:39224/0 (socket says 172.21.15.82:39224) 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77ab7fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f77ac0a46a0 0x7f77ac1d3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.82:53240/0 (socket says 172.21.15.82:53240) 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.772+0000 7f77abfff640 1 -- 172.21.15.82:0/443400272 learned_addr learned my addr 172.21.15.82:0/443400272 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.774+0000 7f77abfff640 1 -- 172.21.15.82:0/443400272 >> v1:172.21.15.82:6790/0 conn(0x7f77ac0694b0 legacy=0x7f77ac1307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.774+0000 7f77ab7fe640 1 -- 172.21.15.82:0/443400272 >> v1:172.21.15.82:6792/0 conn(0x7f77ac0a46a0 legacy=0x7f77ac1d3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.774+0000 7f77a8ff9640 10 client.?.objecter ms_handle_connect 0x7f77ac0694b0 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.774+0000 7f77a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.774+0000 7f77a8ff9640 10 client.?.objecter ms_handle_connect 0x7f77ac0a46a0 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.774+0000 7f77a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:56.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.775+0000 7f77a8ff9640 10 client.?.objecter ms_handle_connect 0x7f77ac0a6830 2024-08-19T09:49:56.785 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.775+0000 7f77b0f47640 1 -- 172.21.15.82:0/443400272 >> v1:172.21.15.1:6790/0 conn(0x7f77ac0a6830 legacy=0x7f77ac1d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:56.785 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:56.775+0000 7f77a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403396785 0 0) 0x7f77ac142570 con 0x7f77ac0a46a0 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): my global_id is 4120 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f778c004eb0 con 0x7f77ac0a46a0 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432352119 0 0) 0x7f77ac139010 con 0x7f77ac0694b0 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 10 monclient(hunting): my global_id is 4125 2024-08-19T09:49:57.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.212+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77ac142570 con 0x7f77ac0694b0 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138011972 0 0) 0x7f77ac138d30 con 0x7f77ac0a6830 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): my global_id is 4115 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77ac139010 con 0x7f77ac0a6830 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1149454701 0 0) 0x7f778c004eb0 con 0x7f77ac0a46a0 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77ac138d30 con 0x7f77ac0a46a0 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f779c001bb0 con 0x7f77ac0a46a0 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): dump: 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.218 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.219 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4006370298 0 0) 0x7f77ac142570 con 0x7f77ac0694b0 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f778c004eb0 con 0x7f77ac0694b0 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7794001d00 con 0x7f77ac0694b0 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): dump: 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.220 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2357319139 0 0) 0x7f77ac139010 con 0x7f77ac0a6830 2024-08-19T09:49:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77ac142570 con 0x7f77ac0a6830 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77a4002050 con 0x7f77ac0a6830 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient(hunting): dump: 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.222 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 982335653 0 0) 0x7f778c004eb0 con 0x7f77ac0694b0 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 >> v1:172.21.15.82:6792/0 conn(0x7f77ac0a46a0 legacy=0x7f77ac1d3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 >> v1:172.21.15.1:6790/0 conn(0x7f77ac0a6830 legacy=0x7f77ac1d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 monclient: found mon.g 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ac136cb0 con 0x7f77ac0694b0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 5 monclient: authenticate success, global_id 4125 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 10 monclient: _renew_subs 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77ac138770 con 0x7f77ac0694b0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7794004a90 con 0x7f77ac0694b0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 10 monclient: _renew_subs 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:49:57.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f77ac138440 con 0x7f77ac0694b0 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77b29aa640 1 librados: init done 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7794004f80 con 0x7f77ac0694b0 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 10 monclient: dump: 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.225 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77aa7fc640 4 set_mon_vals no callback set 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.213+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7794005890 con 0x7f77ac0694b0 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.214+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 3536452576 0 0) 0x7f7794006cd0 con 0x7f77ac0694b0 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.214+0000 7f77a8ff9640 10 client.4125.objecter ms_dispatch 0x7f77ac000b90 osd_map(2..2 src has 1..2) 2024-08-19T09:49:57.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.214+0000 7f77a8ff9640 3 client.4125.objecter handle_osd_map got epochs [2,2] > 0 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.214+0000 7f77a8ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 2 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.214+0000 7f77a8ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.215+0000 7f77b29aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.215+0000 7f77b29aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.215+0000 7f77b29aa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.215+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77ac06be10 con 0x7f77ac0694b0 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.217+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f77ac06be10 con 0x7f77ac0694b0 2024-08-19T09:49:57.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.217+0000 7f77a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.217+0000 7f77a8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:57.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.307+0000 7f77b29aa640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "70148d02-f1a0-47ce-987a-5539fce9ac64", "id": 0}] 2024-08-19T09:49:57.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.307+0000 7f77b29aa640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "70148d02-f1a0-47ce-987a-5539fce9ac64", "id": 0}] 2024-08-19T09:49:57.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.307+0000 7f77b29aa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:49:57.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.307+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "osd new", "uuid": "70148d02-f1a0-47ce-987a-5539fce9ac64", "id": 0} v 0) -- 0x7f77ac051b00 con 0x7f77ac0694b0 2024-08-19T09:49:57.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.311+0000 7f77a8ff9640 1 -- 172.21.15.82:0/443400272 <== mon.3 v1:172.21.15.82:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "70148d02-f1a0-47ce-987a-5539fce9ac64", "id": 0}]=0 v3) ==== 112+0+2 (unknown 1586632787 0 732707515) 0x7f7794005d50 con 0x7f77ac0694b0 2024-08-19T09:49:57.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.311+0000 7f77a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "70148d02-f1a0-47ce-987a-5539fce9ac64", "id": 0}] 2024-08-19T09:49:57.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.311+0000 7f77a8ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:57.313 INFO:teuthology.orchestra.run.smithi082.stdout:0 2024-08-19T09:49:57.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 10 librados: watch_flush enter 2024-08-19T09:49:57.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 10 librados: watch_flush exit 2024-08-19T09:49:57.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 20 client.4125.objecter shutdown clearing up homeless session... 2024-08-19T09:49:57.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 10 client.4125.objecter successfully canceled tick 2024-08-19T09:49:57.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 10 monclient: shutdown 2024-08-19T09:49:57.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 >> v1:172.21.15.82:6790/0 conn(0x7f77ac0694b0 legacy=0x7f77ac1307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 shutdown_connections 2024-08-19T09:49:57.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 >> 172.21.15.82:0/443400272 conn(0x7f77ac0a9020 msgr2=0x7f77ac0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 shutdown_connections 2024-08-19T09:49:57.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 1 -- 172.21.15.82:0/443400272 wait complete. 2024-08-19T09:49:57.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.312+0000 7f77b29aa640 1 librados: shutdown 2024-08-19T09:49:57.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 066ec263-41d5-4d6c-a306-7cb11e1834ed 1 2024-08-19T09:49:57.420 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.419+0000 7f77385f2640 10 monclient: get_monmap_and_config 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.419+0000 7f77385f2640 10 monclient: build_initial_monmap 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.419+0000 7f77385f2640 10 monclient: monmap: 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:57.420881+0000 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:57.420881+0000 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:57.421 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:57.422 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.420+0000 7f77385f2640 1 Processor -- start 2024-08-19T09:49:57.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.420+0000 7f77385f2640 1 -- start start 2024-08-19T09:49:57.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.420+0000 7f77385f2640 10 monclient: init 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient: _add_conns ranks=[8,7,3,6,2,0,4,1,5] 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient(hunting): picked mon.noname-e con 0x7f77300a5610 addr v1:172.21.15.1:6793/0 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient(hunting): picked mon.noname-i con 0x7f77300a59f0 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient(hunting): picked mon.noname-g con 0x7f773013f6a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773006ee20 con 0x7f773013f6a0 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773006e170 con 0x7f77300a59f0 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77300a70e0 con 0x7f77300a5610 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f77385f2640 10 monclient(hunting): authenticate will time out at 648.449297s 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f7735b8d640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f77300a59f0 0x7f77300a5e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.82:53254/0 (socket says 172.21.15.82:53254) 2024-08-19T09:49:57.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f7736b8f640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f773013f6a0 0x7f7730141a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.82:39240/0 (socket says 172.21.15.82:39240) 2024-08-19T09:49:57.425 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.421+0000 7f7735b8d640 1 -- 172.21.15.82:0/157325444 learned_addr learned my addr 172.21.15.82:0/157325444 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:57.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f7736b8f640 1 -- 172.21.15.82:0/157325444 >> v1:172.21.15.82:6790/0 conn(0x7f773013f6a0 legacy=0x7f7730141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f7735b8d640 1 -- 172.21.15.82:0/157325444 >> v1:172.21.15.82:6792/0 conn(0x7f77300a59f0 legacy=0x7f77300a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150735653 0 0) 0x7f773006ee20 con 0x7f773013f6a0 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): my global_id is 4134 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f771c003520 con 0x7f773013f6a0 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279072595 0 0) 0x7f773006e170 con 0x7f77300a59f0 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient(hunting): my global_id is 4129 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f773006ee20 con 0x7f77300a59f0 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773638e640 1 -- 172.21.15.82:0/157325444 >> v1:172.21.15.1:6793/0 conn(0x7f77300a5610 legacy=0x7f77300a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2585837492 0 0) 0x7f771c003520 con 0x7f773013f6a0 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 >> v1:172.21.15.1:6793/0 conn(0x7f77300a5610 legacy=0x7f77300a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 >> v1:172.21.15.82:6792/0 conn(0x7f77300a59f0 legacy=0x7f77300a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 1 monclient: found mon.noname-g 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.423+0000 7f773538c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77301461e0 con 0x7f773013f6a0 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7728001e60 con 0x7f773013f6a0 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 5 monclient: authenticate success, global_id 4134 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: dump: 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.436 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.437 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7728004b60 con 0x7f773013f6a0 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 1 -- 172.21.15.82:0/157325444 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77280050d0 con 0x7f773013f6a0 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 4 set_mon_vals no callback set 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f773538c640 10 monclient: dump: 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.439 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 10 monclient: shutdown 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 1 -- 172.21.15.82:0/157325444 >> v1:172.21.15.82:6790/0 conn(0x7f773013f6a0 legacy=0x7f7730141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 1 -- 172.21.15.82:0/157325444 shutdown_connections 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 1 -- 172.21.15.82:0/157325444 >> 172.21.15.82:0/157325444 conn(0x7f77300aa230 msgr2=0x7f77300a6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 1 -- 172.21.15.82:0/157325444 shutdown_connections 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.424+0000 7f77385f2640 1 -- 172.21.15.82:0/157325444 wait complete. 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 10 monclient: build_initial_monmap 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 10 monclient: monmap: 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-08-19T09:49:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 1 librados: starting msgr at 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 1 librados: starting objecter 2024-08-19T09:49:57.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 1 Processor -- start 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 1 -- start start 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 1 librados: setting wanted keys 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 1 librados: calling monclient init 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.425+0000 7f77385f2640 10 monclient: init 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient: _add_conns ranks=[5,4,7,8,2,3,6,1,0] 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient(hunting): picked mon.noname-h con 0x7f77300a5610 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient(hunting): picked mon.noname-c con 0x7f77300a59f0 addr v1:172.21.15.1:6791/0 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient(hunting): picked mon.noname-i con 0x7f773013f6a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7730122a20 con 0x7f77300a59f0 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7730121070 con 0x7f77300a5610 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7730142c00 con 0x7f773013f6a0 2024-08-19T09:49:57.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f77385f2640 10 monclient(hunting): authenticate will time out at 648.454289s 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f773638e640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f77300a5610 0x7f7730120250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:34478/0 (socket says 172.21.15.82:34478) 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f773638e640 1 -- 172.21.15.82:0/2962706664 learned_addr learned my addr 172.21.15.82:0/2962706664 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.426+0000 7f7736b8f640 1 --1- 172.21.15.82:0/2962706664 >> v1:172.21.15.82:6792/0 conn(0x7f773013f6a0 0x7f77301d6140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.82:53266/0 (socket says 172.21.15.82:53266) 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7736b8f640 1 -- 172.21.15.82:0/2962706664 >> v1:172.21.15.82:6792/0 conn(0x7f773013f6a0 legacy=0x7f77301d6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 client.?.objecter ms_handle_connect 0x7f773013f6a0 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 client.?.objecter ms_handle_connect 0x7f77300a5610 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f773638e640 1 -- 172.21.15.82:0/2962706664 >> v1:172.21.15.82:6791/0 conn(0x7f77300a5610 legacy=0x7f7730120250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844664544 0 0) 0x7f7730142c00 con 0x7f773013f6a0 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): my global_id is 4138 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7714004ee0 con 0x7f773013f6a0 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2709769731 0 0) 0x7f7730121070 con 0x7f77300a5610 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): my global_id is 4118 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7730142c00 con 0x7f77300a5610 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 client.?.objecter ms_handle_connect 0x7f77300a59f0 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7735b8d640 1 -- 172.21.15.82:0/2962706664 >> v1:172.21.15.1:6791/0 conn(0x7f77300a59f0 legacy=0x7f7730120960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3073519675 0 0) 0x7f7714004ee0 con 0x7f773013f6a0 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7730121070 con 0x7f773013f6a0 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77280020b0 con 0x7f773013f6a0 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): dump: 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.445 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3537412404 0 0) 0x7f7730142c00 con 0x7f77300a5610 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7714004ee0 con 0x7f77300a5610 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7724003610 con 0x7f77300a5610 2024-08-19T09:49:57.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.428+0000 7f7722ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient(hunting): dump: 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.447 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544427553 0 0) 0x7f7730122a20 con 0x7f77300a59f0 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient(hunting): my global_id is 4108 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7730142c00 con 0x7f77300a59f0 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 723116349 0 0) 0x7f7730121070 con 0x7f773013f6a0 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 >> v1:172.21.15.82:6791/0 conn(0x7f77300a5610 legacy=0x7f7730120250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 >> v1:172.21.15.1:6791/0 conn(0x7f77300a59f0 legacy=0x7f7730120960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 monclient: found mon.i 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7730073280 con 0x7f773013f6a0 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 5 monclient: authenticate success, global_id 4138 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 10 monclient: _renew_subs 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7730122740 con 0x7f773013f6a0 2024-08-19T09:49:57.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 10 monclient: _renew_subs 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f773013e3e0 con 0x7f773013f6a0 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f77385f2640 1 librados: init done 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7728004b60 con 0x7f773013f6a0 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7728005640 con 0x7f773013f6a0 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:49:57.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7734b8b640 4 set_mon_vals no callback set 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.429+0000 7f7722ffd640 10 monclient: dump: 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.451 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7728005b70 con 0x7f773013f6a0 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 1709525156 0 0) 0x7f7728007020 con 0x7f773013f6a0 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f7722ffd640 10 client.4138.objecter ms_dispatch 0x7f7730000b90 osd_map(3..3 src has 1..3) 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f77385f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f7722ffd640 3 client.4138.objecter handle_osd_map got epochs [3,3] > 0 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f7722ffd640 3 client.4138.objecter handle_osd_map decoding full epoch 3 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f7722ffd640 20 client.4138.objecter dump_active .. 0 homeless 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f77385f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f77385f2640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.430+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7718005080 con 0x7f773013f6a0 2024-08-19T09:49:57.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.432+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f7728005ff0 con 0x7f773013f6a0 2024-08-19T09:49:57.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.432+0000 7f7722ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.432+0000 7f7722ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:57.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.502+0000 7f77385f2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "066ec263-41d5-4d6c-a306-7cb11e1834ed", "id": 1}] 2024-08-19T09:49:57.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.502+0000 7f77385f2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "066ec263-41d5-4d6c-a306-7cb11e1834ed", "id": 1}] 2024-08-19T09:49:57.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.502+0000 7f77385f2640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:57.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.502+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "osd new", "uuid": "066ec263-41d5-4d6c-a306-7cb11e1834ed", "id": 1} v 0) -- 0x7f7718005370 con 0x7f773013f6a0 2024-08-19T09:49:57.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.506+0000 7f7722ffd640 1 -- 172.21.15.82:0/2962706664 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "066ec263-41d5-4d6c-a306-7cb11e1834ed", "id": 1}]=0 v4) ==== 112+0+2 (unknown 3414116415 0 940483276) 0x7f77280062a0 con 0x7f773013f6a0 2024-08-19T09:49:57.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.506+0000 7f7722ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "066ec263-41d5-4d6c-a306-7cb11e1834ed", "id": 1}] 2024-08-19T09:49:57.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.506+0000 7f7722ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:57.507 INFO:teuthology.orchestra.run.smithi082.stdout:1 2024-08-19T09:49:57.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 10 librados: watch_flush enter 2024-08-19T09:49:57.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 10 librados: watch_flush exit 2024-08-19T09:49:57.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 20 client.4138.objecter shutdown clearing up homeless session... 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 10 client.4138.objecter successfully canceled tick 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 10 monclient: shutdown 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 >> v1:172.21.15.82:6792/0 conn(0x7f773013f6a0 legacy=0x7f77301d6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 shutdown_connections 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 >> 172.21.15.82:0/2962706664 conn(0x7f77300aa230 msgr2=0x7f77300a3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 shutdown_connections 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 1 -- 172.21.15.82:0/2962706664 wait complete. 2024-08-19T09:49:57.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.507+0000 7f77385f2640 1 librados: shutdown 2024-08-19T09:49:57.518 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 500b3e1a-274e-4e36-b96c-ff8299d89a03 2 2024-08-19T09:49:57.616 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.614+0000 7fdb0798a640 10 monclient: get_monmap_and_config 2024-08-19T09:49:57.616 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.614+0000 7fdb0798a640 10 monclient: build_initial_monmap 2024-08-19T09:49:57.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.614+0000 7fdb0798a640 10 monclient: monmap: 2024-08-19T09:49:57.617 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:57.617 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.617 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:57.615972+0000 2024-08-19T09:49:57.617 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:57.615972+0000 2024-08-19T09:49:57.617 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:57.618 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.618 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:57.618 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:57.618 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:57.618 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:57.618 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:57.619 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:57.619 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:57.619 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:57.619 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:57.619 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.615+0000 7fdb0798a640 1 Processor -- start 2024-08-19T09:49:57.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.615+0000 7fdb0798a640 1 -- start start 2024-08-19T09:49:57.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.615+0000 7fdb0798a640 10 monclient: init 2024-08-19T09:49:57.630 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient: _add_conns ranks=[4,6,7,5,2,0,8,1,3] 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb000a4590 addr v1:172.21.15.1:6791/0 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient(hunting): picked mon.noname-d con 0x7fdb000a2c30 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient(hunting): picked mon.noname-i con 0x7fdb000a5080 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb00072800 con 0x7fdb000a4590 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb00072ae0 con 0x7fdb000a2c30 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb000693b0 con 0x7fdb000a5080 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb0798a640 10 monclient(hunting): authenticate will time out at 648.644280s 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb05f27640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7fdb000a5080 0x7fdb0013fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.82:53272/0 (socket says 172.21.15.82:53272) 2024-08-19T09:49:57.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.616+0000 7fdb05f27640 1 -- 172.21.15.82:0/190827886 learned_addr learned my addr 172.21.15.82:0/190827886 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdb05f27640 1 -- 172.21.15.82:0/190827886 >> v1:172.21.15.82:6792/0 conn(0x7fdb000a5080 legacy=0x7fdb0013fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043385740 0 0) 0x7fdb000693b0 con 0x7fdb000a5080 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 10 monclient(hunting): my global_id is 4147 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaec003520 con 0x7fdb000a5080 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdb04f25640 1 -- 172.21.15.82:0/190827886 >> v1:172.21.15.1:6792/0 conn(0x7fdb000a2c30 legacy=0x7fdb0013c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdb05726640 1 -- 172.21.15.82:0/190827886 >> v1:172.21.15.1:6791/0 conn(0x7fdb000a4590 legacy=0x7fdb000a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 122823004 0 0) 0x7fdaec003520 con 0x7fdb000a5080 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 >> v1:172.21.15.1:6792/0 conn(0x7fdb000a2c30 legacy=0x7fdb0013c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.618+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 >> v1:172.21.15.1:6791/0 conn(0x7fdb000a4590 legacy=0x7fdb000a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.633 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 1 monclient: found mon.noname-i 2024-08-19T09:49:57.633 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.633 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:49:57.633 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0006cff0 con 0x7fdb000a5080 2024-08-19T09:49:57.633 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.633 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdaf8001e60 con 0x7fdb000a5080 2024-08-19T09:49:57.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 5 monclient: authenticate success, global_id 4147 2024-08-19T09:49:57.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:57.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: dump: 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.635 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.636 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdaf8004c20 con 0x7fdb000a5080 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:57.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 1 -- 172.21.15.82:0/190827886 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdaf8005190 con 0x7fdb000a5080 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 4 set_mon_vals no callback set 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdaf3fff640 10 monclient: dump: 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.638 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.639 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 10 monclient: shutdown 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 1 -- 172.21.15.82:0/190827886 >> v1:172.21.15.82:6792/0 conn(0x7fdb000a5080 legacy=0x7fdb0013fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 1 -- 172.21.15.82:0/190827886 shutdown_connections 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 1 -- 172.21.15.82:0/190827886 >> 172.21.15.82:0/190827886 conn(0x7fdb000a5ff0 msgr2=0x7fdb00068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 1 -- 172.21.15.82:0/190827886 shutdown_connections 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.619+0000 7fdb0798a640 1 -- 172.21.15.82:0/190827886 wait complete. 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 10 monclient: build_initial_monmap 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 10 monclient: monmap: 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-08-19T09:49:57.641 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 1 librados: starting msgr at 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 1 librados: starting objecter 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 1 Processor -- start 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 1 -- start start 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 1 librados: setting wanted keys 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 1 librados: calling monclient init 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.620+0000 7fdb0798a640 10 monclient: init 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient: _add_conns ranks=[6,2,1,7,4,3,0,5,8] 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient(hunting): picked mon.noname-d con 0x7fdb000a2c30 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient(hunting): picked mon.noname-b con 0x7fdb000a4590 addr v1:172.21.15.1:6790/0 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient(hunting): picked mon.noname-f con 0x7fdb000a5080 addr v1:172.21.15.82:6789/0 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb00071660 con 0x7fdb000a5080 2024-08-19T09:49:57.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb000a3f90 con 0x7fdb000a4590 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb000a1fe0 con 0x7fdb000a2c30 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb0798a640 10 monclient(hunting): authenticate will time out at 648.649341s 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb05726640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdb000a2c30 0x7fdb001ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.82:54268/0 (socket says 172.21.15.82:54268) 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb05f27640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fdb000a5080 0x7fdb001d6c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:45276/0 (socket says 172.21.15.82:45276) 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.621+0000 7fdb05726640 1 -- 172.21.15.82:0/1661328630 learned_addr learned my addr 172.21.15.82:0/1661328630 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdb05f27640 1 -- 172.21.15.82:0/1661328630 >> v1:172.21.15.82:6789/0 conn(0x7fdb000a5080 legacy=0x7fdb001d6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 client.?.objecter ms_handle_connect 0x7fdb000a5080 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 client.?.objecter ms_handle_connect 0x7fdb000a2c30 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdb05726640 1 -- 172.21.15.82:0/1661328630 >> v1:172.21.15.1:6792/0 conn(0x7fdb000a2c30 legacy=0x7fdb001ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdb04f25640 1 -- 172.21.15.82:0/1661328630 >> v1:172.21.15.1:6790/0 conn(0x7fdb000a4590 legacy=0x7fdb001d3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 client.?.objecter ms_handle_connect 0x7fdb000a4590 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204937928 0 0) 0x7fdb00071660 con 0x7fdb000a5080 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): my global_id is 4123 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdae4004ee0 con 0x7fdb000a5080 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740270815 0 0) 0x7fdb000a1fe0 con 0x7fdb000a2c30 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): my global_id is 4128 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb00071660 con 0x7fdb000a2c30 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1643160237 0 0) 0x7fdaf4001630 con 0x7fdb000a4590 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): my global_id is 4124 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb000a1fe0 con 0x7fdb000a4590 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1503874180 0 0) 0x7fdae4004ee0 con 0x7fdb000a5080 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdaf4001630 con 0x7fdb000a5080 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdaf80020b0 con 0x7fdb000a5080 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.623+0000 7fdaf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): dump: 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.647 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3701020384 0 0) 0x7fdb00071660 con 0x7fdb000a2c30 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdae4004ee0 con 0x7fdb000a2c30 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdafc003610 con 0x7fdb000a2c30 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): dump: 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.648 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.649 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1620901427 0 0) 0x7fdb000a1fe0 con 0x7fdb000a4590 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb00071660 con 0x7fdb000a4590 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdaf4003950 con 0x7fdb000a4590 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient(hunting): dump: 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.650 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.651 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3995782444 0 0) 0x7fdaf4001630 con 0x7fdb000a5080 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 >> v1:172.21.15.1:6792/0 conn(0x7fdb000a2c30 legacy=0x7fdb001ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 >> v1:172.21.15.1:6790/0 conn(0x7fdb000a4590 legacy=0x7fdb001d3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 monclient: found mon.f 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb001298c0 con 0x7fdb000a5080 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 5 monclient: authenticate success, global_id 4123 2024-08-19T09:49:57.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 10 monclient: _renew_subs 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdb00128500 con 0x7fdb000a5080 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdaf80053c0 con 0x7fdb000a5080 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdaf80058b0 con 0x7fdb000a5080 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 monclient: dump: 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdaf8005de0 con 0x7fdb000a5080 2024-08-19T09:49:57.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf37fe640 4 set_mon_vals no callback set 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 10 monclient: _renew_subs 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdb00070b80 con 0x7fdb000a5080 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdb0798a640 1 librados: init done 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 609646489 0 0) 0x7fdaf8006720 con 0x7fdb000a5080 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 10 client.4123.objecter ms_dispatch 0x7fdb00000b90 osd_map(4..4 src has 1..4) 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.624+0000 7fdaf1ffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.625+0000 7fdb0798a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.625+0000 7fdb0798a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.625+0000 7fdb0798a640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:49:57.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.625+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb0006c230 con 0x7fdb000a5080 2024-08-19T09:49:57.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.627+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fdb0006c230 con 0x7fdb000a5080 2024-08-19T09:49:57.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.627+0000 7fdaf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.627+0000 7fdaf1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:57.698 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.697+0000 7fdb0798a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "500b3e1a-274e-4e36-b96c-ff8299d89a03", "id": 2}] 2024-08-19T09:49:57.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.697+0000 7fdb0798a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "500b3e1a-274e-4e36-b96c-ff8299d89a03", "id": 2}] 2024-08-19T09:49:57.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.697+0000 7fdb0798a640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:49:57.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.697+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "500b3e1a-274e-4e36-b96c-ff8299d89a03", "id": 2} v 0) -- 0x7fdb00051b00 con 0x7fdb000a5080 2024-08-19T09:49:57.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.701+0000 7fdaf1ffb640 1 -- 172.21.15.82:0/1661328630 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "500b3e1a-274e-4e36-b96c-ff8299d89a03", "id": 2}]=0 v5) ==== 112+0+2 (unknown 3111663846 0 216599125) 0x7fdaf80069a0 con 0x7fdb000a5080 2024-08-19T09:49:57.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.701+0000 7fdaf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "500b3e1a-274e-4e36-b96c-ff8299d89a03", "id": 2}] 2024-08-19T09:49:57.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.701+0000 7fdaf1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:57.702 INFO:teuthology.orchestra.run.smithi082.stdout:2 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 10 librados: watch_flush enter 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 10 librados: watch_flush exit 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 20 client.4123.objecter shutdown clearing up homeless session... 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 10 client.4123.objecter successfully canceled tick 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 10 monclient: shutdown 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 >> v1:172.21.15.82:6789/0 conn(0x7fdb000a5080 legacy=0x7fdb001d6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 shutdown_connections 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 >> 172.21.15.82:0/1661328630 conn(0x7fdb000a5ff0 msgr2=0x7fdb000a8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 shutdown_connections 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 1 -- 172.21.15.82:0/1661328630 wait complete. 2024-08-19T09:49:57.704 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.702+0000 7fdb0798a640 1 librados: shutdown 2024-08-19T09:49:57.713 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799 3 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.811+0000 7fbf8cfaa640 10 monclient: get_monmap_and_config 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.811+0000 7fbf8cfaa640 10 monclient: build_initial_monmap 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.811+0000 7fbf8cfaa640 10 monclient: monmap: 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:57.812894+0000 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:57.812894+0000 2024-08-19T09:49:57.813 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.812+0000 7fbf8cfaa640 1 Processor -- start 2024-08-19T09:49:57.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.812+0000 7fbf8cfaa640 1 -- start start 2024-08-19T09:49:57.815 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.812+0000 7fbf8cfaa640 10 monclient: init 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient: _add_conns ranks=[6,5,3,7,1,4,8,2,0] 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient(hunting): picked mon.noname-d con 0x7fbf880a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient(hunting): picked mon.noname-h con 0x7fbf880694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient(hunting): picked mon.noname-g con 0x7fbf880a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf880a3c70 con 0x7fbf880a46a0 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf880a33d0 con 0x7fbf880694b0 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf880a36b0 con 0x7fbf880a6830 2024-08-19T09:49:57.820 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf8cfaa640 10 monclient(hunting): authenticate will time out at 648.841187s 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf85d9b640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fbf880694b0 0x7fbf880a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:34482/0 (socket says 172.21.15.82:34482) 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.813+0000 7fbf85d9b640 1 -- 172.21.15.82:0/3596660495 learned_addr learned my addr 172.21.15.82:0/3596660495 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf85d9b640 1 -- 172.21.15.82:0/3596660495 >> v1:172.21.15.82:6791/0 conn(0x7fbf880694b0 legacy=0x7fbf880a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf86d9d640 1 -- 172.21.15.82:0/3596660495 >> v1:172.21.15.82:6790/0 conn(0x7fbf880a46a0 legacy=0x7fbf88141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316266127 0 0) 0x7fbf880a33d0 con 0x7fbf880694b0 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): my global_id is 4127 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf74003520 con 0x7fbf880694b0 2024-08-19T09:49:57.821 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462262811 0 0) 0x7fbf880a3c70 con 0x7fbf880a46a0 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient(hunting): my global_id is 4143 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf880a33d0 con 0x7fbf880a46a0 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8659c640 1 -- 172.21.15.82:0/3596660495 >> v1:172.21.15.1:6792/0 conn(0x7fbf880a6830 legacy=0x7fbf880a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3636137587 0 0) 0x7fbf74003520 con 0x7fbf880694b0 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 >> v1:172.21.15.1:6792/0 conn(0x7fbf880a6830 legacy=0x7fbf880a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 >> v1:172.21.15.82:6790/0 conn(0x7fbf880a46a0 legacy=0x7fbf88141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 monclient: found mon.noname-h 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.822 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8806dbb0 con 0x7fbf880694b0 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8cfaa640 5 monclient: authenticate success, global_id 4127 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf70001ae0 con 0x7fbf880694b0 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8cfaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8559a640 10 monclient: dump: 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.823 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.815+0000 7fbf8cfaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf700048a0 con 0x7fbf880694b0 2024-08-19T09:49:57.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8559a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8559a640 1 -- 172.21.15.82:0/3596660495 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf70004d90 con 0x7fbf880694b0 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 4 set_mon_vals no callback set 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8559a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8559a640 10 monclient: dump: 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.825 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 10 monclient: shutdown 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/3596660495 >> v1:172.21.15.82:6791/0 conn(0x7fbf880694b0 legacy=0x7fbf880a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/3596660495 shutdown_connections 2024-08-19T09:49:57.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/3596660495 >> 172.21.15.82:0/3596660495 conn(0x7fbf880a9020 msgr2=0x7fbf88068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/3596660495 shutdown_connections 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.816+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/3596660495 wait complete. 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 10 monclient: build_initial_monmap 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 10 monclient: monmap: 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:57.827 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 1 librados: starting msgr at 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 1 librados: starting objecter 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 1 Processor -- start 2024-08-19T09:49:57.828 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 1 -- start start 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 1 librados: setting wanted keys 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 1 librados: calling monclient init 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.817+0000 7fbf8cfaa640 10 monclient: init 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient: _add_conns ranks=[7,5,0,4,1,8,3,6,2] 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient(hunting): picked mon.noname-i con 0x7fbf880694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient(hunting): picked mon.noname-h con 0x7fbf880a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient(hunting): picked mon.noname-a con 0x7fbf880a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf881cf260 con 0x7fbf880a6830 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf88137fa0 con 0x7fbf880a46a0 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf88138570 con 0x7fbf880694b0 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf8cfaa640 10 monclient(hunting): authenticate will time out at 648.846404s 2024-08-19T09:49:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf85d9b640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fbf880a46a0 0x7fbf881ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:34490/0 (socket says 172.21.15.82:34490) 2024-08-19T09:49:57.830 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.818+0000 7fbf85d9b640 1 -- 172.21.15.82:0/1276828223 learned_addr learned my addr 172.21.15.82:0/1276828223 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:57.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf85d9b640 1 -- 172.21.15.82:0/1276828223 >> v1:172.21.15.82:6791/0 conn(0x7fbf880a46a0 legacy=0x7fbf881ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 client.?.objecter ms_handle_connect 0x7fbf880a46a0 2024-08-19T09:49:57.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 client.?.objecter ms_handle_connect 0x7fbf880694b0 2024-08-19T09:49:57.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf8659c640 1 -- 172.21.15.82:0/1276828223 >> v1:172.21.15.82:6792/0 conn(0x7fbf880694b0 legacy=0x7fbf881ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1290663325 0 0) 0x7fbf88137fa0 con 0x7fbf880a46a0 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): my global_id is 4136 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf68004eb0 con 0x7fbf880a46a0 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60901075 0 0) 0x7fbf88138570 con 0x7fbf880694b0 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): my global_id is 4156 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf88137fa0 con 0x7fbf880694b0 2024-08-19T09:49:57.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf86d9d640 1 -- 172.21.15.82:0/1276828223 >> v1:172.21.15.1:6789/0 conn(0x7fbf880a6830 legacy=0x7fbf881d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 client.?.objecter ms_handle_connect 0x7fbf880a6830 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3492550915 0 0) 0x7fbf68004eb0 con 0x7fbf880a46a0 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf88138570 con 0x7fbf880a46a0 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf70001d30 con 0x7fbf880a46a0 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.820+0000 7fbf7effd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): dump: 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.836 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:57.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763295416 0 0) 0x7fbf881cf260 con 0x7fbf880a6830 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): my global_id is 4122 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf68004eb0 con 0x7fbf880a6830 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3535966252 0 0) 0x7fbf88137fa0 con 0x7fbf880694b0 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf881cf260 con 0x7fbf880694b0 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf78001d10 con 0x7fbf880694b0 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient(hunting): dump: 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.838 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.839 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 716774403 0 0) 0x7fbf88138570 con 0x7fbf880a46a0 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 >> v1:172.21.15.82:6792/0 conn(0x7fbf880694b0 legacy=0x7fbf881ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 >> v1:172.21.15.1:6789/0 conn(0x7fbf880a6830 legacy=0x7fbf881d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: _finish_hunting 0 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 monclient: found mon.h 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf881378b0 con 0x7fbf880a46a0 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: _finish_auth 0 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: _check_auth_tickets 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 5 monclient: authenticate success, global_id 4136 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 10 monclient: _renew_subs 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:49:57.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf88136af0 con 0x7fbf880a46a0 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 10 monclient: _renew_subs 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf881d7dc0 con 0x7fbf880a46a0 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf70005040 con 0x7fbf880a46a0 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf8cfaa640 1 librados: init done 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf700055b0 con 0x7fbf880a46a0 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 monclient: dump: 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:57.842 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf84d99640 4 set_mon_vals no callback set 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbf70005bd0 con 0x7fbf880a46a0 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 1923955384 0 0) 0x7fbf70007280 con 0x7fbf880a46a0 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 10 client.4136.objecter ms_dispatch 0x7fbf88000b90 osd_map(5..5 src has 1..5) 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 3 client.4136.objecter handle_osd_map got epochs [5,5] > 0 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 3 client.4136.objecter handle_osd_map decoding full epoch 5 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.821+0000 7fbf7effd640 20 client.4136.objecter dump_active .. 0 homeless 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.822+0000 7fbf8cfaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.822+0000 7fbf8cfaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.822+0000 7fbf8cfaa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.822+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6c005080 con 0x7fbf880a46a0 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.824+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fbf70006df0 con 0x7fbf880a46a0 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.824+0000 7fbf7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:57.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.824+0000 7fbf7effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:57.895 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.894+0000 7fbf8cfaa640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799", "id": 3}] 2024-08-19T09:49:57.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.894+0000 7fbf8cfaa640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799", "id": 3}] 2024-08-19T09:49:57.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.894+0000 7fbf8cfaa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:49:57.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.894+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "osd new", "uuid": "cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799", "id": 3} v 0) -- 0x7fbf6c005370 con 0x7fbf880a46a0 2024-08-19T09:49:57.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.899+0000 7fbf7effd640 1 -- 172.21.15.82:0/1276828223 <== mon.5 v1:172.21.15.82:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799", "id": 3}]=0 v6) ==== 112+0+2 (unknown 2717851162 0 525046306) 0x7fbf700060f0 con 0x7fbf880a46a0 2024-08-19T09:49:57.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799", "id": 3}] 2024-08-19T09:49:57.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf7effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:57.901 INFO:teuthology.orchestra.run.smithi082.stdout:3 2024-08-19T09:49:57.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 10 librados: watch_flush enter 2024-08-19T09:49:57.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 10 librados: watch_flush exit 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 20 client.4136.objecter shutdown clearing up homeless session... 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 10 client.4136.objecter successfully canceled tick 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 10 monclient: shutdown 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 >> v1:172.21.15.82:6791/0 conn(0x7fbf880a46a0 legacy=0x7fbf881ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.900+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 shutdown_connections 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.901+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 >> 172.21.15.82:0/1276828223 conn(0x7fbf880a9020 msgr2=0x7fbf880a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.901+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 shutdown_connections 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.901+0000 7fbf8cfaa640 1 -- 172.21.15.82:0/1276828223 wait complete. 2024-08-19T09:49:57.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:57.901+0000 7fbf8cfaa640 1 librados: shutdown 2024-08-19T09:49:57.914 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 411cb062-e88f-4611-9cb9-bd03b212b768 4 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.010+0000 7f84f0067640 10 monclient: get_monmap_and_config 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.010+0000 7f84f0067640 10 monclient: build_initial_monmap 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.010+0000 7f84f0067640 10 monclient: monmap: 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:58.011904+0000 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:58.011904+0000 2024-08-19T09:49:58.012 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.011+0000 7f84f0067640 1 Processor -- start 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.011+0000 7f84f0067640 1 -- start start 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.011+0000 7f84f0067640 10 monclient: init 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient: _add_conns ranks=[2,5,1,7,4,6,0,3,8] 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient(hunting): picked mon.noname-b con 0x7f84e80a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient(hunting): picked mon.noname-h con 0x7f84e80694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient(hunting): picked mon.noname-f con 0x7f84e80a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e80a3c70 con 0x7f84e80a46a0 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e80a33d0 con 0x7f84e80a6830 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e80a36b0 con 0x7f84e80694b0 2024-08-19T09:49:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84f0067640 10 monclient(hunting): authenticate will time out at 649.040237s 2024-08-19T09:49:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84ed602640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f84e80694b0 0x7f84e80a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:34502/0 (socket says 172.21.15.82:34502) 2024-08-19T09:49:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.012+0000 7f84ed602640 1 -- 172.21.15.82:0/429037084 learned_addr learned my addr 172.21.15.82:0/429037084 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:58.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ed602640 1 -- 172.21.15.82:0/429037084 >> v1:172.21.15.82:6791/0 conn(0x7f84e80694b0 legacy=0x7f84e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ee604640 1 -- 172.21.15.82:0/429037084 >> v1:172.21.15.82:6789/0 conn(0x7f84e80a46a0 legacy=0x7f84e8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ede03640 1 -- 172.21.15.82:0/429037084 >> v1:172.21.15.1:6790/0 conn(0x7f84e80a6830 legacy=0x7f84e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403322887 0 0) 0x7f84e80a36b0 con 0x7f84e80694b0 2024-08-19T09:49:58.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): my global_id is 4145 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84d4003520 con 0x7f84e80694b0 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359185852 0 0) 0x7f84e80a3c70 con 0x7f84e80a46a0 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 10 monclient(hunting): my global_id is 4132 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.014+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e80a36b0 con 0x7f84e80a46a0 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301699211 0 0) 0x7f84e80a33d0 con 0x7f84e80a6830 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient(hunting): my global_id is 4133 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e80a3c70 con 0x7f84e80a6830 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3267154643 0 0) 0x7f84d4003520 con 0x7f84e80694b0 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 >> v1:172.21.15.1:6790/0 conn(0x7f84e80a6830 legacy=0x7f84e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 >> v1:172.21.15.82:6789/0 conn(0x7f84e80a46a0 legacy=0x7f84e8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: _finish_hunting 0 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 monclient: found mon.noname-h 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e806dbb0 con 0x7f84e80694b0 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: _finish_auth 0 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: _check_auth_tickets 2024-08-19T09:49:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f84e4001ae0 con 0x7f84e80694b0 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84f0067640 5 monclient: authenticate success, global_id 4145 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84f0067640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: dump: 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84f0067640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84e40048a0 con 0x7f84e80694b0 2024-08-19T09:49:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84f0067640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 1 -- 172.21.15.82:0/429037084 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f84e4004e10 con 0x7f84e80694b0 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84f0067640 4 set_mon_vals no callback set 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.015+0000 7f84ece01640 10 monclient: dump: 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.025 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 10 monclient: shutdown 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 1 -- 172.21.15.82:0/429037084 >> v1:172.21.15.82:6791/0 conn(0x7f84e80694b0 legacy=0x7f84e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 1 -- 172.21.15.82:0/429037084 shutdown_connections 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 1 -- 172.21.15.82:0/429037084 >> 172.21.15.82:0/429037084 conn(0x7f84e80a9020 msgr2=0x7f84e8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 1 -- 172.21.15.82:0/429037084 shutdown_connections 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 1 -- 172.21.15.82:0/429037084 wait complete. 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 10 monclient: build_initial_monmap 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.016+0000 7f84f0067640 10 monclient: monmap: 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:58.027 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 librados: starting msgr at 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 librados: starting objecter 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 Processor -- start 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 -- start start 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 librados: setting wanted keys 2024-08-19T09:49:58.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 librados: calling monclient init 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient: init 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient: _add_conns ranks=[7,8,4,2,0,6,1,5,3] 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient(hunting): picked mon.noname-i con 0x7f84e80694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient(hunting): picked mon.noname-e con 0x7f84e80a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient(hunting): picked mon.noname-c con 0x7f84e80a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e8142710 con 0x7f84e80a6830 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e8139860 con 0x7f84e80694b0 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e8138b90 con 0x7f84e80a46a0 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84f0067640 10 monclient(hunting): authenticate will time out at 649.045640s 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84ede03640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 0x7f84e8137d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.82:53290/0 (socket says 172.21.15.82:53290) 2024-08-19T09:49:58.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.017+0000 7f84ede03640 1 -- 172.21.15.82:0/1140982693 learned_addr learned my addr 172.21.15.82:0/1140982693 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84ede03640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 legacy=0x7f84e8137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84de7fc640 10 client.?.objecter ms_handle_connect 0x7f84e80694b0 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84de7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84ede03640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 legacy=0x7f84e8137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84ede03640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 legacy=0x7f84e8137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84ede03640 1 --1- 172.21.15.82:0/1140982693 >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 0x7f84e8137d70 :-1 s=OPENED pgs=41 cs=1 l=1).handle_message read tag failed 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84ede03640 1 --1- 172.21.15.82:0/1140982693 >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 0x7f84e8137d70 :-1 s=OPENED pgs=41 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.019+0000 7f84de7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6792/0 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 client.?.objecter ms_handle_connect 0x7f84e80a6830 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84ee604640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.1:6791/0 conn(0x7f84e80a6830 legacy=0x7f84e81d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 client.?.objecter ms_handle_connect 0x7f84e80a46a0 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84ed602640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.1:6793/0 conn(0x7f84e80a46a0 legacy=0x7f84e8138480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047093014 0 0) 0x7f84e8142710 con 0x7f84e80a6830 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): my global_id is 4126 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84cc004eb0 con 0x7f84e80a6830 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2380758590 0 0) 0x7f84e8138b90 con 0x7f84e80a46a0 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): my global_id is 4130 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e8142710 con 0x7f84e80a46a0 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3738155284 0 0) 0x7f84cc004eb0 con 0x7f84e80a6830 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84e8138b90 con 0x7f84e80a6830 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f84e0003990 con 0x7f84e80a6830 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): dump: 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.036 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1728708931 0 0) 0x7f84e8142710 con 0x7f84e80a46a0 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84cc004eb0 con 0x7f84e80a46a0 2024-08-19T09:49:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f84e4001d30 con 0x7f84e80a46a0 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient(hunting): dump: 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 954065412 0 0) 0x7f84e8138b90 con 0x7f84e80a6830 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.1:6793/0 conn(0x7f84e80a46a0 legacy=0x7f84e8138480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.82:6792/0 conn(0x7f84e80694b0 legacy=0x7f84e8137d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 monclient: found mon.c 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:49:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e813b320 con 0x7f84e80a6830 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: _finish_auth 0 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 5 monclient: authenticate success, global_id 4126 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 10 monclient: _renew_subs 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f84e8142400 con 0x7f84e80a6830 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 10 monclient: _renew_subs 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f84e813a5a0 con 0x7f84e80a6830 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84f0067640 1 librados: init done 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84e0003e80 con 0x7f84e80a6830 2024-08-19T09:49:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f84e00044b0 con 0x7f84e80a6830 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84dffff640 4 set_mon_vals no callback set 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 monclient: dump: 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f84e0005ed0 con 0x7f84e80a6830 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 3399583542 0 0) 0x7f84e00072b0 con 0x7f84e80a6830 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 10 client.4126.objecter ms_dispatch 0x7f84e8000b90 osd_map(6..6 src has 1..6) 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 3 client.4126.objecter handle_osd_map got epochs [6,6] > 0 2024-08-19T09:49:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.020+0000 7f84de7fc640 3 client.4126.objecter handle_osd_map decoding full epoch 6 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.021+0000 7f84de7fc640 20 client.4126.objecter dump_active .. 0 homeless 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.021+0000 7f84f0067640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.021+0000 7f84f0067640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.021+0000 7f84f0067640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.021+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d0005080 con 0x7f84e80a6830 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.023+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f84e0004770 con 0x7f84e80a6830 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.023+0000 7f84de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.023+0000 7f84de7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:58.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.094+0000 7f84f0067640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "411cb062-e88f-4611-9cb9-bd03b212b768", "id": 4}] 2024-08-19T09:49:58.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.094+0000 7f84f0067640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "411cb062-e88f-4611-9cb9-bd03b212b768", "id": 4}] 2024-08-19T09:49:58.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.094+0000 7f84f0067640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:49:58.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.094+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "osd new", "uuid": "411cb062-e88f-4611-9cb9-bd03b212b768", "id": 4} v 0) -- 0x7f84d0005370 con 0x7f84e80a6830 2024-08-19T09:49:58.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.098+0000 7f84de7fc640 1 -- 172.21.15.82:0/1140982693 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "411cb062-e88f-4611-9cb9-bd03b212b768", "id": 4}]=0 v7) ==== 112+0+2 (unknown 757113188 0 1697012583) 0x7f84e0006ef0 con 0x7f84e80a6830 2024-08-19T09:49:58.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.099+0000 7f84de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "411cb062-e88f-4611-9cb9-bd03b212b768", "id": 4}] 2024-08-19T09:49:58.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.099+0000 7f84de7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:58.100 INFO:teuthology.orchestra.run.smithi082.stdout:4 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.099+0000 7f84f0067640 10 librados: watch_flush enter 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.099+0000 7f84f0067640 10 librados: watch_flush exit 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.099+0000 7f84f0067640 20 client.4126.objecter shutdown clearing up homeless session... 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.099+0000 7f84f0067640 10 client.4126.objecter successfully canceled tick 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 10 monclient: shutdown 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 >> v1:172.21.15.1:6791/0 conn(0x7f84e80a6830 legacy=0x7f84e81d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 shutdown_connections 2024-08-19T09:49:58.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 >> 172.21.15.82:0/1140982693 conn(0x7f84e80a9020 msgr2=0x7f84e80a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:58.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 shutdown_connections 2024-08-19T09:49:58.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 1 -- 172.21.15.82:0/1140982693 wait complete. 2024-08-19T09:49:58.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.100+0000 7f84f0067640 1 librados: shutdown 2024-08-19T09:49:58.111 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f 5 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.207+0000 7fe33ed88640 10 monclient: get_monmap_and_config 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.207+0000 7fe33ed88640 10 monclient: build_initial_monmap 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.207+0000 7fe33ed88640 10 monclient: monmap: 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:58.208152+0000 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:58.208152+0000 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:58.208 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:58.209 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.210 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.207+0000 7fe33ed88640 1 Processor -- start 2024-08-19T09:49:58.210 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.207+0000 7fe33ed88640 1 -- start start 2024-08-19T09:49:58.210 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.207+0000 7fe33ed88640 10 monclient: init 2024-08-19T09:49:58.210 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient: _add_conns ranks=[5,7,3,1,8,6,0,4,2] 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient(hunting): picked mon.noname-h con 0x7fe3380a4600 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient(hunting): picked mon.noname-i con 0x7fe3380a50f0 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient(hunting): picked mon.noname-g con 0x7fe33813cd10 addr v1:172.21.15.82:6790/0 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3380727b0 con 0x7fe33813cd10 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3380702f0 con 0x7fe3380a4600 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe338070570 con 0x7fe3380a50f0 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33ed88640 10 monclient(hunting): authenticate will time out at 649.236359s 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33cb24640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fe3380a4600 0x7fe3380a49e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:34516/0 (socket says 172.21.15.82:34516) 2024-08-19T09:49:58.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.208+0000 7fe33cb24640 1 -- 172.21.15.82:0/1554637349 learned_addr learned my addr 172.21.15.82:0/1554637349 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe337fff640 1 -- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6792/0 conn(0x7fe3380a50f0 legacy=0x7fe33813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe33cb24640 1 -- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6791/0 conn(0x7fe3380a4600 legacy=0x7fe3380a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe33d325640 1 -- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6790/0 conn(0x7fe33813cd10 legacy=0x7fe33813f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3582151041 0 0) 0x7fe338070570 con 0x7fe3380a50f0 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): my global_id is 4165 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe328003520 con 0x7fe3380a50f0 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1280651195 0 0) 0x7fe3380702f0 con 0x7fe3380a4600 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): my global_id is 4154 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe338070570 con 0x7fe3380a4600 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784117043 0 0) 0x7fe3380727b0 con 0x7fe33813cd10 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 10 monclient(hunting): my global_id is 4152 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.210+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3380702f0 con 0x7fe33813cd10 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3054682011 0 0) 0x7fe328003520 con 0x7fe3380a50f0 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33d325640 1 --1- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6790/0 conn(0x7fe33813cd10 0x7fe33813f100 :-1 s=OPENED pgs=35 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fe330001dd0 mon_map magic: 0 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6791/0 conn(0x7fe3380a4600 legacy=0x7fe3380a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6790/0 conn(0x7fe33813cd10 legacy=0x7fe33813f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: _finish_hunting 0 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 monclient: found mon.noname-i 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe338071610 con 0x7fe3380a50f0 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: _finish_auth 0 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: _check_auth_tickets 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe324001b30 con 0x7fe3380a50f0 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 5 monclient: authenticate success, global_id 4165 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:49:58.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: dump: 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe330001dd0 con 0x7fe33813cd10 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe324003610 con 0x7fe3380a50f0 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 1 -- 172.21.15.82:0/1554637349 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe324003b80 con 0x7fe3380a50f0 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 4 set_mon_vals no callback set 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:49:58.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe3377fe640 10 monclient: dump: 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.221 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 10 monclient: shutdown 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 1 -- 172.21.15.82:0/1554637349 >> v1:172.21.15.82:6792/0 conn(0x7fe3380a50f0 legacy=0x7fe33813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 1 -- 172.21.15.82:0/1554637349 shutdown_connections 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.211+0000 7fe33ed88640 1 -- 172.21.15.82:0/1554637349 >> 172.21.15.82:0/1554637349 conn(0x7fe3380a56f0 msgr2=0x7fe3380a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 -- 172.21.15.82:0/1554637349 shutdown_connections 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 -- 172.21.15.82:0/1554637349 wait complete. 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 10 monclient: build_initial_monmap 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 10 monclient: monmap: 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-08-19T09:49:58.222 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 librados: starting msgr at 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 librados: starting objecter 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 Processor -- start 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 -- start start 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 librados: setting wanted keys 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 1 librados: calling monclient init 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.212+0000 7fe33ed88640 10 monclient: init 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient: _add_conns ranks=[7,2,0,4,5,3,1,8,6] 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient(hunting): picked mon.noname-i con 0x7fe3380a4600 addr v1:172.21.15.82:6792/0 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient(hunting): picked mon.noname-b con 0x7fe3380a50f0 addr v1:172.21.15.1:6790/0 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient(hunting): picked mon.noname-a con 0x7fe33813cd10 addr v1:172.21.15.1:6789/0 2024-08-19T09:49:58.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33813b6b0 con 0x7fe33813cd10 2024-08-19T09:49:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33813c7b0 con 0x7fe3380a50f0 2024-08-19T09:49:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3381384c0 con 0x7fe3380a4600 2024-08-19T09:49:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33ed88640 10 monclient(hunting): authenticate will time out at 649.241644s 2024-08-19T09:49:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33cb24640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7fe3380a4600 0x7fe3380a1130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.82:53310/0 (socket says 172.21.15.82:53310) 2024-08-19T09:49:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.213+0000 7fe33cb24640 1 -- 172.21.15.82:0/1973366434 learned_addr learned my addr 172.21.15.82:0/1973366434 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe33cb24640 1 -- 172.21.15.82:0/1973366434 >> v1:172.21.15.82:6792/0 conn(0x7fe3380a4600 legacy=0x7fe3380a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 10 client.?.objecter ms_handle_connect 0x7fe3380a4600 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 320384355 0 0) 0x7fe3381384c0 con 0x7fe3380a4600 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 10 monclient(hunting): my global_id is 4174 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.215+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe31c004eb0 con 0x7fe3380a4600 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33d325640 1 -- 172.21.15.82:0/1973366434 >> v1:172.21.15.1:6789/0 conn(0x7fe33813cd10 legacy=0x7fe3381d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 client.?.objecter ms_handle_connect 0x7fe33813cd10 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 client.?.objecter ms_handle_connect 0x7fe3380a50f0 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe337fff640 1 -- 172.21.15.82:0/1973366434 >> v1:172.21.15.1:6790/0 conn(0x7fe3380a50f0 legacy=0x7fe3380a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3479575334 0 0) 0x7fe31c004eb0 con 0x7fe3380a4600 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3381384c0 con 0x7fe3380a4600 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe32c004b20 con 0x7fe3380a4600 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): dump: 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.230 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.231 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944063211 0 0) 0x7fe33813b6b0 con 0x7fe33813cd10 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): my global_id is 4131 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe31c004eb0 con 0x7fe33813cd10 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003698631 0 0) 0x7fe338071610 con 0x7fe3380a50f0 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient(hunting): my global_id is 4142 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe33813b6b0 con 0x7fe3380a50f0 2024-08-19T09:49:58.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1691509749 0 0) 0x7fe3381384c0 con 0x7fe3380a4600 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 >> v1:172.21.15.1:6790/0 conn(0x7fe3380a50f0 legacy=0x7fe3380a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 >> v1:172.21.15.1:6789/0 conn(0x7fe33813cd10 legacy=0x7fe3381d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: _finish_hunting 0 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 monclient: found mon.i 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe338138a60 con 0x7fe3380a4600 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: _finish_auth 0 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: _check_auth_tickets 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 5 monclient: authenticate success, global_id 4174 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 10 monclient: _renew_subs 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:58.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe338136de0 con 0x7fe3380a4600 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 10 monclient: _renew_subs 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe338138780 con 0x7fe3380a4600 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe33ed88640 1 librados: init done 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe32c004d40 con 0x7fe3380a4600 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe336ffd640 4 set_mon_vals no callback set 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe32c0052b0 con 0x7fe3380a4600 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 monclient: dump: 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-08-19T09:49:58.234 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.235 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe32c0054d0 con 0x7fe3380a4600 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 3121028774 0 0) 0x7fe32c006f90 con 0x7fe3380a4600 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 10 client.4174.objecter ms_dispatch 0x7fe338000b90 osd_map(7..7 src has 1..7) 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 3 client.4174.objecter handle_osd_map got epochs [7,7] > 0 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 3 client.4174.objecter handle_osd_map decoding full epoch 7 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.216+0000 7fe3357fa640 20 client.4174.objecter dump_active .. 0 homeless 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.217+0000 7fe33ed88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.217+0000 7fe33ed88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.217+0000 7fe33ed88640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.217+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe318005080 con 0x7fe3380a4600 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.219+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fe32c006b20 con 0x7fe3380a4600 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.219+0000 7fe3357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.219+0000 7fe3357fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:58.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.289+0000 7fe33ed88640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f", "id": 5}] 2024-08-19T09:49:58.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.289+0000 7fe33ed88640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f", "id": 5}] 2024-08-19T09:49:58.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.289+0000 7fe33ed88640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:49:58.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.289+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "osd new", "uuid": "fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f", "id": 5} v 0) -- 0x7fe318005370 con 0x7fe3380a4600 2024-08-19T09:49:58.297 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.293+0000 7fe3357fa640 1 -- 172.21.15.82:0/1973366434 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f", "id": 5}]=0 v8) ==== 112+0+2 (unknown 454381629 0 1988412176) 0x7fe32c005dd0 con 0x7fe3380a4600 2024-08-19T09:49:58.298 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.293+0000 7fe3357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f", "id": 5}] 2024-08-19T09:49:58.298 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.293+0000 7fe3357fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:58.298 INFO:teuthology.orchestra.run.smithi082.stdout:5 2024-08-19T09:49:58.299 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 10 librados: watch_flush enter 2024-08-19T09:49:58.299 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 10 librados: watch_flush exit 2024-08-19T09:49:58.299 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 20 client.4174.objecter shutdown clearing up homeless session... 2024-08-19T09:49:58.299 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 10 client.4174.objecter successfully canceled tick 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 10 monclient: shutdown 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.294+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 >> v1:172.21.15.82:6792/0 conn(0x7fe3380a4600 legacy=0x7fe3380a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.295+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 shutdown_connections 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.295+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 >> 172.21.15.82:0/1973366434 conn(0x7fe3380a56f0 msgr2=0x7fe3380a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.295+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 shutdown_connections 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.295+0000 7fe33ed88640 1 -- 172.21.15.82:0/1973366434 wait complete. 2024-08-19T09:49:58.300 INFO:teuthology.orchestra.run.smithi082.stderr:2024-08-19T09:49:58.295+0000 7fe33ed88640 1 librados: shutdown 2024-08-19T09:49:58.306 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-19T09:49:58.306 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-08-19T09:49:58.309 INFO:tasks.ceph.osd.0:Started 2024-08-19T09:49:58.309 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-19T09:49:58.309 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-08-19T09:49:58.312 INFO:tasks.ceph.osd.1:Started 2024-08-19T09:49:58.312 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-19T09:49:58.312 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-08-19T09:49:58.314 INFO:tasks.ceph.osd.2:Started 2024-08-19T09:49:58.314 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-19T09:49:58.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-08-19T09:49:58.316 INFO:tasks.ceph.osd.3:Started 2024-08-19T09:49:58.316 INFO:tasks.ceph.osd.4:Restarting daemon 2024-08-19T09:49:58.317 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-08-19T09:49:58.320 INFO:tasks.ceph.osd.4:Started 2024-08-19T09:49:58.320 INFO:tasks.ceph.osd.5:Restarting daemon 2024-08-19T09:49:58.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-08-19T09:49:58.322 INFO:tasks.ceph.osd.5:Started 2024-08-19T09:49:58.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T09:49:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.434+0000 7ff6a5c60640 10 monclient: get_monmap_and_config 2024-08-19T09:49:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.434+0000 7ff6a5c60640 10 monclient: build_initial_monmap 2024-08-19T09:49:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.434+0000 7ff6a5c60640 10 monclient: monmap: 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:58.435714+0000 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:58.435714+0000 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.435+0000 7ff6a5c60640 1 Processor -- start 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.435+0000 7ff6a5c60640 1 -- start start 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.435+0000 7ff6a5c60640 10 monclient: init 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient: _add_conns ranks=[3,4,0,5,7,6,1,2,8] 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient(hunting): picked mon.noname-g con 0x7ff6a0068d60 addr v1:172.21.15.82:6790/0 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a00a8450 addr v1:172.21.15.1:6791/0 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a0143e00 addr v1:172.21.15.1:6789/0 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00a7ad0 con 0x7ff6a0143e00 2024-08-19T09:49:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00a7650 con 0x7ff6a0068d60 2024-08-19T09:49:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00a5d90 con 0x7ff6a00a8450 2024-08-19T09:49:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff6a5c60640 10 monclient(hunting): authenticate will time out at 654.512151s 2024-08-19T09:49:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff696ffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff6a00a8450 0x7ff6a00a6f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:51894/0 (socket says 172.21.15.1:51894) 2024-08-19T09:49:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.436+0000 7ff696ffd640 1 -- 172.21.15.1:0/3020500694 learned_addr learned my addr 172.21.15.1:0/3020500694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff696ffd640 1 -- 172.21.15.1:0/3020500694 >> v1:172.21.15.1:6791/0 conn(0x7ff6a00a8450 legacy=0x7ff6a00a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69ffff640 1 -- 172.21.15.1:0/3020500694 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0143e00 legacy=0x7ff6a01461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69f7fe640 1 -- 172.21.15.1:0/3020500694 >> v1:172.21.15.82:6790/0 conn(0x7ff6a0068d60 legacy=0x7ff6a00a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322791298 0 0) 0x7ff6a00a5d90 con 0x7ff6a00a8450 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): my global_id is 4153 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff68c003520 con 0x7ff6a00a8450 2024-08-19T09:49:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109992205 0 0) 0x7ff6a00a7ad0 con 0x7ff6a0143e00 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): my global_id is 4167 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a00a5d90 con 0x7ff6a0143e00 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772519266 0 0) 0x7ff6a00a7650 con 0x7ff6a0068d60 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient(hunting): my global_id is 4179 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a00a7ad0 con 0x7ff6a0068d60 2024-08-19T09:49:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2057372088 0 0) 0x7ff6a00a7ad0 con 0x7ff6a0068d60 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 >> v1:172.21.15.1:6791/0 conn(0x7ff6a00a8450 legacy=0x7ff6a00a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0143e00 legacy=0x7ff6a01461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient: _finish_hunting 0 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 monclient: found mon.noname-g 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a0069740 con 0x7ff6a0068d60 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient: _finish_auth 0 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient: _check_auth_tickets 2024-08-19T09:49:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff690001a50 con 0x7ff6a0068d60 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.438+0000 7ff69effd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 10 monclient: dump: 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.451 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff688001ae0 con 0x7ff6a00a8450 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 5 monclient: authenticate success, global_id 4179 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff690001ca0 con 0x7ff6a0068d60 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 1 -- 172.21.15.1:0/3020500694 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff690003950 con 0x7ff6a0068d60 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:49:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff69effd640 10 monclient: dump: 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 4 set_mon_vals no callback set 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 10 monclient: shutdown 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3020500694 >> v1:172.21.15.82:6790/0 conn(0x7ff6a0068d60 legacy=0x7ff6a00a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3020500694 shutdown_connections 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3020500694 >> 172.21.15.1:0/3020500694 conn(0x7ff6a00a9210 msgr2=0x7ff6a0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3020500694 shutdown_connections 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.439+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3020500694 wait complete. 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 10 monclient: build_initial_monmap 2024-08-19T09:49:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 10 monclient: monmap: 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 1 librados: starting msgr at 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 1 librados: starting objecter 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 1 Processor -- start 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 1 -- start start 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 1 librados: setting wanted keys 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 1 librados: calling monclient init 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.440+0000 7ff6a5c60640 10 monclient: init 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient: _add_conns ranks=[0,8,5,2,6,4,1,7,3] 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a0068d60 addr v1:172.21.15.1:6789/0 2024-08-19T09:49:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient(hunting): picked mon.noname-e con 0x7ff6a00a8450 addr v1:172.21.15.1:6793/0 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient(hunting): picked mon.noname-h con 0x7ff6a0143e00 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a0072680 con 0x7ff6a0068d60 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00a9d30 con 0x7ff6a0143e00 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a009f910 con 0x7ff6a00a8450 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff6a5c60640 10 monclient(hunting): authenticate will time out at 654.517407s 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff69f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 0x7ff6a0070e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36808/0 (socket says 172.21.15.1:36808) 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.441+0000 7ff69f7fe640 1 -- 172.21.15.1:0/3846044891 learned_addr learned my addr 172.21.15.1:0/3846044891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69f7fe640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 legacy=0x7ff6a0070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 client.?.objecter ms_handle_connect 0x7ff6a0068d60 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff696ffd640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6793/0 conn(0x7ff6a00a8450 legacy=0x7ff6a0071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 client.?.objecter ms_handle_connect 0x7ff6a00a8450 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 client.?.objecter ms_handle_connect 0x7ff6a0143e00 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69ffff640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.82:6791/0 conn(0x7ff6a0143e00 legacy=0x7ff6a01da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951612368 0 0) 0x7ff6a0072680 con 0x7ff6a0068d60 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): my global_id is 4176 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff680004eb0 con 0x7ff6a0068d60 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331769957 0 0) 0x7ff6a009f910 con 0x7ff6a00a8450 2024-08-19T09:49:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): my global_id is 4139 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a0072680 con 0x7ff6a00a8450 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69f7fe640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 legacy=0x7ff6a0070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69f7fe640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 legacy=0x7ff6a0070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69f7fe640 1 --1- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 0x7ff6a0070e30 :-1 s=OPENED pgs=41 cs=1 l=1).handle_message read tag failed 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69f7fe640 1 --1- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 0x7ff6a0070e30 :-1 s=OPENED pgs=41 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6789/0 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 703696876 0 0) 0x7ff6a00a9d30 con 0x7ff6a0143e00 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 10 monclient(hunting): my global_id is 4172 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.443+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a009f910 con 0x7ff6a0143e00 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1149800500 0 0) 0x7ff6a0072680 con 0x7ff6a00a8450 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6a00a9d30 con 0x7ff6a00a8450 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff6880047e0 con 0x7ff6a00a8450 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): dump: 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.461 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2197359725 0 0) 0x7ff6a009f910 con 0x7ff6a0143e00 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6a0072680 con 0x7ff6a0143e00 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff698003990 con 0x7ff6a0143e00 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient(hunting): dump: 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1811267560 0 0) 0x7ff6a00a9d30 con 0x7ff6a00a8450 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.82:6791/0 conn(0x7ff6a0143e00 legacy=0x7ff6a01da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6789/0 conn(0x7ff6a0068d60 legacy=0x7ff6a0070e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:49:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: _finish_hunting 0 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 monclient: found mon.e 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a009f070 con 0x7ff6a00a8450 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: _finish_auth 0 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: _check_auth_tickets 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 5 monclient: authenticate success, global_id 4139 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 10 monclient: _renew_subs 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a009e110 con 0x7ff6a00a8450 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 10 monclient: _renew_subs 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6a0071c50 con 0x7ff6a00a8450 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff6a5c60640 1 librados: init done 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff688004c30 con 0x7ff6a00a8450 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff688005120 con 0x7ff6a00a8450 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69e7fc640 4 set_mon_vals no callback set 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 monclient: dump: 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:49:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff688005750 con 0x7ff6a00a8450 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 603733623 0 0) 0x7ff688006ff0 con 0x7ff6a00a8450 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 10 client.4139.objecter ms_dispatch 0x7ff6a0000b90 osd_map(8..8 src has 1..8) 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 3 client.4139.objecter handle_osd_map got epochs [8,8] > 0 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 3 client.4139.objecter handle_osd_map decoding full epoch 8 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.444+0000 7ff69cff9640 20 client.4139.objecter dump_active .. 0 homeless 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.445+0000 7ff6a5c60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.445+0000 7ff6a5c60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.445+0000 7ff6a5c60640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.445+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff684005080 con 0x7ff6a00a8450 2024-08-19T09:49:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.447+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7ff6880069c0 con 0x7ff6a00a8450 2024-08-19T09:49:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.447+0000 7ff69cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.447+0000 7ff69cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.508+0000 7ff6a5c60640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.508+0000 7ff6a5c60640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.508+0000 7ff6a5c60640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.508+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff684005370 con 0x7ff6a00a8450 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff69cff9640 1 -- 172.21.15.1:0/3846044891 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 3449583846) 0x7ff688006d70 con 0x7ff6a00a8450 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff69cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:49:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff69cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:49:58.511 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:49:58.291381+0000","last_up_change":"0.000000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:49:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 10 librados: watch_flush enter 2024-08-19T09:49:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 10 librados: watch_flush exit 2024-08-19T09:49:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 20 client.4139.objecter shutdown clearing up homeless session... 2024-08-19T09:49:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 10 client.4139.objecter successfully canceled tick 2024-08-19T09:49:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 10 monclient: shutdown 2024-08-19T09:49:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 >> v1:172.21.15.1:6793/0 conn(0x7ff6a00a8450 legacy=0x7ff6a0071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 shutdown_connections 2024-08-19T09:49:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.509+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 >> 172.21.15.1:0/3846044891 conn(0x7ff6a00a9210 msgr2=0x7ff6a00a6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.510+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 shutdown_connections 2024-08-19T09:49:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.510+0000 7ff6a5c60640 1 -- 172.21.15.1:0/3846044891 wait complete. 2024-08-19T09:49:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:49:58.510+0000 7ff6a5c60640 1 librados: shutdown 2024-08-19T09:49:58.520 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-19T09:49:58.520 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-19T09:49:59.125 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-19T09:49:59.217 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.216+0000 7f9514ada640 10 monclient: get_monmap_and_config 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.216+0000 7f9514ada640 10 monclient: build_initial_monmap 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.216+0000 7f9514ada640 10 monclient: monmap: 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:59.217768+0000 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:59.217768+0000 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:59.218 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:59.219 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:59.219 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:59.219 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:59.219 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:59.219 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:59.219 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.217+0000 7f9514ada640 1 Processor -- start 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.217+0000 7f9514ada640 1 -- start start 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.217+0000 7f9514ada640 10 monclient: init 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient: _add_conns ranks=[1,3,6,2,8,0,4,5,7] 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient(hunting): picked mon.noname-f con 0x7f95100a6720 addr v1:172.21.15.82:6789/0 2024-08-19T09:49:59.220 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient(hunting): picked mon.noname-g con 0x7f951006eee0 addr v1:172.21.15.82:6790/0 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient(hunting): picked mon.noname-d con 0x7f95100a7210 addr v1:172.21.15.1:6792/0 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95100a3220 con 0x7f95100a6720 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95100a3ac0 con 0x7f951006eee0 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f951006e9e0 con 0x7f95100a7210 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f9514ada640 10 monclient(hunting): authenticate will time out at 655.294136s 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f950ed9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f95100a7210 0x7f9510141e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:49692/0 (socket says 172.21.15.1:49692) 2024-08-19T09:49:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.218+0000 7f950ed9d640 1 -- 172.21.15.1:0/4112851709 learned_addr learned my addr 172.21.15.1:0/4112851709 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:49:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950ed9d640 1 -- 172.21.15.1:0/4112851709 >> v1:172.21.15.1:6792/0 conn(0x7f95100a7210 legacy=0x7f9510141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950dd9b640 1 -- 172.21.15.1:0/4112851709 >> v1:172.21.15.82:6790/0 conn(0x7f951006eee0 legacy=0x7f951013e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950e59c640 1 -- 172.21.15.1:0/4112851709 >> v1:172.21.15.82:6789/0 conn(0x7f95100a6720 legacy=0x7f95100a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577121769 0 0) 0x7f951006e9e0 con 0x7f95100a7210 2024-08-19T09:49:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): my global_id is 4155 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f8003520 con 0x7f95100a7210 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458100231 0 0) 0x7f95100a3ac0 con 0x7f951006eee0 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): my global_id is 4188 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f951006e9e0 con 0x7f951006eee0 2024-08-19T09:49:59.226 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306768164 0 0) 0x7f95100a3220 con 0x7f95100a6720 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient(hunting): my global_id is 4141 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95100a3ac0 con 0x7f95100a6720 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1581603026 0 0) 0x7f94f8003520 con 0x7f95100a7210 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 >> v1:172.21.15.82:6790/0 conn(0x7f951006eee0 legacy=0x7f951013e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 >> v1:172.21.15.82:6789/0 conn(0x7f95100a6720 legacy=0x7f95100a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: _finish_hunting 0 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 monclient: found mon.noname-d 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:49:59.227 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100a44f0 con 0x7f95100a7210 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: _finish_auth 0 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: _check_auth_tickets 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9508001e60 con 0x7f95100a7210 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f9514ada640 5 monclient: authenticate success, global_id 4155 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f9514ada640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f950d59a640 10 monclient: dump: 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.228 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.220+0000 7f9514ada640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:49:59.229 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9508004c20 con 0x7f95100a7210 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f950d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f950d59a640 1 -- 172.21.15.1:0/4112851709 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95080051d0 con 0x7f95100a7210 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 10 monclient: get_monmap_and_config success 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 4 set_mon_vals no callback set 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f950d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f950d59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f950d59a640 10 monclient: dump: 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.230 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 10 monclient: shutdown 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 1 -- 172.21.15.1:0/4112851709 >> v1:172.21.15.1:6792/0 conn(0x7f95100a7210 legacy=0x7f9510141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 1 -- 172.21.15.1:0/4112851709 shutdown_connections 2024-08-19T09:49:59.231 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 1 -- 172.21.15.1:0/4112851709 >> 172.21.15.1:0/4112851709 conn(0x7f95100a8820 msgr2=0x7f95100a8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 1 -- 172.21.15.1:0/4112851709 shutdown_connections 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.221+0000 7f9514ada640 1 -- 172.21.15.1:0/4112851709 wait complete. 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 10 monclient: build_initial_monmap 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 10 monclient: monmap: 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:49:59.232 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 1 librados: starting msgr at 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 1 librados: starting objecter 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 1 Processor -- start 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 1 -- start start 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 1 librados: setting wanted keys 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 1 librados: calling monclient init 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.222+0000 7f9514ada640 10 monclient: init 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient: _reopen_session rank -1 2024-08-19T09:49:59.233 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient: _add_conns ranks=[4,2,5,0,7,3,1,8,6] 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient(hunting): picked mon.noname-c con 0x7f951006eee0 addr v1:172.21.15.1:6791/0 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient(hunting): picked mon.noname-b con 0x7f95100a6720 addr v1:172.21.15.1:6790/0 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient(hunting): picked mon.noname-h con 0x7f95100a7210 addr v1:172.21.15.82:6791/0 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f951006e1c0 con 0x7f95100a6720 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9510129770 con 0x7f951006eee0 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9510129a50 con 0x7f95100a7210 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient(hunting): _renew_subs 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f9514ada640 10 monclient(hunting): authenticate will time out at 655.299261s 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f950e59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f951006eee0 0x7f9510125cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:51906/0 (socket says 172.21.15.1:51906) 2024-08-19T09:49:59.234 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.223+0000 7f950e59c640 1 -- 172.21.15.1:0/3119267410 learned_addr learned my addr 172.21.15.1:0/3119267410 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:49:59.238 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f950dd9b640 1 -- 172.21.15.1:0/3119267410 >> v1:172.21.15.1:6790/0 conn(0x7f95100a6720 legacy=0x7f9510126400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:59.238 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 client.?.objecter ms_handle_connect 0x7f95100a6720 2024-08-19T09:49:59.238 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:59.238 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f950e59c640 1 -- 172.21.15.1:0/3119267410 >> v1:172.21.15.1:6791/0 conn(0x7f951006eee0 legacy=0x7f9510125cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 client.?.objecter ms_handle_connect 0x7f951006eee0 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f950ed9d640 1 -- 172.21.15.1:0/3119267410 >> v1:172.21.15.82:6791/0 conn(0x7f95100a7210 legacy=0x7f95101d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 client.?.objecter ms_handle_connect 0x7f95100a7210 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736859048 0 0) 0x7f951006e9e0 con 0x7f95100a6720 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:59.239 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): my global_id is 4160 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f0004ee0 con 0x7f95100a6720 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825967964 0 0) 0x7f9510129770 con 0x7f951006eee0 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): my global_id is 4162 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f951006e9e0 con 0x7f951006eee0 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360839294 0 0) 0x7f9510129a50 con 0x7f95100a7210 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _negotiate 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:49:59.240 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): my global_id is 4181 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9510129770 con 0x7f95100a7210 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2158341684 0 0) 0x7f94f0004ee0 con 0x7f95100a6720 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9510129a50 con 0x7f95100a6720 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9504003610 con 0x7f95100a6720 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): dump: 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:59.241 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.225+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3766190433 0 0) 0x7f951006e9e0 con 0x7f951006eee0 2024-08-19T09:49:59.242 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94f0004ee0 con 0x7f951006eee0 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 189302848 0 0) 0x7f9510129770 con 0x7f95100a7210 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f951006e9e0 con 0x7f95100a7210 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9500001d40 con 0x7f951006eee0 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient(hunting): dump: 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:59.243 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95080020b0 con 0x7f95100a7210 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient(hunting): dump: 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:49:59.244 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:59.245 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2977118723 0 0) 0x7f9510129a50 con 0x7f95100a6720 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 >> v1:172.21.15.82:6791/0 conn(0x7f95100a7210 legacy=0x7f95101d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 >> v1:172.21.15.1:6791/0 conn(0x7f951006eee0 legacy=0x7f9510125cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: _finish_hunting 0 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 monclient: found mon.b 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9510128be0 con 0x7f95100a6720 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: _finish_auth 0 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: _check_auth_tickets 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:49:59.246 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 5 monclient: authenticate success, global_id 4160 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 10 monclient: _renew_subs 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f95101276f0 con 0x7f95100a6720 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 10 monclient: _renew_subs 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f95101284c0 con 0x7f95100a6720 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f9514ada640 1 librados: init done 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9504003b60 con 0x7f95100a6720 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9504004050 con 0x7f95100a6720 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:49:59.247 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f950cd99640 4 set_mon_vals no callback set 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 monclient: dump: 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:49:59.248 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9504005b50 con 0x7f95100a6720 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 603733623 0 0) 0x7f9504007070 con 0x7f95100a6720 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 10 client.4160.objecter ms_dispatch 0x7f9510000b90 osd_map(8..8 src has 1..8) 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 3 client.4160.objecter handle_osd_map got epochs [8,8] > 0 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 3 client.4160.objecter handle_osd_map decoding full epoch 8 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.226+0000 7f94feffd640 20 client.4160.objecter dump_active .. 0 homeless 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.227+0000 7f9514ada640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.227+0000 7f9514ada640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.227+0000 7f9514ada640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.227+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f4005080 con 0x7f95100a6720 2024-08-19T09:49:59.249 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.229+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f951006e1c0 con 0x7f95100a6720 2024-08-19T09:49:59.250 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.229+0000 7f94feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:49:59.250 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.229+0000 7f94feffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f9514ada640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f9514ada640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f9514ada640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94f4005370 con 0x7f95100a6720 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f94feffd640 1 -- 172.21.15.1:0/3119267410 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 3449583846) 0x7f9504006b90 con 0x7f95100a6720 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f94feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:49:59.291 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.290+0000 7f94feffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:49:59.292 INFO:teuthology.misc.health.smithi001.stdout: 2024-08-19T09:49:59.292 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":8,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:49:58.291381+0000","last_up_change":"0.000000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 10 librados: watch_flush enter 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 10 librados: watch_flush exit 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 20 client.4160.objecter shutdown clearing up homeless session... 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 10 client.4160.objecter successfully canceled tick 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 10 monclient: shutdown 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 >> v1:172.21.15.1:6790/0 conn(0x7f95100a6720 legacy=0x7f9510126400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 shutdown_connections 2024-08-19T09:49:59.293 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 >> 172.21.15.1:0/3119267410 conn(0x7f95100a8820 msgr2=0x7f9510142cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:49:59.294 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 shutdown_connections 2024-08-19T09:49:59.294 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 1 -- 172.21.15.1:0/3119267410 wait complete. 2024-08-19T09:49:59.294 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:49:59.291+0000 7f9514ada640 1 librados: shutdown 2024-08-19T09:49:59.300 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-08-19T09:49:59.520 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-08-19T09:49:59.519+0000 7f568de0d640 -1 Falling back to public interface 2024-08-19T09:49:59.523 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-08-19T09:49:59.522+0000 7fc36fb01640 -1 Falling back to public interface 2024-08-19T09:49:59.782 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-08-19T09:49:59.781+0000 7fbab860c640 -1 Falling back to public interface 2024-08-19T09:50:00.008 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-08-19T09:50:00.006+0000 7fb5eb020640 -1 Falling back to public interface 2024-08-19T09:50:00.047 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-08-19T09:50:00.046+0000 7f9dde00e640 -1 Falling back to public interface 2024-08-19T09:50:00.051 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:00.050+0000 7f0f2da0d640 -1 Falling back to public interface 2024-08-19T09:50:02.415 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.414+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-08-19T09:50:02.417 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.416+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-08-19T09:50:02.417 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.416+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-08-19T09:50:02.418 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.417+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-08-19T09:50:02.420 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.419+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-08-19T09:50:02.420 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.419+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-08-19T09:50:02.420 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.419+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-19T09:50:02.467 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-08-19T09:50:02.466+0000 7f9460e27640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-19T09:50:03.341 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-08-19T09:50:03.340+0000 7f568de0d640 -1 osd.1 0 log_to_monitors true 2024-08-19T09:50:03.566 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-08-19T09:50:03.565+0000 7fc36fb01640 -1 osd.0 0 log_to_monitors true 2024-08-19T09:50:03.622 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:03.618+0000 7f0f2da0d640 -1 osd.5 0 log_to_monitors true 2024-08-19T09:50:03.778 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-08-19T09:50:03.777+0000 7fb5eb020640 -1 osd.2 0 log_to_monitors true 2024-08-19T09:50:03.830 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-08-19T09:50:03.829+0000 7fbab860c640 -1 osd.4 0 log_to_monitors true 2024-08-19T09:50:04.044 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-08-19T09:50:04.043+0000 7f9dde00e640 -1 osd.3 0 log_to_monitors true 2024-08-19T09:50:04.414 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-08-19T09:50:04.413+0000 7f56897ff640 -1 osd.1 0 waiting for initial osdmap 2024-08-19T09:50:04.421 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-08-19T09:50:04.420+0000 7f5685626640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:05.425 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:05.423+0000 7f0f29451640 -1 osd.5 0 waiting for initial osdmap 2024-08-19T09:50:05.504 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:05.503+0000 7f0f25278640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:05.577 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:05.576+0000 7f0f20d61640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:05.578 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:05.577+0000 7f0f20d61640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:05.578 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T09:50:05.577+0000 7f0f20d61640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:05.905 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-19T09:50:06.037 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.035+0000 7fee00e4f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:06.037 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.035+0000 7fee00e4f640 10 monclient: build_initial_monmap 2024-08-19T09:50:06.037 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.035+0000 7fee00e4f640 10 monclient: monmap: 2024-08-19T09:50:06.037 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-08-19T09:50:06.037 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:06.037 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:50:06.036868+0000 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:50:06.036868+0000 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:06.038 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:06.039 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.036+0000 7fee00e4f640 1 Processor -- start 2024-08-19T09:50:06.039 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.036+0000 7fee00e4f640 1 -- start start 2024-08-19T09:50:06.039 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.036+0000 7fee00e4f640 10 monclient: init 2024-08-19T09:50:06.039 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient: _add_conns ranks=[2,7,5,4,3,1,6,0,8] 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient(hunting): picked mon.noname-b con 0x7fedfc09da50 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient(hunting): picked mon.noname-i con 0x7fedfc09e540 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient(hunting): picked mon.noname-h con 0x7fedfc0a1720 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedfc09f8f0 con 0x7fedfc09da50 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedfc09f410 con 0x7fedfc0a1720 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fedfa59c640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fedfc09da50 0x7fedfc09de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51292/0 (socket says 172.21.15.1:51292) 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fedfa59c640 1 -- 172.21.15.1:0/757282419 learned_addr learned my addr 172.21.15.1:0/757282419 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 1 -- 172.21.15.1:0/757282419 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedfc09eb10 con 0x7fedfc09e540 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:06.040 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.037+0000 7fee00e4f640 10 monclient(hunting): authenticate will time out at 662.113476s 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedfa59c640 1 -- 172.21.15.1:0/757282419 >> v1:172.21.15.1:6790/0 conn(0x7fedfc09da50 legacy=0x7fedfc09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedfad9d640 1 -- 172.21.15.1:0/757282419 >> v1:172.21.15.82:6791/0 conn(0x7fedfc0a1720 legacy=0x7fedfc13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf9d9b640 1 -- 172.21.15.1:0/757282419 >> v1:172.21.15.82:6792/0 conn(0x7fedfc09e540 legacy=0x7fedfc13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786695172 0 0) 0x7fedf4001820 con 0x7fedfc0a1720 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): _negotiate 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): my global_id is 4235 2024-08-19T09:50:06.043 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fede4003520 con 0x7fedfc0a1720 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376124310 0 0) 0x7fedfc09f8f0 con 0x7fedfc09da50 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): _negotiate 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.039+0000 7fedf959a640 10 monclient(hunting): my global_id is 4205 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf4001820 con 0x7fedfc09da50 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1614209948 0 0) 0x7fede4003520 con 0x7fedfc0a1720 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 >> v1:172.21.15.82:6792/0 conn(0x7fedfc09e540 legacy=0x7fedfc13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 >> v1:172.21.15.1:6790/0 conn(0x7fedfc09da50 legacy=0x7fedfc09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: _finish_hunting 0 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 monclient: found mon.noname-h 2024-08-19T09:50:06.044 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedfc0a0d50 con 0x7fedfc0a1720 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: _finish_auth 0 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: _check_auth_tickets 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedf4002080 con 0x7fedfc0a1720 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fee00e4f640 5 monclient: authenticate success, global_id 4235 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fee00e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: dump: 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:06.045 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:06.046 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fee00e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedf4003a80 con 0x7fedfc0a1720 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 1 -- 172.21.15.1:0/757282419 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedf4003fb0 con 0x7fedfc0a1720 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fedf959a640 10 monclient: dump: 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.047 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:06.048 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fee00e4f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.040+0000 7fee00e4f640 4 set_mon_vals no callback set 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.041+0000 7fee00e4f640 10 monclient: shutdown 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.041+0000 7fee00e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.041+0000 7fee00e4f640 1 -- 172.21.15.1:0/757282419 >> v1:172.21.15.82:6791/0 conn(0x7fedfc0a1720 legacy=0x7fedfc13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.041+0000 7fee00e4f640 1 -- 172.21.15.1:0/757282419 shutdown_connections 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.041+0000 7fee00e4f640 1 -- 172.21.15.1:0/757282419 >> 172.21.15.1:0/757282419 conn(0x7fedfc0a7f70 msgr2=0x7fedfc0a2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:06.049 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.041+0000 7fee00e4f640 1 -- 172.21.15.1:0/757282419 shutdown_connections 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.044+0000 7fee00e4f640 1 -- 172.21.15.1:0/757282419 wait complete. 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.045+0000 7fee00e4f640 10 monclient: build_initial_monmap 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.045+0000 7fee00e4f640 10 monclient: monmap: 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:06.054 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:06.055 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.045+0000 7fee00e4f640 1 librados: starting msgr at 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.045+0000 7fee00e4f640 1 librados: starting objecter 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.046+0000 7fee00e4f640 1 Processor -- start 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.046+0000 7fee00e4f640 1 -- start start 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.046+0000 7fee00e4f640 1 librados: setting wanted keys 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.046+0000 7fee00e4f640 1 librados: calling monclient init 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.046+0000 7fee00e4f640 10 monclient: init 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient: _add_conns ranks=[3,0,6,1,2,4,8,7,5] 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient(hunting): picked mon.noname-g con 0x7fedfc09da50 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient(hunting): picked mon.noname-a con 0x7fedfc09e540 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient(hunting): picked mon.noname-d con 0x7fedfc0a1720 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedfc120f40 con 0x7fedfc09e540 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedfc122ce0 con 0x7fedfc09da50 2024-08-19T09:50:06.056 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedfc122a00 con 0x7fedfc0a1720 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fee00e4f640 10 monclient(hunting): authenticate will time out at 662.123474s 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fedfa59c640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fedfc09da50 0x7fedfc1337c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:43552/0 (socket says 172.21.15.1:43552) 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.047+0000 7fedfa59c640 1 -- 172.21.15.1:0/3532773660 learned_addr learned my addr 172.21.15.1:0/3532773660 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.049+0000 7fedfad9d640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.1:6792/0 conn(0x7fedfc0a1720 legacy=0x7fedfc1ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.049+0000 7fedfa59c640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.82:6790/0 conn(0x7fedfc09da50 legacy=0x7fedfc1337c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.049+0000 7fedf9d9b640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.1:6789/0 conn(0x7fedfc09e540 legacy=0x7fedfc1c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 client.?.objecter ms_handle_connect 0x7fedfc0a1720 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 client.?.objecter ms_handle_connect 0x7fedfc09da50 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:06.057 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 client.?.objecter ms_handle_connect 0x7fedfc09e540 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428055104 0 0) 0x7fedfc120f40 con 0x7fedfc09e540 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): my global_id is 4230 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feddc004ee0 con 0x7fedfc09e540 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443959051 0 0) 0x7fedfc122a00 con 0x7fedfc0a1720 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): my global_id is 4191 2024-08-19T09:50:06.058 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedfc120f40 con 0x7fedfc0a1720 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423550565 0 0) 0x7fedfc122ce0 con 0x7fedfc09da50 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): my global_id is 4215 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedfc122a00 con 0x7fedfc09da50 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 150167559 0 0) 0x7fedfc120f40 con 0x7fedfc0a1720 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedfc122ce0 con 0x7fedfc0a1720 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedf4003f70 con 0x7fedfc0a1720 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.050+0000 7fedeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:06.059 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient(hunting): dump: 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:06.060 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2860999492 0 0) 0x7feddc004ee0 con 0x7fedfc09e540 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedfc120f40 con 0x7fedfc09e540 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2381049255 0 0) 0x7fedfc122ce0 con 0x7fedfc0a1720 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.82:6790/0 conn(0x7fedfc09da50 legacy=0x7fedfc1337c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.1:6789/0 conn(0x7fedfc09e540 legacy=0x7fedfc1c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 monclient: found mon.d 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:06.061 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedfc133ed0 con 0x7fedfc0a1720 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedf4002060 con 0x7fedfc0a1720 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedf4004800 con 0x7fedfc0a1720 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedeaffd640 10 monclient: dump: 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:06.062 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:06.063 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fee00e4f640 5 monclient: authenticate success, global_id 4191 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fee00e4f640 10 monclient: _renew_subs 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fee00e4f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedfc1d2dd0 con 0x7fedfc0a1720 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.051+0000 7fedf8d99640 4 set_mon_vals no callback set 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.052+0000 7fee00e4f640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.052+0000 7fee00e4f640 10 monclient: _renew_subs 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.052+0000 7fee00e4f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.052+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 4) ==== 49789+0+0 (unknown 2883679630 0 0) 0x7fedf4011dd0 con 0x7fedfc0a1720 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.052+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fedfc1d3180 con 0x7fedfc0a1720 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.053+0000 7fee00e4f640 1 librados: init done 2024-08-19T09:50:06.064 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.053+0000 7fedfa59c640 10 monclient: get_auth_request con 0x7feddc0413d0 auth_method 0 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.053+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(11..11 src has 1..11) ==== 2067+0+0 (unknown 3569360922 0 0) 0x7fedf404d670 con 0x7fedfc0a1720 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fedeaffd640 10 client.4191.objecter ms_dispatch 0x7fedfc000b90 osd_map(11..11 src has 1..11) 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fedeaffd640 3 client.4191.objecter handle_osd_map got epochs [11,11] > 0 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fedeaffd640 3 client.4191.objecter handle_osd_map decoding full epoch 11 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fedeaffd640 20 client.4191.objecter dump_active .. 0 homeless 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fee00e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fee00e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fee00e4f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.054+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedfc06b9e0 con 0x7fedfc0a1720 2024-08-19T09:50:06.066 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.055+0000 7fedeaffd640 10 client.4191.objecter ms_handle_connect 0x7feddc0413d0 2024-08-19T09:50:06.067 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.063+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fedfc09f410 con 0x7fedfc0a1720 2024-08-19T09:50:06.067 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.063+0000 7fedeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:06.067 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.063+0000 7fedeaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:06.336 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-08-19T09:50:06.334+0000 7fc36c2ab640 -1 osd.0 0 waiting for initial osdmap 2024-08-19T09:50:06.340 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-08-19T09:50:06.339+0000 7fc3638b6640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:06.361 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-08-19T09:50:06.360+0000 7fb5e6917640 -1 osd.2 0 waiting for initial osdmap 2024-08-19T09:50:06.368 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-08-19T09:50:06.368+0000 7fb5de736640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:06.393 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fee00e4f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:06.393 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fee00e4f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:06.393 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fee00e4f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:06.393 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fedfc0a4bb0 con 0x7fedfc0a1720 2024-08-19T09:50:06.394 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fedeaffd640 1 -- 172.21.15.1:0/3532773660 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+5855 (unknown 238102980 0 4204500561) 0x7fedfc0a4bb0 con 0x7fedfc0a1720 2024-08-19T09:50:06.394 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fedeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:06.394 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.392+0000 7fedeaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:06.394 INFO:teuthology.misc.health.smithi001.stdout: 2024-08-19T09:50:06.394 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:50:05.407784+0000","last_up_change":"2024-08-19T09:50:05.407784+0000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1668590231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1668590231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1668590231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1668590231}]},"public_addr":"172.21.15.1:6800/1668590231","cluster_addr":"172.21.15.1:6801/1668590231","heartbeat_back_addr":"172.21.15.1:6803/1668590231","heartbeat_front_addr":"172.21.15.1:6802/1668590231","state":["exists","up"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 10 librados: watch_flush enter 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 10 librados: watch_flush exit 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 20 client.4191.objecter shutdown clearing up homeless session... 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 10 client.4191.objecter successfully canceled tick 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.82:6812/2549104415 conn(0x7feddc0413d0 legacy=0x7feddc043890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 10 monclient: shutdown 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:06.396 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.394+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 >> v1:172.21.15.1:6792/0 conn(0x7fedfc0a1720 legacy=0x7fedfc1ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:06.397 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.395+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 shutdown_connections 2024-08-19T09:50:06.397 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.395+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 >> 172.21.15.1:0/3532773660 conn(0x7fedfc0a7f70 msgr2=0x7fedfc0a5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:06.397 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.395+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 shutdown_connections 2024-08-19T09:50:06.397 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.395+0000 7fee00e4f640 1 -- 172.21.15.1:0/3532773660 wait complete. 2024-08-19T09:50:06.397 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:06.395+0000 7fee00e4f640 1 librados: shutdown 2024-08-19T09:50:06.405 DEBUG:teuthology.misc:1 of 6 OSDs are up 2024-08-19T09:50:06.641 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-08-19T09:50:06.640+0000 7f9dda0b1640 -1 osd.3 0 waiting for initial osdmap 2024-08-19T09:50:06.647 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-08-19T09:50:06.645+0000 7f9dd16bc640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:06.690 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-08-19T09:50:06.689+0000 7fbab4038640 -1 osd.4 0 waiting for initial osdmap 2024-08-19T09:50:06.694 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-08-19T09:50:06.693+0000 7fbaabe57640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T09:50:13.011 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.113+0000 7f75611fd640 10 monclient: get_monmap_and_config 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.113+0000 7f75611fd640 10 monclient: build_initial_monmap 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.113+0000 7f75611fd640 10 monclient: monmap: 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:50:13.114148+0000 2024-08-19T09:50:13.114 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:50:13.114148+0000 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:13.115 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.113+0000 7f75611fd640 1 Processor -- start 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.113+0000 7f75611fd640 1 -- start start 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.113+0000 7f75611fd640 10 monclient: init 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient: _add_conns ranks=[1,2,4,3,7,0,5,6,8] 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient(hunting): picked mon.noname-f con 0x7f755c0a6720 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient(hunting): picked mon.noname-b con 0x7f755c139bf0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient(hunting): picked mon.noname-c con 0x7f755c139fd0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755c0a2f20 con 0x7f755c0a6720 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755c073f00 con 0x7f755c139bf0 2024-08-19T09:50:13.117 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755c0a3220 con 0x7f755c139fd0 2024-08-19T09:50:13.118 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:13.118 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f75611fd640 10 monclient(hunting): authenticate will time out at 669.190420s 2024-08-19T09:50:13.118 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f755b59e640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f755c139fd0 0x7f755c141e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59806/0 (socket says 172.21.15.1:59806) 2024-08-19T09:50:13.118 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f755b59e640 1 -- 172.21.15.1:0/2163740637 learned_addr learned my addr 172.21.15.1:0/2163740637 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:13.118 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.114+0000 7f755a59c640 1 --1- 172.21.15.1:0/2163740637 >> v1:172.21.15.1:6790/0 conn(0x7f755c139bf0 0x7f755c13e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51566/0 (socket says 172.21.15.1:51566) 2024-08-19T09:50:13.125 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f755b59e640 1 -- 172.21.15.1:0/2163740637 >> v1:172.21.15.1:6791/0 conn(0x7f755c139fd0 legacy=0x7f755c141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.125 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f755a59c640 1 -- 172.21.15.1:0/2163740637 >> v1:172.21.15.1:6790/0 conn(0x7f755c139bf0 legacy=0x7f755c13e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f755ad9d640 1 -- 172.21.15.1:0/2163740637 >> v1:172.21.15.82:6789/0 conn(0x7f755c0a6720 legacy=0x7f755c1394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879210865 0 0) 0x7f755c0a3220 con 0x7f755c139fd0 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): my global_id is 4225 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7544003520 con 0x7f755c139fd0 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1624261129 0 0) 0x7f755c073f00 con 0x7f755c139bf0 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.126 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): my global_id is 4223 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f755c0a3220 con 0x7f755c139bf0 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531221488 0 0) 0x7f755c0a2f20 con 0x7f755c0a6720 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 10 monclient(hunting): my global_id is 4204 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.116+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f755c073f00 con 0x7f755c0a6720 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2435431896 0 0) 0x7f755c0a3220 con 0x7f755c139bf0 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 >> v1:172.21.15.1:6791/0 conn(0x7f755c139fd0 legacy=0x7f755c141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.127 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 >> v1:172.21.15.82:6789/0 conn(0x7f755c0a6720 legacy=0x7f755c1394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 monclient: found mon.noname-b 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755c074290 con 0x7f755c139bf0 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: _finish_auth 0 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f75500019d0 con 0x7f755c139bf0 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 5 monclient: authenticate success, global_id 4223 2024-08-19T09:50:13.128 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: dump: 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.129 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7550001ca0 con 0x7f755c139bf0 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 1 -- 172.21.15.1:0/2163740637 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7550004b20 con 0x7f755c139bf0 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:13.130 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f7559d9b640 10 monclient: dump: 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.131 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 4 set_mon_vals no callback set 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 10 monclient: shutdown 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 1 -- 172.21.15.1:0/2163740637 >> v1:172.21.15.1:6790/0 conn(0x7f755c139bf0 legacy=0x7f755c13e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 1 -- 172.21.15.1:0/2163740637 shutdown_connections 2024-08-19T09:50:13.132 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.117+0000 7f75611fd640 1 -- 172.21.15.1:0/2163740637 >> 172.21.15.1:0/2163740637 conn(0x7f755c0ac140 msgr2=0x7f755c0ac550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.118+0000 7f75611fd640 1 -- 172.21.15.1:0/2163740637 shutdown_connections 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.118+0000 7f75611fd640 1 -- 172.21.15.1:0/2163740637 wait complete. 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.118+0000 7f75611fd640 10 monclient: build_initial_monmap 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.118+0000 7f75611fd640 10 monclient: monmap: 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:13.133 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.118+0000 7f75611fd640 1 librados: starting msgr at 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.118+0000 7f75611fd640 1 librados: starting objecter 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 Processor -- start 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 -- start start 2024-08-19T09:50:13.134 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 librados: setting wanted keys 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 librados: calling monclient init 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient: init 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient: _add_conns ranks=[6,8,7,4,1,3,0,2,5] 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient(hunting): picked mon.noname-d con 0x7f755c0a6720 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient(hunting): picked mon.noname-e con 0x7f755c139bf0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient(hunting): picked mon.noname-i con 0x7f755c139fd0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755c13f9e0 con 0x7f755c0a6720 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755c13edf0 con 0x7f755c139fd0 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f755c1408b0 con 0x7f755c139bf0 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f75611fd640 10 monclient(hunting): authenticate will time out at 669.195830s 2024-08-19T09:50:13.135 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f755ad9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f755c0a6720 0x7f755c13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:56204/0 (socket says 172.21.15.1:56204) 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.119+0000 7f755ad9d640 1 -- 172.21.15.1:0/3411056423 learned_addr learned my addr 172.21.15.1:0/3411056423 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f755ad9d640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.1:6792/0 conn(0x7f755c0a6720 legacy=0x7f755c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 client.?.objecter ms_handle_connect 0x7f755c0a6720 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 client.?.objecter ms_handle_connect 0x7f755c139bf0 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f755a59c640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.1:6793/0 conn(0x7f755c139bf0 legacy=0x7f755c13db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004738259 0 0) 0x7f755c13f9e0 con 0x7f755c0a6720 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): my global_id is 4218 2024-08-19T09:50:13.136 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7548004ee0 con 0x7f755c0a6720 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 client.?.objecter ms_handle_connect 0x7f755c139fd0 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f755b59e640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.82:6792/0 conn(0x7f755c139fd0 legacy=0x7f755c1d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493541702 0 0) 0x7f755c1408b0 con 0x7f755c139bf0 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): my global_id is 4202 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f755c13f9e0 con 0x7f755c139bf0 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1011247720 0 0) 0x7f7548004ee0 con 0x7f755c0a6720 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f755c1408b0 con 0x7f755c0a6720 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f754c003610 con 0x7f755c0a6720 2024-08-19T09:50:13.137 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): dump: 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.138 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3088180098 0 0) 0x7f755c13edf0 con 0x7f755c139fd0 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): my global_id is 4237 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7548004ee0 con 0x7f755c139fd0 2024-08-19T09:50:13.139 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2545250816 0 0) 0x7f755c13f9e0 con 0x7f755c139bf0 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f755c13edf0 con 0x7f755c139bf0 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7550004b20 con 0x7f755c139bf0 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient(hunting): dump: 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.140 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4095515118 0 0) 0x7f755c1408b0 con 0x7f755c0a6720 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.1:6793/0 conn(0x7f755c139bf0 legacy=0x7f755c13db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.82:6792/0 conn(0x7f755c139fd0 legacy=0x7f755c1d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.141 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 monclient: found mon.d 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755c140b90 con 0x7f755c0a6720 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 5 monclient: authenticate success, global_id 4218 2024-08-19T09:50:13.142 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 10 monclient: _renew_subs 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f755c140e70 con 0x7f755c0a6720 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 10 monclient: _renew_subs 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f755c142630 con 0x7f755c0a6720 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75611fd640 1 librados: init done 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f754c003ba0 con 0x7f755c0a6720 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f754c004090 con 0x7f755c0a6720 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f755959a640 4 set_mon_vals no callback set 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:13.143 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.122+0000 7f75437fe640 10 monclient: dump: 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.144 INFO:teuthology.misc.health.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr: 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.123+0000 7f75611fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.123+0000 7f75611fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.123+0000 7f75611fd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.123+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f753c005080 con 0x7f755c0a6720 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.123+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f754c0042e0 con 0x7f755c0a6720 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.124+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 4148063755 0 0) 0x7f754c04d9c0 con 0x7f755c0a6720 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.124+0000 7f75437fe640 10 client.4218.objecter ms_dispatch 0x7f755c000b90 osd_map(15..15 src has 1..15) 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.124+0000 7f75437fe640 3 client.4218.objecter handle_osd_map got epochs [15,15] > 0 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.124+0000 7f75437fe640 3 client.4218.objecter handle_osd_map decoding full epoch 15 2024-08-19T09:50:13.145 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.124+0000 7f75437fe640 20 client.4218.objecter dump_active .. 0 homeless 2024-08-19T09:50:13.146 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.124+0000 7f755a59c640 10 monclient: get_auth_request con 0x7f7548041780 auth_method 0 2024-08-19T09:50:13.146 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.126+0000 7f75437fe640 10 client.4218.objecter ms_handle_connect 0x7f7548041780 2024-08-19T09:50:13.146 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.128+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f754c01b4b0 con 0x7f755c0a6720 2024-08-19T09:50:13.146 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.128+0000 7f75437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:13.146 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.128+0000 7f75437fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:13.315 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.313+0000 7f75611fd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:13.315 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.313+0000 7f75611fd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:13.315 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.313+0000 7f75611fd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:13.315 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.313+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f753c005370 con 0x7f755c0a6720 2024-08-19T09:50:13.316 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.314+0000 7f75437fe640 1 -- 172.21.15.1:0/3411056423 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8993 (unknown 4221056075 0 2089572091) 0x7f754c01ff60 con 0x7f755c0a6720 2024-08-19T09:50:13.316 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.314+0000 7f75437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:13.316 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.314+0000 7f75437fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:13.316 INFO:teuthology.misc.health.smithi001.stdout: 2024-08-19T09:50:13.316 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":15,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:50:09.948445+0000","last_up_change":"2024-08-19T09:50:07.419178+0000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T09:50:07.171793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":3862255507}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":3862255507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":3862255507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":3862255507}]},"public_addr":"172.21.15.1:6804/3862255507","cluster_addr":"172.21.15.1:6805/3862255507","heartbeat_back_addr":"172.21.15.1:6807/3862255507","heartbeat_front_addr":"172.21.15.1:6806/3862255507","state":["exists","up"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1668590231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1668590231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1668590231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1668590231}]},"public_addr":"172.21.15.1:6800/1668590231","cluster_addr":"172.21.15.1:6801/1668590231","heartbeat_back_addr":"172.21.15.1:6803/1668590231","heartbeat_front_addr":"172.21.15.1:6802/1668590231","state":["exists","up"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":207870648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":207870648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":207870648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":207870648}]},"public_addr":"172.21.15.1:6808/207870648","cluster_addr":"172.21.15.1:6809/207870648","heartbeat_back_addr":"172.21.15.1:6811/207870648","heartbeat_front_addr":"172.21.15.1:6810/207870648","state":["exists","up"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6804","nonce":394826682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6805","nonce":394826682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6807","nonce":394826682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6806","nonce":394826682}]},"public_addr":"172.21.15.82:6804/394826682","cluster_addr":"172.21.15.82:6805/394826682","heartbeat_back_addr":"172.21.15.82:6807/394826682","heartbeat_front_addr":"172.21.15.82:6806/394826682","state":["exists","up"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6800","nonce":2865329823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6801","nonce":2865329823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6803","nonce":2865329823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6802","nonce":2865329823}]},"public_addr":"172.21.15.82:6800/2865329823","cluster_addr":"172.21.15.82:6801/2865329823","heartbeat_back_addr":"172.21.15.82:6803/2865329823","heartbeat_front_addr":"172.21.15.82:6802/2865329823","state":["exists","up"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6808","nonce":1813220032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6809","nonce":1813220032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6811","nonce":1813220032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6810","nonce":1813220032}]},"public_addr":"172.21.15.82:6808/1813220032","cluster_addr":"172.21.15.82:6809/1813220032","heartbeat_back_addr":"172.21.15.82:6811/1813220032","heartbeat_front_addr":"172.21.15.82:6810/1813220032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.564842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.315230+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.809229+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:05.004322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.803207+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.616501+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:50:13.317 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 10 librados: watch_flush enter 2024-08-19T09:50:13.317 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 10 librados: watch_flush exit 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 20 client.4218.objecter shutdown clearing up homeless session... 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 10 client.4218.objecter successfully canceled tick 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7548041780 legacy=0x7f7548043c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 10 monclient: shutdown 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 >> v1:172.21.15.1:6792/0 conn(0x7f755c0a6720 legacy=0x7f755c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 shutdown_connections 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 >> 172.21.15.1:0/3411056423 conn(0x7f755c0ac140 msgr2=0x7f755c0aa130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 shutdown_connections 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 -- 172.21.15.1:0/3411056423 wait complete. 2024-08-19T09:50:13.318 INFO:teuthology.misc.health.smithi001.stderr:2024-08-19T09:50:13.316+0000 7f75611fd640 1 librados: shutdown 2024-08-19T09:50:13.326 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-08-19T09:50:13.327 INFO:tasks.ceph:Creating RBD pool 2024-08-19T09:50:13.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.426+0000 7fcc3635a640 10 monclient: get_monmap_and_config 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.426+0000 7fcc3635a640 10 monclient: build_initial_monmap 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.426+0000 7fcc3635a640 10 monclient: monmap: 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:13.427030+0000 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:13.427030+0000 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:13.427 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:13.428 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:13.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.426+0000 7fcc3635a640 1 Processor -- start 2024-08-19T09:50:13.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 1 -- start start 2024-08-19T09:50:13.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient: init 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient: _add_conns ranks=[6,2,3,5,0,8,4,1,7] 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient(hunting): picked mon.noname-d con 0x7fcc300a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient(hunting): picked mon.noname-b con 0x7fcc300694b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient(hunting): picked mon.noname-g con 0x7fcc300a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc300a3c70 con 0x7fcc300694b0 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc300a33d0 con 0x7fcc300a46a0 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc300a36b0 con 0x7fcc300a6830 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc3635a640 10 monclient(hunting): authenticate will time out at 669.503562s 2024-08-19T09:50:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc2ffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fcc300a6830 0x7fcc300a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:56214/0 (socket says 172.21.15.1:56214) 2024-08-19T09:50:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.427+0000 7fcc2ffff640 1 -- 172.21.15.1:0/290218931 learned_addr learned my addr 172.21.15.1:0/290218931 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.429+0000 7fcc2ffff640 1 -- 172.21.15.1:0/290218931 >> v1:172.21.15.1:6792/0 conn(0x7fcc300a6830 legacy=0x7fcc300a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.429+0000 7fcc27fff640 1 -- 172.21.15.1:0/290218931 >> v1:172.21.15.1:6790/0 conn(0x7fcc300694b0 legacy=0x7fcc300a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185766498 0 0) 0x7fcc300a36b0 con 0x7fcc300a6830 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): my global_id is 4227 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc1c003520 con 0x7fcc300a6830 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831263537 0 0) 0x7fcc300a3c70 con 0x7fcc300694b0 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient(hunting): my global_id is 4232 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc300a36b0 con 0x7fcc300694b0 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc348f7640 1 -- 172.21.15.1:0/290218931 >> v1:172.21.15.82:6790/0 conn(0x7fcc300a46a0 legacy=0x7fcc30141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 379610583 0 0) 0x7fcc1c003520 con 0x7fcc300a6830 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 >> v1:172.21.15.82:6790/0 conn(0x7fcc300a46a0 legacy=0x7fcc30141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 >> v1:172.21.15.1:6790/0 conn(0x7fcc300694b0 legacy=0x7fcc300a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 monclient: found mon.noname-d 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3006dbb0 con 0x7fcc300a6830 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc20001ae0 con 0x7fcc300a6830 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: dump: 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.443 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 5 monclient: authenticate success, global_id 4227 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:13.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc200048a0 con 0x7fcc300a6830 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 1 -- 172.21.15.1:0/290218931 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc20004e10 con 0x7fcc300a6830 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc2f7fe640 10 monclient: dump: 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.445 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 4 set_mon_vals no callback set 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 10 monclient: shutdown 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.430+0000 7fcc3635a640 1 -- 172.21.15.1:0/290218931 >> v1:172.21.15.1:6792/0 conn(0x7fcc300a6830 legacy=0x7fcc300a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 -- 172.21.15.1:0/290218931 shutdown_connections 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 -- 172.21.15.1:0/290218931 >> 172.21.15.1:0/290218931 conn(0x7fcc300a9020 msgr2=0x7fcc30068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 -- 172.21.15.1:0/290218931 shutdown_connections 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 -- 172.21.15.1:0/290218931 wait complete. 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 10 monclient: build_initial_monmap 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 10 monclient: monmap: 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:13.447 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 librados: starting msgr at 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 librados: starting objecter 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 Processor -- start 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 -- start start 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 librados: setting wanted keys 2024-08-19T09:50:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 1 librados: calling monclient init 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.431+0000 7fcc3635a640 10 monclient: init 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient: _add_conns ranks=[4,3,8,2,5,7,6,0,1] 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient(hunting): picked mon.noname-c con 0x7fcc300694b0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient(hunting): picked mon.noname-g con 0x7fcc300a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient(hunting): picked mon.noname-e con 0x7fcc300a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc301388b0 con 0x7fcc300a46a0 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc30138b90 con 0x7fcc300694b0 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc300a9d30 con 0x7fcc300a6830 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc3635a640 10 monclient(hunting): authenticate will time out at 669.508403s 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc2ffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fcc300694b0 0x7fcc30134d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59816/0 (socket says 172.21.15.1:59816) 2024-08-19T09:50:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.432+0000 7fcc2ffff640 1 -- 172.21.15.1:0/187532257 learned_addr learned my addr 172.21.15.1:0/187532257 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc348f7640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.1:6793/0 conn(0x7fcc300a6830 legacy=0x7fcc301d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 client.?.objecter ms_handle_connect 0x7fcc300a6830 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2ffff640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.1:6791/0 conn(0x7fcc300694b0 legacy=0x7fcc30134d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 client.?.objecter ms_handle_connect 0x7fcc300694b0 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2ffff640 1 --1- 172.21.15.1:0/187532257 >> v1:172.21.15.1:6791/0 conn(0x7fcc300694b0 0x7fcc30134d90 :-1 s=OPENED pgs=51 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fcc30138b90 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761103563 0 0) 0x7fcc300a9d30 con 0x7fcc300a6830 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 monclient(hunting): my global_id is 4211 2024-08-19T09:50:13.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc10004eb0 con 0x7fcc300a6830 2024-08-19T09:50:13.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 client.?.objecter ms_handle_connect 0x7fcc300a46a0 2024-08-19T09:50:13.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc2d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:13.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.434+0000 7fcc27fff640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.82:6790/0 conn(0x7fcc300a46a0 legacy=0x7fcc301354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:13.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 314151504 0 0) 0x7fcc10004eb0 con 0x7fcc300a6830 2024-08-19T09:50:13.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc300a9d30 con 0x7fcc300a6830 2024-08-19T09:50:13.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc28003990 con 0x7fcc300a6830 2024-08-19T09:50:13.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:13.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): dump: 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.453 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.454 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2838291121 0 0) 0x7fcc301388b0 con 0x7fcc300a46a0 2024-08-19T09:50:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient(hunting): my global_id is 4251 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc10004eb0 con 0x7fcc300a46a0 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2061632748 0 0) 0x7fcc300a9d30 con 0x7fcc300a6830 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.1:6791/0 conn(0x7fcc300694b0 legacy=0x7fcc30134d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.82:6790/0 conn(0x7fcc300a46a0 legacy=0x7fcc301354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 monclient: found mon.e 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc30136040 con 0x7fcc300a6830 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762782411 0 0) 0x7fcc30138b90 con 0x7fcc300694b0 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 5 monclient: authenticate success, global_id 4211 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 10 monclient: _renew_subs 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:13.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc30136830 con 0x7fcc300a6830 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 10 monclient: _renew_subs 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc30135bb0 con 0x7fcc300a6830 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc3635a640 1 librados: init done 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc28003ed0 con 0x7fcc300a6830 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc28004400 con 0x7fcc300a6830 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2effd640 4 set_mon_vals no callback set 2024-08-19T09:50:13.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.435+0000 7fcc2d7fa640 10 monclient: dump: 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc3635a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc3635a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc3635a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc14005080 con 0x7fcc300a6830 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcc280045a0 con 0x7fcc300a6830 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 4148063755 0 0) 0x7fcc2804dc30 con 0x7fcc300a6830 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2d7fa640 10 client.4211.objecter ms_dispatch 0x7fcc30000b90 osd_map(15..15 src has 1..15) 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2d7fa640 3 client.4211.objecter handle_osd_map got epochs [15,15] > 0 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2d7fa640 3 client.4211.objecter handle_osd_map decoding full epoch 15 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2d7fa640 20 client.4211.objecter dump_active .. 0 homeless 2024-08-19T09:50:13.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.436+0000 7fcc2ffff640 10 monclient: get_auth_request con 0x7fcc10040ff0 auth_method 0 2024-08-19T09:50:13.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.439+0000 7fcc2d7fa640 10 client.4211.objecter ms_handle_connect 0x7fcc10040ff0 2024-08-19T09:50:13.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.441+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fcc28017940 con 0x7fcc300a6830 2024-08-19T09:50:13.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.441+0000 7fcc2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:13.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.441+0000 7fcc2d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.625+0000 7fcc3635a640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-19T09:50:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.625+0000 7fcc3635a640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-19T09:50:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.625+0000 7fcc3635a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:13.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:13.625+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fcc14005c40 con 0x7fcc300a6830 2024-08-19T09:50:14.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.758+0000 7fcc2d7fa640 1 -- 172.21.15.1:0/187532257 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7fcc28017240 con 0x7fcc300a6830 2024-08-19T09:50:14.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.758+0000 7fcc2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-19T09:50:14.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.758+0000 7fcc2d7fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-08-19T09:50:14.760 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'rbd' created 2024-08-19T09:50:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 10 librados: watch_flush enter 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 10 librados: watch_flush exit 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 20 client.4211.objecter shutdown clearing up homeless session... 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 10 client.4211.objecter successfully canceled tick 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcc10040ff0 legacy=0x7fcc100434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 10 monclient: shutdown 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 >> v1:172.21.15.1:6793/0 conn(0x7fcc300a6830 legacy=0x7fcc301d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 shutdown_connections 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 >> 172.21.15.1:0/187532257 conn(0x7fcc300a9020 msgr2=0x7fcc300a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 shutdown_connections 2024-08-19T09:50:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 -- 172.21.15.1:0/187532257 wait complete. 2024-08-19T09:50:14.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.762+0000 7fcc3635a640 1 librados: shutdown 2024-08-19T09:50:14.776 DEBUG:teuthology.orchestra.run.smithi001:> rbd --cluster ceph pool init rbd 2024-08-19T09:50:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.813+0000 7f26b191bf00 10 monclient: get_monmap_and_config 2024-08-19T09:50:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.813+0000 7f26b191bf00 10 monclient: build_initial_monmap 2024-08-19T09:50:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.813+0000 7f26b191bf00 10 monclient: monmap: 2024-08-19T09:50:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:14.814247+0000 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:14.814247+0000 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:14.815 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:14.816 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 1 Processor -- start 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 1 -- start start 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient: init 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient: _reopen_session rank -1 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient: _add_conns ranks=[2,7,8,6,4,3,0,1,5] 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient(hunting): picked mon.noname-b con 0x55714435e520 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient(hunting): picked mon.noname-i con 0x5571442c94a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient(hunting): picked mon.noname-e con 0x55714435f010 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5571441599a0 con 0x55714435e520 2024-08-19T09:50:14.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x557144132010 con 0x5571442c94a0 2024-08-19T09:50:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5571441170c0 con 0x55714435f010 2024-08-19T09:50:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient(hunting): _renew_subs 2024-08-19T09:50:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b191bf00 10 monclient(hunting): authenticate will time out at 670.890590s 2024-08-19T09:50:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26abfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x55714435e520 0x55714435e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51582/0 (socket says 172.21.15.1:51582) 2024-08-19T09:50:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26b166a640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x55714435f010 0x557144367110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:33512/0 (socket says 172.21.15.1:33512) 2024-08-19T09:50:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.814+0000 7f26abfff640 1 -- 172.21.15.1:0/1567227890 learned_addr learned my addr 172.21.15.1:0/1567227890 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.816+0000 7f26abfff640 1 -- 172.21.15.1:0/1567227890 >> v1:172.21.15.1:6790/0 conn(0x55714435e520 legacy=0x55714435e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26b166a640 1 -- 172.21.15.1:0/1567227890 >> v1:172.21.15.1:6793/0 conn(0x55714435f010 legacy=0x557144367110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26ab7fe640 1 -- 172.21.15.1:0/1567227890 >> v1:172.21.15.82:6792/0 conn(0x5571442c94a0 legacy=0x5571443639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970196396 0 0) 0x5571441599a0 con 0x55714435e520 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:14.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): my global_id is 4241 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2698001e10 con 0x55714435e520 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267927145 0 0) 0x5571441170c0 con 0x55714435f010 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient(hunting): my global_id is 4220 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5571441599a0 con 0x55714435f010 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2133831611 0 0) 0x7f2698001e10 con 0x55714435e520 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 >> v1:172.21.15.1:6793/0 conn(0x55714435f010 legacy=0x557144367110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 >> v1:172.21.15.82:6792/0 conn(0x5571442c94a0 legacy=0x5571443639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 monclient: found mon.noname-b 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:14.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571442c9230 con 0x55714435e520 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f269c001ae0 con 0x55714435e520 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26b191bf00 5 monclient: authenticate success, global_id 4241 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26b191bf00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: dump: 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:14.825 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26b191bf00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f269c003190 con 0x55714435e520 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 1 -- 172.21.15.1:0/1567227890 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f269c003700 con 0x55714435e520 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26aaffd640 10 monclient: dump: 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.826 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26b191bf00 10 monclient: get_monmap_and_config success 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.817+0000 7f26b191bf00 4 set_mon_vals no callback set 2024-08-19T09:50:14.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 10 monclient: shutdown 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 1 -- 172.21.15.1:0/1567227890 >> v1:172.21.15.1:6790/0 conn(0x55714435e520 legacy=0x55714435e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 1 -- 172.21.15.1:0/1567227890 shutdown_connections 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 1 -- 172.21.15.1:0/1567227890 >> 172.21.15.1:0/1567227890 conn(0x5571442ae8b0 msgr2=0x5571442aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 1 -- 172.21.15.1:0/1567227890 shutdown_connections 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.818+0000 7f26b191bf00 1 -- 172.21.15.1:0/1567227890 wait complete. 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient: build_initial_monmap 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient: monmap: 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:14.828 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 librados: starting msgr at 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 librados: starting objecter 2024-08-19T09:50:14.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 Processor -- start 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 -- start start 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 librados: setting wanted keys 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 librados: calling monclient init 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient: init 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient: _reopen_session rank -1 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient: _add_conns ranks=[5,7,0,8,6,1,2,3,4] 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient(hunting): picked mon.noname-h con 0x5571442c94a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient(hunting): picked mon.noname-i con 0x55714435e520 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient(hunting): picked mon.noname-a con 0x55714435f010 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55714434a460 con 0x55714435f010 2024-08-19T09:50:14.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5571442d0380 con 0x5571442c94a0 2024-08-19T09:50:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x557144348bd0 con 0x55714435e520 2024-08-19T09:50:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient(hunting): _renew_subs 2024-08-19T09:50:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.819+0000 7f26b191bf00 10 monclient(hunting): authenticate will time out at 670.895952s 2024-08-19T09:50:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.820+0000 7f26b166a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x55714435f010 0x557144374530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42720/0 (socket says 172.21.15.1:42720) 2024-08-19T09:50:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.820+0000 7f26b166a640 1 -- 172.21.15.1:0/950073101 learned_addr learned my addr 172.21.15.1:0/950073101 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:14.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26b166a640 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.1:6789/0 conn(0x55714435f010 legacy=0x557144374530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:14.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 client.?.objecter ms_handle_connect 0x55714435f010 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26ab7fe640 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.82:6792/0 conn(0x55714435e520 legacy=0x557144370e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 client.?.objecter ms_handle_connect 0x55714435e520 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26abfff640 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.82:6791/0 conn(0x5571442c94a0 legacy=0x55714435cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 client.?.objecter ms_handle_connect 0x5571442c94a0 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3337763975 0 0) 0x55714434a460 con 0x55714435f010 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): my global_id is 4257 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f268c002fe0 con 0x55714435f010 2024-08-19T09:50:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 127073606 0 0) 0x5571442d0380 con 0x5571442c94a0 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): my global_id is 4253 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55714434a460 con 0x5571442c94a0 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 418464111 0 0) 0x7f268c002fe0 con 0x55714435f010 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x5571442d0380 con 0x55714435f010 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26a4003450 con 0x55714435f010 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): dump: 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:14.836 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:14.837 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 218105582 0 0) 0x55714434a460 con 0x5571442c94a0 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f268c002fe0 con 0x5571442c94a0 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f269c003680 con 0x5571442c94a0 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient(hunting): dump: 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.838 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2880720178 0 0) 0x5571442d0380 con 0x55714435f010 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.82:6792/0 conn(0x55714435e520 legacy=0x557144370e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.82:6791/0 conn(0x5571442c94a0 legacy=0x55714435cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 monclient: found mon.a 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x557144349280 con 0x55714435f010 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:14.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26b191bf00 5 monclient: authenticate success, global_id 4257 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26b191bf00 10 monclient: _renew_subs 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26b191bf00 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.822+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x55714434ad40 con 0x55714435f010 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26b191bf00 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26b191bf00 10 monclient: _renew_subs 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26b191bf00 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x557144348830 con 0x55714435f010 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26b191bf00 1 librados: init done 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26b191bf00 10 librados: wait_for_osdmap waiting 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26a40038a0 con 0x55714435f010 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26a4003dd0 con 0x55714435f010 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 10 monclient: dump: 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:14.842 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26aa7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.823+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f26a4004070 con 0x55714435f010 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (unknown 917980866 0 0) 0x7f26a404c660 con 0x55714435f010 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26a3fff640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_map(16..16 src has 1..16) 2024-08-19T09:50:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map got epochs [16,16] > 0 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map decoding full epoch 16 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26a3fff640 20 client.4257.objecter dump_active .. 0 homeless 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26b191bf00 10 librados: wait_for_osdmap done waiting 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26b191bf00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26b191bf00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26b191bf00 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55714436bdb0 con 0x55714435f010 2024-08-19T09:50:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.824+0000 7f26abfff640 10 monclient: get_auth_request con 0x7f268c03f520 auth_method 0 2024-08-19T09:50:14.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:14.826+0000 7f26a3fff640 10 client.4257.objecter ms_handle_connect 0x7f268c03f520 2024-08-19T09:50:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.759+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (unknown 1851909647 0 0) 0x7f26a4015b50 con 0x55714435f010 2024-08-19T09:50:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.759+0000 7f26a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-19T09:50:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.759+0000 7f26a3fff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-08-19T09:50:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.760+0000 7f26b191bf00 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.760+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f269c003190 con 0x55714435f010 2024-08-19T09:50:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.766+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f26a4015500 con 0x55714435f010 2024-08-19T09:50:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.766+0000 7f26a3fff640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-08-19T09:50:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26aa7fc640 10 client.4257.objecter _get_latest_version latest 17, waiting 2024-08-19T09:50:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26aa7fc640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26aa7fc640 10 monclient: _renew_subs 2024-08-19T09:50:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26aa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:15.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26aa7fc640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=17}) -- 0x7f2690003d70 con 0x55714435f010 2024-08-19T09:50:15.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 11 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (unknown 194358457 0 0) 0x7f26a401e600 con 0x55714435f010 2024-08-19T09:50:15.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26a3fff640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_map(17..17 src has 1..17) 2024-08-19T09:50:15.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map got epochs [17,17] > 16 2024-08-19T09:50:15.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map decoding incremental epoch 17 2024-08-19T09:50:15.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.767+0000 7f26a3fff640 20 client.4257.objecter dump_active .. 0 homeless 2024-08-19T09:50:15.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.769+0000 7f26b191bf00 10 librados: create oid=rbd_trash nspace= 2024-08-19T09:50:15.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.770+0000 7f26b191bf00 10 client.4257.objecter _op_submit op 0x557144376070 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.770+0000 7f26b191bf00 20 client.4257.objecter _calc_target epoch 17 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 20 client.4257.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 10 client.4257.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 20 client.4257.objecter _get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 10 client.4257.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 20 client.4257.objecter get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 15 client.4257.objecter _session_op_assign 5 1 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 15 client.4257.objecter _send_op 1 to 2.3 on osd.5 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6808/1813220032 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55714437a540 con 0x557144376a30 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26b191bf00 5 client.4257.objecter 1 in flight 2024-08-19T09:50:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.771+0000 7f26ab7fe640 10 monclient: get_auth_request con 0x557144376a30 auth_method 0 2024-08-19T09:50:15.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.776+0000 7f26a3fff640 10 client.4257.objecter ms_handle_connect 0x557144376a30 2024-08-19T09:50:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 1 -- 172.21.15.1:0/950073101 <== osd.5 v1:172.21.15.82:6808/1813220032 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 1703302865 0 0) 0x7f26940018f0 con 0x557144376a30 2024-08-19T09:50:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) 2024-08-19T09:50:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 10 client.4257.objecter in handle_osd_op_reply 2024-08-19T09:50:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 7 client.4257.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-08-19T09:50:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 10 client.4257.objecter op 0 rval 0 len 0 2024-08-19T09:50:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 15 client.4257.objecter handle_osd_op_reply completed tid 1 2024-08-19T09:50:15.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 15 client.4257.objecter _finish_op 1 2024-08-19T09:50:15.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:15.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 15 client.4257.objecter _session_op_remove 5 1 2024-08-19T09:50:15.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26ab7fe640 5 client.4257.objecter 0 in flight 2024-08-19T09:50:15.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.778+0000 7f26b191bf00 10 librados: Objecter returned from create r=0 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 10 client.4257.objecter _op_submit op 0x5571443cc6f0 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 20 client.4257.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 20 client.4257.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 10 client.4257.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 20 client.4257.objecter _get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 10 client.4257.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 20 client.4257.objecter get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 15 client.4257.objecter _session_op_assign 5 2 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 15 client.4257.objecter _send_op 2 to 2.2 on osd.5 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6808/1813220032 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x5571443ccc40 con 0x557144376a30 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:15.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.780+0000 7f26b191bf00 5 client.4257.objecter 1 in flight 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 1 -- 172.21.15.1:0/950073101 <== osd.5 v1:172.21.15.82:6808/1813220032 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4274215525 0 0) 0x7f26940018f0 con 0x557144376a30 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter in handle_osd_op_reply 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 7 client.4257.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter op 0 rval 0 len 0 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 15 client.4257.objecter handle_osd_op_reply completed tid 2 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 15 client.4257.objecter _finish_op 2 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:15.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 15 client.4257.objecter _session_op_remove 5 2 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 5 client.4257.objecter 0 in flight 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 10 client.4257.objecter _op_submit op 0x5571443cb880 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 20 client.4257.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 20 client.4257.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 10 client.4257.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 20 client.4257.objecter _get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 10 client.4257.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 20 client.4257.objecter get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 15 client.4257.objecter _session_op_assign 5 3 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 15 client.4257.objecter _send_op 3 to 2.2 on osd.5 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6808/1813220032 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x5571443cbd30 con 0x557144376a30 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:15.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26b191bf00 5 client.4257.objecter 1 in flight 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 1 -- 172.21.15.1:0/950073101 <== osd.5 v1:172.21.15.82:6808/1813220032 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4287864937 0 0) 0x7f26940018f0 con 0x557144376a30 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter in handle_osd_op_reply 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 7 client.4257.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 10 client.4257.objecter op 0 rval 0 len 0 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 15 client.4257.objecter handle_osd_op_reply completed tid 3 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 15 client.4257.objecter _finish_op 3 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 15 client.4257.objecter _session_op_remove 5 3 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26ab7fe640 5 client.4257.objecter 0 in flight 2024-08-19T09:50:15.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26a9ffb640 10 client.4257.objecter allocate_selfmanaged_snap; pool: 2 2024-08-19T09:50:15.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26a9ffb640 10 client.4257.objecter pool_op_submit 4 2024-08-19T09:50:15.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26a9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:15.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:15.781+0000 7f26a9ffb640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f26840030d0 con 0x55714435f010 2024-08-19T09:50:16.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 12 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (unknown 1358618226 0 0) 0x7f26a401ea90 con 0x55714435f010 2024-08-19T09:50:16.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 pool_op_reply(tid 4 (0) Success v18) 2024-08-19T09:50:16.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 client.4257.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v18) 2024-08-19T09:50:16.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 client.4257.objecter have request 4 at 0x7f2684003060 Op: create unmanaged snap 2024-08-19T09:50:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 20 client.4257.objecter waiting for client to reach epoch 18 before calling back 2024-08-19T09:50:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 monclient: _renew_subs 2024-08-19T09:50:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=18}) -- 0x5571442d0380 con 0x55714435f010 2024-08-19T09:50:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.766+0000 7f26a3fff640 10 client.4257.objecter done 2024-08-19T09:50:16.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.767+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 13 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (unknown 3192517823 0 0) 0x7f26a4011e40 con 0x55714435f010 2024-08-19T09:50:16.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.767+0000 7f26a3fff640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_map(18..18 src has 1..18) 2024-08-19T09:50:16.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.767+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map got epochs [18,18] > 17 2024-08-19T09:50:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.767+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map decoding incremental epoch 18 2024-08-19T09:50:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.767+0000 7f26a3fff640 20 client.4257.objecter dump_active .. 0 homeless 2024-08-19T09:50:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 20 librados: queue_aio_write 0x5571443c7c80 completion 0x5571442d0b50 write_seq 1 2024-08-19T09:50:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 10 client.4257.objecter _op_submit op 0x7f2690006460 2024-08-19T09:50:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 20 client.4257.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 20 client.4257.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 10 client.4257.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 20 client.4257.objecter _get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 10 client.4257.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 20 client.4257.objecter get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 15 client.4257.objecter _session_op_assign 5 5 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 15 client.4257.objecter _send_op 5 to 2.2 on osd.5 2024-08-19T09:50:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6808/1813220032 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f2690006aa0 con 0x557144376a30 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.768+0000 7f26aa7fc640 5 client.4257.objecter 1 in flight 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 1 -- 172.21.15.1:0/950073101 <== osd.5 v1:172.21.15.82:6808/1813220032 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 2910594948 0 0) 0x7f26940018f0 con 0x557144376a30 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 10 client.4257.objecter in handle_osd_op_reply 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 7 client.4257.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 10 client.4257.objecter op 0 rval 0 len 0 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 10 client.4257.objecter op 1 rval 0 len 0 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 15 client.4257.objecter handle_osd_op_reply completed tid 5 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 15 client.4257.objecter _finish_op 5 2024-08-19T09:50:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 15 client.4257.objecter _session_op_remove 5 5 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 5 client.4257.objecter 0 in flight 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26ab7fe640 20 librados: complete_aio_write 0x5571442d0b50 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26a9ffb640 10 client.4257.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26a9ffb640 10 client.4257.objecter pool_op_submit 6 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26a9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:16.769+0000 7f26a9ffb640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f2684004540 con 0x55714435f010 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 14 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (unknown 2802565529 0 0) 0x7f26a40150f0 con 0x55714435f010 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 pool_op_reply(tid 6 (0) Success v19) 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 client.4257.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v19) 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 client.4257.objecter have request 6 at 0x7f2684003060 Op: delete unmanaged snap 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 20 client.4257.objecter waiting for client to reach epoch 19 before calling back 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 monclient: _renew_subs 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=19}) -- 0x55714434a460 con 0x55714435f010 2024-08-19T09:50:17.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 client.4257.objecter done 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 1 -- 172.21.15.1:0/950073101 <== mon.0 v1:172.21.15.1:6789/0 15 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (unknown 1096579771 0 0) 0x7f26a4014880 con 0x55714435f010 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_map(19..19 src has 1..19) 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map got epochs [19,19] > 18 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 3 client.4257.objecter handle_osd_map decoding incremental epoch 19 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26a3fff640 20 client.4257.objecter dump_active .. 0 homeless 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 20 librados: queue_aio_write 0x5571443c7c80 completion 0x7f2684001d60 write_seq 2 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 10 client.4257.objecter _op_submit op 0x7f2690006460 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 20 client.4257.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 20 client.4257.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 10 client.4257.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 20 client.4257.objecter _get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 10 client.4257.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 20 client.4257.objecter get_session s=0x55714434b6a0 osd=5 3 2024-08-19T09:50:17.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 15 client.4257.objecter _session_op_assign 5 7 2024-08-19T09:50:17.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 15 client.4257.objecter _send_op 7 to 2.2 on osd.5 2024-08-19T09:50:17.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.768+0000 7f26aa7fc640 1 -- 172.21.15.1:0/950073101 --> v1:172.21.15.82:6808/1813220032 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f2690006e80 con 0x557144376a30 2024-08-19T09:50:17.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.769+0000 7f26aa7fc640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:17.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.769+0000 7f26aa7fc640 5 client.4257.objecter 1 in flight 2024-08-19T09:50:17.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 1 -- 172.21.15.1:0/950073101 <== osd.5 v1:172.21.15.82:6808/1813220032 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 900577457 0 0) 0x7f26940018f0 con 0x557144376a30 2024-08-19T09:50:17.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 10 client.4257.objecter ms_dispatch 0x5571441b0a00 osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) 2024-08-19T09:50:17.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 10 client.4257.objecter in handle_osd_op_reply 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 7 client.4257.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 10 client.4257.objecter op 0 rval 0 len 0 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 15 client.4257.objecter handle_osd_op_reply completed tid 7 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 15 client.4257.objecter _finish_op 7 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 4 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 15 client.4257.objecter _session_op_remove 5 7 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 5 client.4257.objecter 0 in flight 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26ab7fe640 20 librados: complete_aio_write 0x7f2684001d60 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 10 librados: watch_flush enter 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 10 librados: watch_flush exit 2024-08-19T09:50:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 10 client.4257.objecter close_session for osd.5 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.82:6808/1813220032 conn(0x557144376a30 legacy=0x557144378e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 20 client.4257.objecter put_session s=0x55714434b6a0 osd=5 1 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 20 client.4257.objecter shutdown clearing up homeless session... 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 10 client.4257.objecter successfully canceled tick 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.82:6812/2549104415 conn(0x7f268c03f520 legacy=0x7f268c0419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 10 monclient: shutdown 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 >> v1:172.21.15.1:6789/0 conn(0x55714435f010 legacy=0x557144374530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b166a640 1 -- 172.21.15.1:0/950073101 reap_dead start 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 shutdown_connections 2024-08-19T09:50:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 >> 172.21.15.1:0/950073101 conn(0x5571442ae8b0 msgr2=0x5571442cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 shutdown_connections 2024-08-19T09:50:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 -- 172.21.15.1:0/950073101 wait complete. 2024-08-19T09:50:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.770+0000 7f26b191bf00 1 librados: shutdown 2024-08-19T09:50:17.784 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-19T09:50:17.785 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-19T09:50:17.785 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-19T09:50:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.931+0000 7fd96a83f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.932+0000 7fd96a83f640 10 monclient: build_initial_monmap 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.932+0000 7fd96a83f640 10 monclient: monmap: 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:17.932949+0000 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:17.932949+0000 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:17.934 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:17.934 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:17.934 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:17.934 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:17.934 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:17.934 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:17.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.932+0000 7fd96a83f640 1 Processor -- start 2024-08-19T09:50:17.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.932+0000 7fd96a83f640 1 -- start start 2024-08-19T09:50:17.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.932+0000 7fd96a83f640 10 monclient: init 2024-08-19T09:50:17.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:17.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient: _add_conns ranks=[0,2,5,7,1,8,6,3,4] 2024-08-19T09:50:17.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9640a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9640a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient(hunting): picked mon.noname-h con 0x7fd96413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9640a3b40 con 0x7fd9640a67c0 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9640a8130 con 0x7fd9640a6ba0 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96406ede0 con 0x7fd96413f5c0 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd96a83f640 10 monclient(hunting): authenticate will time out at 674.009498s 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd9637fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fd9640a6ba0 0x7fd9640a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52728/0 (socket says 172.21.15.1:52728) 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd9637fe640 1 -- 172.21.15.1:0/2724700752 learned_addr learned my addr 172.21.15.1:0/2724700752 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.933+0000 7fd963fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9640a67c0 0x7fd9640a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37082/0 (socket says 172.21.15.1:37082) 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd9637fe640 1 -- 172.21.15.1:0/2724700752 >> v1:172.21.15.1:6790/0 conn(0x7fd9640a6ba0 legacy=0x7fd9640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd963fff640 1 -- 172.21.15.1:0/2724700752 >> v1:172.21.15.1:6789/0 conn(0x7fd9640a67c0 legacy=0x7fd9640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3737300093 0 0) 0x7fd9640a8130 con 0x7fd9640a6ba0 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): my global_id is 4250 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd94c003520 con 0x7fd9640a6ba0 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566019812 0 0) 0x7fd9640a3b40 con 0x7fd9640a67c0 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:17.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 10 monclient(hunting): my global_id is 4266 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.935+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9640a8130 con 0x7fd9640a67c0 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd968ddc640 1 -- 172.21.15.1:0/2724700752 >> v1:172.21.15.82:6791/0 conn(0x7fd96413f5c0 legacy=0x7fd9641419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1327883554 0 0) 0x7fd9640a8130 con 0x7fd9640a67c0 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 >> v1:172.21.15.82:6791/0 conn(0x7fd96413f5c0 legacy=0x7fd9641419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 >> v1:172.21.15.1:6790/0 conn(0x7fd9640a6ba0 legacy=0x7fd9640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 monclient: found mon.noname-a 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9640a92a0 con 0x7fd9640a67c0 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:17.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd954001ac0 con 0x7fd9640a67c0 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd96a83f640 5 monclient: authenticate success, global_id 4266 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd96a83f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: dump: 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:17.942 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd96a83f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9540047e0 con 0x7fd9640a67c0 2024-08-19T09:50:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 1 -- 172.21.15.1:0/2724700752 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd954004d50 con 0x7fd9640a67c0 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd962ffd640 10 monclient: dump: 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:17.944 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd96a83f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.936+0000 7fd96a83f640 4 set_mon_vals no callback set 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 10 monclient: shutdown 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 -- 172.21.15.1:0/2724700752 >> v1:172.21.15.1:6789/0 conn(0x7fd9640a67c0 legacy=0x7fd9640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 -- 172.21.15.1:0/2724700752 shutdown_connections 2024-08-19T09:50:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 -- 172.21.15.1:0/2724700752 >> 172.21.15.1:0/2724700752 conn(0x7fd9640ac4b0 msgr2=0x7fd9640a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 -- 172.21.15.1:0/2724700752 shutdown_connections 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 -- 172.21.15.1:0/2724700752 wait complete. 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 10 monclient: build_initial_monmap 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 10 monclient: monmap: 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:17.946 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 librados: starting msgr at 2024-08-19T09:50:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.937+0000 7fd96a83f640 1 librados: starting objecter 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 Processor -- start 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 -- start start 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 librados: setting wanted keys 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 librados: calling monclient init 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient: init 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient: _add_conns ranks=[5,0,4,3,1,8,2,6,7] 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient(hunting): picked mon.noname-h con 0x7fd9640a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9640a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient(hunting): picked mon.noname-c con 0x7fd96413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9640aa1d0 con 0x7fd9640a6ba0 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9640a0390 con 0x7fd96413f5c0 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96409f850 con 0x7fd9640a67c0 2024-08-19T09:50:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:17.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd96a83f640 10 monclient(hunting): authenticate will time out at 674.014760s 2024-08-19T09:50:17.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd9637fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9640a6ba0 0x7fd96409f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37090/0 (socket says 172.21.15.1:37090) 2024-08-19T09:50:17.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.938+0000 7fd9637fe640 1 -- 172.21.15.1:0/3968226435 learned_addr learned my addr 172.21.15.1:0/3968226435 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:17.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.940+0000 7fd9637fe640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.1:6789/0 conn(0x7fd9640a6ba0 legacy=0x7fd96409f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:17.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.940+0000 7fd960ff9640 10 client.?.objecter ms_handle_connect 0x7fd9640a6ba0 2024-08-19T09:50:17.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.940+0000 7fd960ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:17.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 client.?.objecter ms_handle_connect 0x7fd96413f5c0 2024-08-19T09:50:17.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:17.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd968ddc640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.1:6791/0 conn(0x7fd96413f5c0 legacy=0x7fd9641d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:17.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd9637fe640 1 --1- 172.21.15.1:0/3968226435 >> v1:172.21.15.1:6789/0 conn(0x7fd9640a6ba0 0x7fd96409f140 :-1 s=OPENED pgs=73 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fd9640aa1d0 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:50:17.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 448719677 0 0) 0x7fd9640a0390 con 0x7fd96413f5c0 2024-08-19T09:50:17.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): my global_id is 4243 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd950004eb0 con 0x7fd96413f5c0 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 client.?.objecter ms_handle_connect 0x7fd9640a67c0 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd963fff640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.82:6791/0 conn(0x7fd9640a67c0 legacy=0x7fd96409ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:17.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 655699707 0 0) 0x7fd950004eb0 con 0x7fd96413f5c0 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9640a0390 con 0x7fd96413f5c0 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150187859 0 0) 0x7fd96409f850 con 0x7fd9640a67c0 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): my global_id is 4271 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd950004eb0 con 0x7fd9640a67c0 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd95c002040 con 0x7fd96413f5c0 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:17.958 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:17.959 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 193613001 0 0) 0x7fd9640a0390 con 0x7fd96413f5c0 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.82:6791/0 conn(0x7fd9640a67c0 legacy=0x7fd96409ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.1:6789/0 conn(0x7fd9640a6ba0 legacy=0x7fd96409f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 monclient: found mon.c 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9640a26f0 con 0x7fd96413f5c0 2024-08-19T09:50:17.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703032266 0 0) 0x7fd9640aa1d0 con 0x7fd9640a6ba0 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 5 monclient: authenticate success, global_id 4243 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 10 monclient: _renew_subs 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9640a1160 con 0x7fd96413f5c0 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 10 monclient: _renew_subs 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9641d7970 con 0x7fd96413f5c0 2024-08-19T09:50:17.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd96a83f640 1 librados: init done 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd95c003c40 con 0x7fd96413f5c0 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd95c004130 con 0x7fd96413f5c0 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd9627fc640 4 set_mon_vals no callback set 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.941+0000 7fd960ff9640 10 monclient: dump: 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd95c004400 con 0x7fd96413f5c0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd96a83f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd96a83f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd96a83f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd944005080 con 0x7fd96413f5c0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fd95c04dd50 con 0x7fd96413f5c0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd960ff9640 10 client.4243.objecter ms_dispatch 0x7fd964000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd960ff9640 3 client.4243.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.942+0000 7fd960ff9640 3 client.4243.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.943+0000 7fd960ff9640 20 client.4243.objecter dump_active .. 0 homeless 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.943+0000 7fd963fff640 10 monclient: get_auth_request con 0x7fd950040f50 auth_method 0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.945+0000 7fd960ff9640 10 client.4243.objecter ms_handle_connect 0x7fd950040f50 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.947+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fd95c0177f0 con 0x7fd96413f5c0 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.947+0000 7fd960ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:17.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:17.947+0000 7fd960ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.131+0000 7fd96a83f640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-08-19T09:50:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.131+0000 7fd96a83f640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-08-19T09:50:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.131+0000 7fd96a83f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.131+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fd944005370 con 0x7fd96413f5c0 2024-08-19T09:50:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.132+0000 7fd960ff9640 1 -- 172.21.15.1:0/3968226435 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v5) ==== 86+0+159 (unknown 882900978 0 3468804871) 0x7fd95c0170f0 con 0x7fd96413f5c0 2024-08-19T09:50:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.132+0000 7fd960ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-08-19T09:50:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.132+0000 7fd960ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:18.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.133+0000 7fd96a83f640 10 librados: watch_flush enter 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.133+0000 7fd96a83f640 10 librados: watch_flush exit 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.133+0000 7fd96a83f640 20 client.4243.objecter shutdown clearing up homeless session... 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.133+0000 7fd96a83f640 10 client.4243.objecter successfully canceled tick 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd950040f50 legacy=0x7fd950043410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 10 monclient: shutdown 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 >> v1:172.21.15.1:6791/0 conn(0x7fd96413f5c0 legacy=0x7fd9641d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 shutdown_connections 2024-08-19T09:50:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 >> 172.21.15.1:0/3968226435 conn(0x7fd9640ac4b0 msgr2=0x7fd9640a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 shutdown_connections 2024-08-19T09:50:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 -- 172.21.15.1:0/3968226435 wait complete. 2024-08-19T09:50:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.134+0000 7fd96a83f640 1 librados: shutdown 2024-08-19T09:50:18.145 INFO:teuthology.orchestra.run.smithi001.stdout:[{"version":5,"timestamp":"2024-08-19T09:50:07.176095+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"6651.181283"}]}] 2024-08-19T09:50:18.145 INFO:tasks.ceph_manager:config epoch is 5 2024-08-19T09:50:18.146 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-19T09:50:18.146 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-19T09:50:18.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.243+0000 7fcef317f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.243+0000 7fcef317f640 10 monclient: build_initial_monmap 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.243+0000 7fcef317f640 10 monclient: monmap: 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:18.244486+0000 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:18.244486+0000 2024-08-19T09:50:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:18.246 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 1 Processor -- start 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 1 -- start start 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient: init 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient: _add_conns ranks=[7,5,8,6,3,4,1,0,2] 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient(hunting): picked mon.noname-i con 0x7fceec0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient(hunting): picked mon.noname-h con 0x7fceec0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient(hunting): picked mon.noname-e con 0x7fceec13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:18.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec0a3b40 con 0x7fceec0a6ba0 2024-08-19T09:50:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec0a8130 con 0x7fceec0a67c0 2024-08-19T09:50:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec06ede0 con 0x7fceec13f5c0 2024-08-19T09:50:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef317f640 10 monclient(hunting): authenticate will time out at 674.320829s 2024-08-19T09:50:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef171c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 0x7fceec1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44590/0 (socket says 172.21.15.1:44590) 2024-08-19T09:50:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.244+0000 7fcef171c640 1 -- 172.21.15.1:0/968249854 learned_addr learned my addr 172.21.15.1:0/968249854 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef171c640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 legacy=0x7fceec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584141092 0 0) 0x7fceec06ede0 con 0x7fceec13f5c0 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): my global_id is 4229 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcedc003520 con 0x7fceec13f5c0 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef0f1b640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6792/0 conn(0x7fceec0a67c0 legacy=0x7fceec0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceebfff640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6791/0 conn(0x7fceec0a6ba0 legacy=0x7fceec0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1279147617 0 0) 0x7fcedc003520 con 0x7fceec13f5c0 2024-08-19T09:50:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6792/0 conn(0x7fceec0a67c0 legacy=0x7fceec0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6791/0 conn(0x7fceec0a6ba0 legacy=0x7fceec0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 monclient: found mon.noname-e 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceec0a92a0 con 0x7fceec13f5c0 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef317f640 5 monclient: authenticate success, global_id 4229 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcee4001e50 con 0x7fceec13f5c0 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef317f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: dump: 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.252 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef171c640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 legacy=0x7fceec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef171c640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 legacy=0x7fceec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef171c640 1 --1- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 0x7fceec1419b0 :-1 s=OPENED pgs=47 cs=1 l=1).handle_message read tag failed 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef171c640 1 --1- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 0x7fceec1419b0 :-1 s=OPENED pgs=47 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fcef317f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.1:6793/0 2024-08-19T09:50:18.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fceec13f5c0 legacy=0x7fceec1419b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient: _add_conns ranks=[3,8,5,2,0,4,7,1,6] 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): picked mon.g con 0x7fcedc0043a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): picked mon.e con 0x7fcedc005140 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): picked mon.h con 0x7fceec0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcedc003520 con 0x7fcedc0043a0 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fceec06ede0 con 0x7fceec0a6ba0 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcedc011b90 con 0x7fcedc005140 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.247+0000 7fceeb7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6793/0 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.249+0000 7fceebfff640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fcedc005140 legacy=0x7fcedc00a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.249+0000 7fcef171c640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6791/0 conn(0x7fceec0a6ba0 legacy=0x7fcedc00e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.249+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60530080 0 0) 0x7fcedc011b90 con 0x7fcedc005140 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.249+0000 7fceeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.249+0000 7fceeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.249+0000 7fceeb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.1:6793/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fcedc010b10 con 0x7fcedc005140 2024-08-19T09:50:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef0f1b640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6790/0 conn(0x7fcedc0043a0 legacy=0x7fcedc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1545552337 0 0) 0x7fceec06ede0 con 0x7fceec0a6ba0 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.82:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fcedc011b90 con 0x7fceec0a6ba0 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 4066908402 0 0) 0x7fcedc010b10 con 0x7fcedc005140 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6791/0 conn(0x7fceec0a6ba0 legacy=0x7fcedc00e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.82:6790/0 conn(0x7fcedc0043a0 legacy=0x7fcedc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 monclient: found mon.e 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=2+}) -- 0x7fcedc012ef0 con 0x7fcedc005140 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fced8001b00 con 0x7fcedc005140 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: dump: 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 1 -- 172.21.15.1:0/968249854 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fced8003610 con 0x7fcedc005140 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fceeb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 4 set_mon_vals no callback set 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 10 monclient: shutdown 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 1 -- 172.21.15.1:0/968249854 >> v1:172.21.15.1:6793/0 conn(0x7fcedc005140 legacy=0x7fcedc00a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 1 -- 172.21.15.1:0/968249854 shutdown_connections 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 1 -- 172.21.15.1:0/968249854 >> 172.21.15.1:0/968249854 conn(0x7fceec0ac4b0 msgr2=0x7fceec0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 1 -- 172.21.15.1:0/968249854 shutdown_connections 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.250+0000 7fcef317f640 1 -- 172.21.15.1:0/968249854 wait complete. 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 10 monclient: build_initial_monmap 2024-08-19T09:50:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 10 monclient: monmap: 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 1 librados: starting msgr at 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 1 librados: starting objecter 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 1 Processor -- start 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 1 -- start start 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 1 librados: setting wanted keys 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 1 librados: calling monclient init 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.251+0000 7fcef317f640 10 monclient: init 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient: _add_conns ranks=[0,3,6,2,1,7,8,4,5] 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient(hunting): picked mon.noname-a con 0x7fcedc0043a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient(hunting): picked mon.noname-g con 0x7fcedc005140 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient(hunting): picked mon.noname-d con 0x7fceec0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec1aedb0 con 0x7fcedc0043a0 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec0aa450 con 0x7fcedc005140 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec1aca50 con 0x7fceec0a67c0 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef317f640 10 monclient(hunting): authenticate will time out at 674.328445s 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef0f1b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcedc0043a0 0x7fceec1c13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37098/0 (socket says 172.21.15.1:37098) 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef0f1b640 1 -- 172.21.15.1:0/732100068 learned_addr learned my addr 172.21.15.1:0/732100068 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.252+0000 7fcef171c640 1 --1- 172.21.15.1:0/732100068 >> v1:172.21.15.1:6792/0 conn(0x7fceec0a67c0 0x7fceec1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52808/0 (socket says 172.21.15.1:52808) 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcef0f1b640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.1:6789/0 conn(0x7fcedc0043a0 legacy=0x7fceec1c13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 client.?.objecter ms_handle_connect 0x7fcedc0043a0 2024-08-19T09:50:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcef171c640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.1:6792/0 conn(0x7fceec0a67c0 legacy=0x7fceec1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 client.?.objecter ms_handle_connect 0x7fceec0a67c0 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 client.?.objecter ms_handle_connect 0x7fcedc005140 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fceebfff640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.82:6790/0 conn(0x7fcedc005140 legacy=0x7fceec1d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909339835 0 0) 0x7fceec1aedb0 con 0x7fcedc0043a0 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): my global_id is 4284 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fced0004eb0 con 0x7fcedc0043a0 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636012162 0 0) 0x7fceec1aca50 con 0x7fceec0a67c0 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 10 monclient(hunting): my global_id is 4236 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.254+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceec1aedb0 con 0x7fceec0a67c0 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966401914 0 0) 0x7fceec0aa450 con 0x7fcedc005140 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): my global_id is 4260 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceec1aca50 con 0x7fcedc005140 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3774212050 0 0) 0x7fced0004eb0 con 0x7fcedc0043a0 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fceec0aa450 con 0x7fcedc0043a0 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcee0003610 con 0x7fcedc0043a0 2024-08-19T09:50:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): dump: 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.271 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3831225552 0 0) 0x7fceec1aedb0 con 0x7fceec0a67c0 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fced0004eb0 con 0x7fceec0a67c0 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcee40050b0 con 0x7fceec0a67c0 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient(hunting): dump: 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.272 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 361992141 0 0) 0x7fceec0aa450 con 0x7fcedc0043a0 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.1:6792/0 conn(0x7fceec0a67c0 legacy=0x7fceec1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.82:6790/0 conn(0x7fcedc005140 legacy=0x7fceec1d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 monclient: found mon.a 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceec1ae510 con 0x7fcedc0043a0 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 5 monclient: authenticate success, global_id 4284 2024-08-19T09:50:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 10 monclient: _renew_subs 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fceec1ae7f0 con 0x7fcedc0043a0 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 10 monclient: _renew_subs 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fceec1c1b00 con 0x7fcedc0043a0 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcef317f640 1 librados: init done 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcee0003b10 con 0x7fcedc0043a0 2024-08-19T09:50:18.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcee0004040 con 0x7fcedc0043a0 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fceeaffd640 4 set_mon_vals no callback set 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.255+0000 7fcee97fa640 10 monclient: dump: 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.276 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcef317f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcef317f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcee00041e0 con 0x7fcedc0043a0 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcef317f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcecc005080 con 0x7fcedc0043a0 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fcee004dac0 con 0x7fcedc0043a0 2024-08-19T09:50:18.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcee97fa640 10 client.4284.objecter ms_dispatch 0x7fceec000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcee97fa640 3 client.4284.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcee97fa640 3 client.4284.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fcee97fa640 20 client.4284.objecter dump_active .. 0 homeless 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.256+0000 7fceebfff640 10 monclient: get_auth_request con 0x7fced0041510 auth_method 0 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.259+0000 7fcee97fa640 10 client.4284.objecter ms_handle_connect 0x7fced0041510 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.261+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fcee00175e0 con 0x7fcedc0043a0 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.261+0000 7fcee97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.261+0000 7fcee97fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:18.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.494+0000 7fcef317f640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-08-19T09:50:18.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.494+0000 7fcef317f640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-08-19T09:50:18.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.495+0000 7fcef317f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.495+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcecc005370 con 0x7fcedc0043a0 2024-08-19T09:50:18.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.496+0000 7fcee97fa640 1 -- 172.21.15.1:0/732100068 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+96221 (unknown 481731318 0 988825745) 0x7fcee0016ee0 con 0x7fcedc0043a0 2024-08-19T09:50:18.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.496+0000 7fcee97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-08-19T09:50:18.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.496+0000 7fcee97fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:18.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 10 librados: watch_flush enter 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 10 librados: watch_flush exit 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 20 client.4284.objecter shutdown clearing up homeless session... 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 10 client.4284.objecter successfully canceled tick 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.82:6812/2549104415 conn(0x7fced0041510 legacy=0x7fced00439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 10 monclient: shutdown 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 >> v1:172.21.15.1:6789/0 conn(0x7fcedc0043a0 legacy=0x7fceec1c13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 shutdown_connections 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 >> 172.21.15.1:0/732100068 conn(0x7fceec0ac4b0 msgr2=0x7fceec0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:18.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 shutdown_connections 2024-08-19T09:50:18.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 -- 172.21.15.1:0/732100068 wait complete. 2024-08-19T09:50:18.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.498+0000 7fcef317f640 1 librados: shutdown 2024-08-19T09:50:18.509 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4116,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6812","nonce":2549104415}]},"active_addr":"172.21.15.82:6812/2549104415","active_change":"2024-08-19T09:50:00.380797+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":2363465578}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":621385039}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3157788984}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3789040331}]}]} 2024-08-19T09:50:18.512 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-19T09:50:18.512 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-19T09:50:18.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T09:50:18.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.609+0000 7f56bdeda640 10 monclient: get_monmap_and_config 2024-08-19T09:50:18.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.609+0000 7f56bdeda640 10 monclient: build_initial_monmap 2024-08-19T09:50:18.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.609+0000 7f56bdeda640 10 monclient: monmap: 2024-08-19T09:50:18.610 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:18.610 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.610 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:18.610112+0000 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:18.610112+0000 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:18.611 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 1 Processor -- start 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 1 -- start start 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient: init 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient: _add_conns ranks=[3,5,1,0,7,6,8,2,4] 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient(hunting): picked mon.noname-g con 0x7f56b80a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient(hunting): picked mon.noname-h con 0x7f56b80a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient(hunting): picked mon.noname-f con 0x7f56b813f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b80a3b40 con 0x7f56b813f5c0 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b80a8130 con 0x7f56b80a67c0 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b806ede0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56bdeda640 10 monclient(hunting): authenticate will time out at 674.686615s 2024-08-19T09:50:18.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.610+0000 7f56b77fe640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f56b80a67c0 0x7f56b80a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48364/0 (socket says 172.21.15.1:48364) 2024-08-19T09:50:18.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.611+0000 7f56b7fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f56b813f5c0 0x7f56b81419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:45488/0 (socket says 172.21.15.1:45488) 2024-08-19T09:50:18.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.611+0000 7f56b77fe640 1 -- 172.21.15.1:0/2586489751 learned_addr learned my addr 172.21.15.1:0/2586489751 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:18.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56affff640 1 -- 172.21.15.1:0/2586489751 >> v1:172.21.15.82:6791/0 conn(0x7f56b80a6ba0 legacy=0x7f56b80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b77fe640 1 -- 172.21.15.1:0/2586489751 >> v1:172.21.15.82:6790/0 conn(0x7f56b80a67c0 legacy=0x7f56b80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b7fff640 1 -- 172.21.15.1:0/2586489751 >> v1:172.21.15.82:6789/0 conn(0x7f56b813f5c0 legacy=0x7f56b81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464182558 0 0) 0x7f56b806ede0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): my global_id is 4289 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a4003520 con 0x7f56b80a6ba0 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249209378 0 0) 0x7f56b80a8130 con 0x7f56b80a67c0 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): my global_id is 4269 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56b806ede0 con 0x7f56b80a67c0 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282028832 0 0) 0x7f56b80a3b40 con 0x7f56b813f5c0 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient(hunting): my global_id is 4213 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56b80a8130 con 0x7f56b813f5c0 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2288525390 0 0) 0x7f56a4003520 con 0x7f56b80a6ba0 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 >> v1:172.21.15.82:6790/0 conn(0x7f56b80a67c0 legacy=0x7f56b80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 >> v1:172.21.15.82:6789/0 conn(0x7f56b813f5c0 legacy=0x7f56b81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 monclient: found mon.noname-h 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b80a92a0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f56a0001b20 con 0x7f56b80a6ba0 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: dump: 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.621 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56bdeda640 5 monclient: authenticate success, global_id 4289 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56bdeda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56a0003610 con 0x7f56b80a6ba0 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 1 -- 172.21.15.1:0/2586489751 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f56a0003b40 con 0x7f56b80a6ba0 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56b6ffd640 10 monclient: dump: 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56bdeda640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.613+0000 7f56bdeda640 4 set_mon_vals no callback set 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 10 monclient: shutdown 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 1 -- 172.21.15.1:0/2586489751 >> v1:172.21.15.82:6791/0 conn(0x7f56b80a6ba0 legacy=0x7f56b80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 1 -- 172.21.15.1:0/2586489751 shutdown_connections 2024-08-19T09:50:18.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 1 -- 172.21.15.1:0/2586489751 >> 172.21.15.1:0/2586489751 conn(0x7f56b80ac4b0 msgr2=0x7f56b80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 1 -- 172.21.15.1:0/2586489751 shutdown_connections 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 1 -- 172.21.15.1:0/2586489751 wait complete. 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.614+0000 7f56bdeda640 10 monclient: build_initial_monmap 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient: monmap: 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:18.625 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 librados: starting msgr at 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 librados: starting objecter 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 Processor -- start 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 -- start start 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 librados: setting wanted keys 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 librados: calling monclient init 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient: init 2024-08-19T09:50:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient: _add_conns ranks=[5,0,7,4,6,2,8,3,1] 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient(hunting): picked mon.noname-h con 0x7f56b80a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient(hunting): picked mon.noname-a con 0x7f56b80a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient(hunting): picked mon.noname-i con 0x7f56b813f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b80a17a0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b813aaa0 con 0x7f56b80a67c0 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b80a9cc0 con 0x7f56b813f5c0 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56bdeda640 10 monclient(hunting): authenticate will time out at 674.691833s 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56affff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56b80a6ba0 0x7f56b81d3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37110/0 (socket says 172.21.15.1:37110) 2024-08-19T09:50:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.615+0000 7f56affff640 1 -- 172.21.15.1:0/804704731 learned_addr learned my addr 172.21.15.1:0/804704731 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:18.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56affff640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.1:6789/0 conn(0x7f56b80a6ba0 legacy=0x7f56b81d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 client.?.objecter ms_handle_connect 0x7f56b80a6ba0 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 376226456 0 0) 0x7f56b80a17a0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): my global_id is 4293 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5698004eb0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b77fe640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.82:6791/0 conn(0x7f56b80a67c0 legacy=0x7f56b81ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 client.?.objecter ms_handle_connect 0x7f56b80a67c0 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 client.?.objecter ms_handle_connect 0x7f56b813f5c0 2024-08-19T09:50:18.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b7fff640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.82:6792/0 conn(0x7f56b813f5c0 legacy=0x7f56b81d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2585993736 0 0) 0x7f5698004eb0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56b80a17a0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f56a0003b00 con 0x7f56b80a6ba0 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.632 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3049575586 0 0) 0x7f56b813aaa0 con 0x7f56b80a67c0 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): my global_id is 4298 2024-08-19T09:50:18.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5698004eb0 con 0x7f56b80a67c0 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1212708949 0 0) 0x7f56b80a9cc0 con 0x7f56b813f5c0 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient(hunting): my global_id is 4255 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56b813aaa0 con 0x7f56b813f5c0 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4153181105 0 0) 0x7f56b80a17a0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.82:6792/0 conn(0x7f56b813f5c0 legacy=0x7f56b81d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.82:6791/0 conn(0x7f56b80a67c0 legacy=0x7f56b81ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 monclient: found mon.a 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b8142aa0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 5 monclient: authenticate success, global_id 4293 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 10 monclient: _renew_subs 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56b813a7c0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 10 monclient: _renew_subs 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f56b80a0fc0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56bdeda640 1 librados: init done 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56a0003610 con 0x7f56b80a6ba0 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f56a0004470 con 0x7f56b80a6ba0 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b67fc640 4 set_mon_vals no callback set 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.618+0000 7f56b4ff9640 10 monclient: dump: 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.636 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.619+0000 7f56bdeda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.619+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f56a0011a80 con 0x7f56b80a6ba0 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.619+0000 7f56bdeda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.619+0000 7f56bdeda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.619+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f569c005080 con 0x7f56b80a6ba0 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.620+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f56a004dad0 con 0x7f56b80a6ba0 2024-08-19T09:50:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.620+0000 7f56b4ff9640 10 client.4293.objecter ms_dispatch 0x7f56b8000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.620+0000 7f56b4ff9640 3 client.4293.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.620+0000 7f56b4ff9640 3 client.4293.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.620+0000 7f56b4ff9640 20 client.4293.objecter dump_active .. 0 homeless 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.620+0000 7f56b77fe640 10 monclient: get_auth_request con 0x7f5698041090 auth_method 0 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.622+0000 7f56b4ff9640 10 client.4293.objecter ms_handle_connect 0x7f5698041090 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.624+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f56a0017690 con 0x7f56b80a6ba0 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.624+0000 7f56b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.624+0000 7f56b4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.812+0000 7f56bdeda640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.812+0000 7f56bdeda640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.812+0000 7f56bdeda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.812+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f569c005370 con 0x7f56b80a6ba0 2024-08-19T09:50:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.813+0000 7f56b4ff9640 1 -- 172.21.15.1:0/804704731 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10897 (unknown 941590788 0 3619314660) 0x7f56a0016f90 con 0x7f56b80a6ba0 2024-08-19T09:50:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.813+0000 7f56b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.813+0000 7f56b4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:18.815 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:18.815 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:50:17.763343+0000","last_up_change":"2024-08-19T09:50:07.419178+0000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T09:50:07.171793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T09:50:13.629390+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":3862255507}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":3862255507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":3862255507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":3862255507}]},"public_addr":"172.21.15.1:6804/3862255507","cluster_addr":"172.21.15.1:6805/3862255507","heartbeat_back_addr":"172.21.15.1:6807/3862255507","heartbeat_front_addr":"172.21.15.1:6806/3862255507","state":["exists","up"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1668590231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1668590231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1668590231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1668590231}]},"public_addr":"172.21.15.1:6800/1668590231","cluster_addr":"172.21.15.1:6801/1668590231","heartbeat_back_addr":"172.21.15.1:6803/1668590231","heartbeat_front_addr":"172.21.15.1:6802/1668590231","state":["exists","up"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":207870648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":207870648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":207870648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":207870648}]},"public_addr":"172.21.15.1:6808/207870648","cluster_addr":"172.21.15.1:6809/207870648","heartbeat_back_addr":"172.21.15.1:6811/207870648","heartbeat_front_addr":"172.21.15.1:6810/207870648","state":["exists","up"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6804","nonce":394826682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6805","nonce":394826682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6807","nonce":394826682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6806","nonce":394826682}]},"public_addr":"172.21.15.82:6804/394826682","cluster_addr":"172.21.15.82:6805/394826682","heartbeat_back_addr":"172.21.15.82:6807/394826682","heartbeat_front_addr":"172.21.15.82:6806/394826682","state":["exists","up"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6800","nonce":2865329823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6801","nonce":2865329823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6803","nonce":2865329823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6802","nonce":2865329823}]},"public_addr":"172.21.15.82:6800/2865329823","cluster_addr":"172.21.15.82:6801/2865329823","heartbeat_back_addr":"172.21.15.82:6803/2865329823","heartbeat_front_addr":"172.21.15.82:6802/2865329823","state":["exists","up"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6808","nonce":1813220032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6809","nonce":1813220032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6811","nonce":1813220032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6810","nonce":1813220032}]},"public_addr":"172.21.15.82:6808/1813220032","cluster_addr":"172.21.15.82:6809/1813220032","heartbeat_back_addr":"172.21.15.82:6811/1813220032","heartbeat_front_addr":"172.21.15.82:6810/1813220032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.564842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.315230+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.809229+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:05.004322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.803207+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.616501+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 10 librados: watch_flush enter 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 10 librados: watch_flush exit 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 20 client.4293.objecter shutdown clearing up homeless session... 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 10 client.4293.objecter successfully canceled tick 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5698041090 legacy=0x7f5698043550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 10 monclient: shutdown 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 >> v1:172.21.15.1:6789/0 conn(0x7f56b80a6ba0 legacy=0x7f56b81d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 shutdown_connections 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 >> 172.21.15.1:0/804704731 conn(0x7f56b80ac4b0 msgr2=0x7f56b80a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 shutdown_connections 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 -- 172.21.15.1:0/804704731 wait complete. 2024-08-19T09:50:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.815+0000 7f56bdeda640 1 librados: shutdown 2024-08-19T09:50:18.827 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-19T09:50:18.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T09:50:18.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.924+0000 7fcc39faa640 10 monclient: get_monmap_and_config 2024-08-19T09:50:18.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.924+0000 7fcc39faa640 10 monclient: build_initial_monmap 2024-08-19T09:50:18.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.924+0000 7fcc39faa640 10 monclient: monmap: 2024-08-19T09:50:18.926 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:18.926 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:18.925578+0000 2024-08-19T09:50:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:18.925578+0000 2024-08-19T09:50:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:18.928 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:18.928 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:18.928 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:18.928 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:18.928 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:18.928 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:18.929 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 1 Processor -- start 2024-08-19T09:50:18.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 1 -- start start 2024-08-19T09:50:18.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 10 monclient: init 2024-08-19T09:50:18.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 10 monclient: _add_conns ranks=[8,1,7,5,0,2,6,3,4] 2024-08-19T09:50:18.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 10 monclient(hunting): picked mon.noname-e con 0x7fcc340a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:18.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.925+0000 7fcc39faa640 10 monclient(hunting): picked mon.noname-f con 0x7fcc340a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:18.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc39faa640 10 monclient(hunting): picked mon.noname-i con 0x7fcc3413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:18.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc39faa640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a3b40 con 0x7fcc340a6ba0 2024-08-19T09:50:18.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc39faa640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a8130 con 0x7fcc3413f5c0 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc39faa640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3406ede0 con 0x7fcc340a67c0 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc39faa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc39faa640 10 monclient(hunting): authenticate will time out at 675.002098s 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc337fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcc340a67c0 0x7fcc340a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44598/0 (socket says 172.21.15.1:44598) 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.926+0000 7fcc337fe640 1 -- 172.21.15.1:0/3270640140 learned_addr learned my addr 172.21.15.1:0/3270640140 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc337fe640 1 -- 172.21.15.1:0/3270640140 >> v1:172.21.15.1:6793/0 conn(0x7fcc340a67c0 legacy=0x7fcc340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2271779792 0 0) 0x7fcc3406ede0 con 0x7fcc340a67c0 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient(hunting): my global_id is 4238 2024-08-19T09:50:18.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc1c003520 con 0x7fcc340a67c0 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc32ffd640 1 -- 172.21.15.1:0/3270640140 >> v1:172.21.15.82:6789/0 conn(0x7fcc340a6ba0 legacy=0x7fcc340a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc33fff640 1 -- 172.21.15.1:0/3270640140 >> v1:172.21.15.82:6792/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1395318259 0 0) 0x7fcc1c003520 con 0x7fcc340a67c0 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 >> v1:172.21.15.82:6792/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 >> v1:172.21.15.82:6789/0 conn(0x7fcc340a6ba0 legacy=0x7fcc340a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 monclient: found mon.noname-e 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340a92a0 con 0x7fcc340a67c0 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc24001ad0 con 0x7fcc340a67c0 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc39faa640 5 monclient: authenticate success, global_id 4238 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc39faa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: dump: 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.951 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc39faa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc240048a0 con 0x7fcc340a67c0 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 1 -- 172.21.15.1:0/3270640140 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc24004dd0 con 0x7fcc340a67c0 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc327fc640 10 monclient: dump: 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc39faa640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.928+0000 7fcc39faa640 4 set_mon_vals no callback set 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 10 monclient: shutdown 2024-08-19T09:50:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 1 -- 172.21.15.1:0/3270640140 >> v1:172.21.15.1:6793/0 conn(0x7fcc340a67c0 legacy=0x7fcc340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 1 -- 172.21.15.1:0/3270640140 shutdown_connections 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 1 -- 172.21.15.1:0/3270640140 >> 172.21.15.1:0/3270640140 conn(0x7fcc340ac4b0 msgr2=0x7fcc340a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 1 -- 172.21.15.1:0/3270640140 shutdown_connections 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 1 -- 172.21.15.1:0/3270640140 wait complete. 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 10 monclient: build_initial_monmap 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.929+0000 7fcc39faa640 10 monclient: monmap: 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 librados: starting msgr at 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 librados: starting objecter 2024-08-19T09:50:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 Processor -- start 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 -- start start 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 librados: setting wanted keys 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 librados: calling monclient init 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient: init 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient: _add_conns ranks=[8,0,3,4,1,5,6,2,7] 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient(hunting): picked mon.noname-e con 0x7fcc340a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc340a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient(hunting): picked mon.noname-g con 0x7fcc3413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc34139a90 con 0x7fcc340a6ba0 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc341cf020 con 0x7fcc3413f5c0 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a9cc0 con 0x7fcc340a67c0 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:18.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.930+0000 7fcc39faa640 10 monclient(hunting): authenticate will time out at 675.006992s 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.931+0000 7fcc337fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcc340a67c0 0x7fcc341ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44610/0 (socket says 172.21.15.1:44610) 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.931+0000 7fcc337fe640 1 -- 172.21.15.1:0/1506341070 learned_addr learned my addr 172.21.15.1:0/1506341070 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc32ffd640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.1:6789/0 conn(0x7fcc340a6ba0 legacy=0x7fcc341ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 client.?.objecter ms_handle_connect 0x7fcc340a6ba0 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc337fe640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.1:6793/0 conn(0x7fcc340a67c0 legacy=0x7fcc341ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 client.?.objecter ms_handle_connect 0x7fcc340a67c0 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1518473455 0 0) 0x7fcc340a9cc0 con 0x7fcc340a67c0 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): my global_id is 4247 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc20004eb0 con 0x7fcc340a67c0 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557787836 0 0) 0x7fcc34139a90 con 0x7fcc340a6ba0 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): my global_id is 4302 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc340a9cc0 con 0x7fcc340a6ba0 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 client.?.objecter ms_handle_connect 0x7fcc3413f5c0 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc33fff640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.82:6790/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846140832 0 0) 0x7fcc341cf020 con 0x7fcc3413f5c0 2024-08-19T09:50:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): my global_id is 4278 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc34139a90 con 0x7fcc3413f5c0 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3036062486 0 0) 0x7fcc20004eb0 con 0x7fcc340a67c0 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc341cf020 con 0x7fcc340a67c0 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc24004d90 con 0x7fcc340a67c0 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): dump: 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 221655788 0 0) 0x7fcc340a9cc0 con 0x7fcc340a6ba0 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc20004eb0 con 0x7fcc340a6ba0 2024-08-19T09:50:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc28003610 con 0x7fcc340a6ba0 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient(hunting): dump: 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 822869510 0 0) 0x7fcc341cf020 con 0x7fcc340a67c0 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.82:6790/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.1:6789/0 conn(0x7fcc340a6ba0 legacy=0x7fcc341ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 monclient: found mon.e 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3413b550 con 0x7fcc340a67c0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient: _finish_auth 0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 5 monclient: authenticate success, global_id 4247 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 10 monclient: _renew_subs 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3413e5f0 con 0x7fcc340a67c0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 10 monclient: _renew_subs 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc3413d220 con 0x7fcc340a67c0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.933+0000 7fcc39faa640 1 librados: init done 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc24001a70 con 0x7fcc340a67c0 2024-08-19T09:50:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc1bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc240048a0 con 0x7fcc340a67c0 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc31ffb640 4 set_mon_vals no callback set 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc1bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.934+0000 7fcc1bfff640 10 monclient: dump: 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc39faa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcc240119c0 con 0x7fcc340a67c0 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc39faa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc39faa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc14005080 con 0x7fcc340a67c0 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fcc2404db10 con 0x7fcc340a67c0 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc1bfff640 10 client.4247.objecter ms_dispatch 0x7fcc34000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc1bfff640 3 client.4247.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc1bfff640 3 client.4247.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc1bfff640 20 client.4247.objecter dump_active .. 0 homeless 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.935+0000 7fcc32ffd640 10 monclient: get_auth_request con 0x7fcc20041650 auth_method 0 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.937+0000 7fcc1bfff640 10 client.4247.objecter ms_handle_connect 0x7fcc20041650 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.939+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fcc24017670 con 0x7fcc340a67c0 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.939+0000 7fcc1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:18.939+0000 7fcc1bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.124+0000 7fcc39faa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:19.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.125+0000 7fcc39faa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:19.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.125+0000 7fcc39faa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.125+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc14005370 con 0x7fcc340a67c0 2024-08-19T09:50:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.125+0000 7fcc1bfff640 1 -- 172.21.15.1:0/1506341070 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10897 (unknown 941590788 0 3619314660) 0x7fcc24016f70 con 0x7fcc340a67c0 2024-08-19T09:50:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.125+0000 7fcc1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.125+0000 7fcc1bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:19.127 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:19.127 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:50:17.763343+0000","last_up_change":"2024-08-19T09:50:07.419178+0000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T09:50:07.171793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T09:50:13.629390+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":3862255507}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":3862255507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":3862255507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":3862255507}]},"public_addr":"172.21.15.1:6804/3862255507","cluster_addr":"172.21.15.1:6805/3862255507","heartbeat_back_addr":"172.21.15.1:6807/3862255507","heartbeat_front_addr":"172.21.15.1:6806/3862255507","state":["exists","up"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1668590231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1668590231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1668590231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1668590231}]},"public_addr":"172.21.15.1:6800/1668590231","cluster_addr":"172.21.15.1:6801/1668590231","heartbeat_back_addr":"172.21.15.1:6803/1668590231","heartbeat_front_addr":"172.21.15.1:6802/1668590231","state":["exists","up"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":207870648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":207870648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":207870648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":207870648}]},"public_addr":"172.21.15.1:6808/207870648","cluster_addr":"172.21.15.1:6809/207870648","heartbeat_back_addr":"172.21.15.1:6811/207870648","heartbeat_front_addr":"172.21.15.1:6810/207870648","state":["exists","up"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6804","nonce":394826682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6805","nonce":394826682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6807","nonce":394826682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6806","nonce":394826682}]},"public_addr":"172.21.15.82:6804/394826682","cluster_addr":"172.21.15.82:6805/394826682","heartbeat_back_addr":"172.21.15.82:6807/394826682","heartbeat_front_addr":"172.21.15.82:6806/394826682","state":["exists","up"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6800","nonce":2865329823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6801","nonce":2865329823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6803","nonce":2865329823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6802","nonce":2865329823}]},"public_addr":"172.21.15.82:6800/2865329823","cluster_addr":"172.21.15.82:6801/2865329823","heartbeat_back_addr":"172.21.15.82:6803/2865329823","heartbeat_front_addr":"172.21.15.82:6802/2865329823","state":["exists","up"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6808","nonce":1813220032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6809","nonce":1813220032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6811","nonce":1813220032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6810","nonce":1813220032}]},"public_addr":"172.21.15.82:6808/1813220032","cluster_addr":"172.21.15.82:6809/1813220032","heartbeat_back_addr":"172.21.15.82:6811/1813220032","heartbeat_front_addr":"172.21.15.82:6810/1813220032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.564842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.315230+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.809229+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:05.004322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.803207+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.616501+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 10 librados: watch_flush enter 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 10 librados: watch_flush exit 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 20 client.4247.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 10 client.4247.objecter successfully canceled tick 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcc20041650 legacy=0x7fcc20043b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 10 monclient: shutdown 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 >> v1:172.21.15.1:6793/0 conn(0x7fcc340a67c0 legacy=0x7fcc341ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 shutdown_connections 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 >> 172.21.15.1:0/1506341070 conn(0x7fcc340ac4b0 msgr2=0x7fcc34142130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 shutdown_connections 2024-08-19T09:50:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 -- 172.21.15.1:0/1506341070 wait complete. 2024-08-19T09:50:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.127+0000 7fcc39faa640 1 librados: shutdown 2024-08-19T09:50:19.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-19T09:50:19.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-19T09:50:19.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-19T09:50:19.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-19T09:50:19.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-19T09:50:19.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-19T09:50:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.254+0000 7fa7f2533640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.254+0000 7f56a839f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.254+0000 7f56a839f640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.254+0000 7f56a839f640 10 monclient: monmap: 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.255740+0000 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.255740+0000 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.257 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.254+0000 7fa7f2533640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.254+0000 7fa7f2533640 10 monclient: monmap: 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.255624+0000 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.255624+0000 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.255+0000 7f56a839f640 1 Processor -- start 2024-08-19T09:50:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.255+0000 7f56a839f640 1 -- start start 2024-08-19T09:50:19.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.255+0000 7f56a839f640 10 monclient: init 2024-08-19T09:50:19.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.255+0000 7fa7f2533640 1 Processor -- start 2024-08-19T09:50:19.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.255+0000 7fa7f2533640 1 -- start start 2024-08-19T09:50:19.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.255+0000 7fa7f2533640 10 monclient: init 2024-08-19T09:50:19.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient: _add_conns ranks=[3,5,1,7,2,4,0,6,8] 2024-08-19T09:50:19.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient(hunting): picked mon.noname-g con 0x7fa7ec0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient(hunting): picked mon.noname-h con 0x7fa7ec0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient(hunting): picked mon.noname-f con 0x7fa7ec0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 10 monclient: _add_conns ranks=[5,6,8,7,4,2,0,3,1] 2024-08-19T09:50:19.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0a3c70 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0a33d0 con 0x7fa7ec0a6830 2024-08-19T09:50:19.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0a36b0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 10 monclient(hunting): picked mon.noname-h con 0x7f56a009b460 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7f2533640 10 monclient(hunting): authenticate will time out at 675.332603s 2024-08-19T09:50:19.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 10 monclient(hunting): picked mon.noname-d con 0x7f56a009b840 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7ebfff640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fa7ec0a6830 0x7fa7ec0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48380/0 (socket says 172.21.15.1:48380) 2024-08-19T09:50:19.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7fa7ebfff640 1 -- 172.21.15.1:0/4158363329 learned_addr learned my addr 172.21.15.1:0/4158363329 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 10 monclient(hunting): picked mon.noname-e con 0x7f56a009bc90 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a00a0060 con 0x7f56a009b460 2024-08-19T09:50:19.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a839f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a00a0b20 con 0x7f56a009b840 2024-08-19T09:50:19.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a593a640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f56a009b840 0x7f56a013a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52820/0 (socket says 172.21.15.1:52820) 2024-08-19T09:50:19.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.256+0000 7f56a593a640 1 -- 172.21.15.1:0/2249422103 learned_addr learned my addr 172.21.15.1:0/2249422103 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.257+0000 7f56a839f640 1 -- 172.21.15.1:0/2249422103 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a00a0e00 con 0x7f56a009bc90 2024-08-19T09:50:19.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.257+0000 7f56a839f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.257+0000 7f56a839f640 10 monclient(hunting): authenticate will time out at 675.333042s 2024-08-19T09:50:19.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.258+0000 7fa7ebfff640 1 -- 172.21.15.1:0/4158363329 >> v1:172.21.15.82:6790/0 conn(0x7fa7ec0a6830 legacy=0x7fa7ec0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eb7fe640 1 -- 172.21.15.1:0/4158363329 >> v1:172.21.15.82:6791/0 conn(0x7fa7ec0694b0 legacy=0x7fa7ec0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a593a640 1 -- 172.21.15.1:0/2249422103 >> v1:172.21.15.1:6792/0 conn(0x7f56a009b840 legacy=0x7f56a013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7f0ad0640 1 -- 172.21.15.1:0/4158363329 >> v1:172.21.15.82:6789/0 conn(0x7fa7ec0a46a0 legacy=0x7fa7ec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a693c640 1 -- 172.21.15.1:0/2249422103 >> v1:172.21.15.1:6793/0 conn(0x7f56a009bc90 legacy=0x7f56a013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a613b640 1 -- 172.21.15.1:0/2249422103 >> v1:172.21.15.82:6791/0 conn(0x7f56a009b460 legacy=0x7f56a00a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203220274 0 0) 0x7f56a00a0b20 con 0x7f56a009b840 2024-08-19T09:50:19.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368376458 0 0) 0x7fa7ec0a36b0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): my global_id is 4307 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d4003520 con 0x7fa7ec0694b0 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504225982 0 0) 0x7fa7ec0a33d0 con 0x7fa7ec0a6830 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): my global_id is 4287 2024-08-19T09:50:19.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ec0a36b0 con 0x7fa7ec0a6830 2024-08-19T09:50:19.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906755587 0 0) 0x7fa7ec0a3c70 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient(hunting): my global_id is 4231 2024-08-19T09:50:19.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ec0a33d0 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): my global_id is 4245 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568c003520 con 0x7f56a009b840 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307046366 0 0) 0x7f56a00a0e00 con 0x7f56a009bc90 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): my global_id is 4256 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a00a0b20 con 0x7f56a009bc90 2024-08-19T09:50:19.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3505375366 0 0) 0x7f56a00a0060 con 0x7f56a009b460 2024-08-19T09:50:19.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient(hunting): my global_id is 4316 2024-08-19T09:50:19.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a00a0e00 con 0x7f56a009b460 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3949157342 0 0) 0x7fa7d4003520 con 0x7fa7ec0694b0 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 >> v1:172.21.15.82:6790/0 conn(0x7fa7ec0a6830 legacy=0x7fa7ec0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 >> v1:172.21.15.82:6789/0 conn(0x7fa7ec0a46a0 legacy=0x7fa7ec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 monclient: found mon.noname-h 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec06dbb0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7dc001b30 con 0x7fa7ec0694b0 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7eaffd640 10 monclient: dump: 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.274 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7f2533640 5 monclient: authenticate success, global_id 4307 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7fa7f2533640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2720115934 0 0) 0x7f568c003520 con 0x7f56a009b840 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7dc003610 con 0x7fa7ec0694b0 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7eaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7eaffd640 1 -- 172.21.15.1:0/4158363329 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7dc003b00 con 0x7fa7ec0694b0 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7eaffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7eaffd640 10 monclient: dump: 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7f2533640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7f2533640 4 set_mon_vals no callback set 2024-08-19T09:50:19.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 >> v1:172.21.15.1:6793/0 conn(0x7f56a009bc90 legacy=0x7f56a013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 >> v1:172.21.15.82:6791/0 conn(0x7f56a009b460 legacy=0x7f56a00a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 monclient: found mon.noname-d 2024-08-19T09:50:19.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a00a5a40 con 0x7f56a009b840 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a5139640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a839f640 5 monclient: authenticate success, global_id 4245 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.259+0000 7f56a839f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5690001ba0 con 0x7f56a009b840 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: dump: 2024-08-19T09:50:19.281 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a839f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56900048a0 con 0x7f56a009b840 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a839f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a839f640 4 set_mon_vals no callback set 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 1 -- 172.21.15.1:0/2249422103 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5690004d90 con 0x7f56a009b840 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7f56a5139640 10 monclient: dump: 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.284 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.285 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7f2533640 10 monclient: shutdown 2024-08-19T09:50:19.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7f2533640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.260+0000 7fa7f2533640 1 -- 172.21.15.1:0/4158363329 >> v1:172.21.15.82:6791/0 conn(0x7fa7ec0694b0 legacy=0x7fa7ec0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7fa7f2533640 1 -- 172.21.15.1:0/4158363329 shutdown_connections 2024-08-19T09:50:19.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7fa7f2533640 1 -- 172.21.15.1:0/4158363329 >> 172.21.15.1:0/4158363329 conn(0x7fa7ec0a9020 msgr2=0x7fa7ec068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 10 monclient: shutdown 2024-08-19T09:50:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 1 -- 172.21.15.1:0/2249422103 >> v1:172.21.15.1:6792/0 conn(0x7f56a009b840 legacy=0x7f56a013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7fa7f2533640 1 -- 172.21.15.1:0/4158363329 shutdown_connections 2024-08-19T09:50:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7fa7f2533640 1 -- 172.21.15.1:0/4158363329 wait complete. 2024-08-19T09:50:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 1 -- 172.21.15.1:0/2249422103 shutdown_connections 2024-08-19T09:50:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 1 -- 172.21.15.1:0/2249422103 >> 172.21.15.1:0/2249422103 conn(0x7f56a00a4cb0 msgr2=0x7f56a00a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 1 -- 172.21.15.1:0/2249422103 shutdown_connections 2024-08-19T09:50:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.261+0000 7f56a839f640 1 -- 172.21.15.1:0/2249422103 wait complete. 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 10 monclient: monmap: 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.289 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 10 monclient: monmap: 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.291 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 1 librados: starting msgr at 2024-08-19T09:50:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 1 librados: starting objecter 2024-08-19T09:50:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 1 librados: starting msgr at 2024-08-19T09:50:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 1 librados: starting objecter 2024-08-19T09:50:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 1 Processor -- start 2024-08-19T09:50:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 1 Processor -- start 2024-08-19T09:50:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 1 -- start start 2024-08-19T09:50:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 1 librados: setting wanted keys 2024-08-19T09:50:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 1 librados: calling monclient init 2024-08-19T09:50:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7f56a839f640 10 monclient: init 2024-08-19T09:50:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 1 -- start start 2024-08-19T09:50:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 1 librados: setting wanted keys 2024-08-19T09:50:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 1 librados: calling monclient init 2024-08-19T09:50:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.262+0000 7fa7f2533640 10 monclient: init 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient: _add_conns ranks=[4,8,0,3,7,5,1,2,6] 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient(hunting): picked mon.noname-c con 0x7f56a009b460 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient(hunting): picked mon.noname-e con 0x7f56a009b840 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient(hunting): picked mon.noname-a con 0x7f56a009bc90 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a01ca6f0 con 0x7f56a009bc90 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a0073970 con 0x7f56a009b460 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a00a33b0 con 0x7f56a009b840 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a839f640 10 monclient(hunting): authenticate will time out at 675.339412s 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a693c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56a009bc90 0x7f56a01d1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37130/0 (socket says 172.21.15.1:37130) 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7f56a693c640 1 -- 172.21.15.1:0/717025457 learned_addr learned my addr 172.21.15.1:0/717025457 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient: _add_conns ranks=[3,8,2,7,1,5,4,0,6] 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient(hunting): picked mon.noname-g con 0x7fa7ec0694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient(hunting): picked mon.noname-e con 0x7fa7ec0a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ec0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0a9f10 con 0x7fa7ec0a6830 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec13ad10 con 0x7fa7ec0694b0 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec13aff0 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7f2533640 10 monclient(hunting): authenticate will time out at 675.339519s 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7eb7fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fa7ec0a46a0 0x7fa7ec137150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44624/0 (socket says 172.21.15.1:44624) 2024-08-19T09:50:19.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.263+0000 7fa7eb7fe640 1 -- 172.21.15.1:0/1458090130 learned_addr learned my addr 172.21.15.1:0/1458090130 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f56a693c640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.1:6789/0 conn(0x7f56a009bc90 legacy=0x7f56a01d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f5696ffd640 10 client.?.objecter ms_handle_connect 0x7f56a009bc90 2024-08-19T09:50:19.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f5696ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f5696ffd640 10 client.?.objecter ms_handle_connect 0x7f56a009b840 2024-08-19T09:50:19.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f5696ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f56a593a640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.1:6793/0 conn(0x7f56a009b840 legacy=0x7f56a01c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f56a613b640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.1:6791/0 conn(0x7f56a009b460 legacy=0x7f56a01c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:50:19.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7fa7ebfff640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.82:6790/0 conn(0x7fa7ec0694b0 legacy=0x7fa7ec136a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.265+0000 7fa7eb7fe640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.1:6793/0 conn(0x7fa7ec0a46a0 legacy=0x7fa7ec137150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7f0ad0640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.1:6790/0 conn(0x7fa7ec0a6830 legacy=0x7fa7ec1d6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa7ec0694b0 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa7ec0a6830 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa7ec0a46a0 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119311271 0 0) 0x7fa7ec13ad10 con 0x7fa7ec0694b0 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): my global_id is 4296 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d8004eb0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241834864 0 0) 0x7fa7ec0a33d0 con 0x7fa7ec0a6830 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): my global_id is 4259 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ec13ad10 con 0x7fa7ec0a6830 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 290317411 0 0) 0x7fa7ec13aff0 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): my global_id is 4274 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ec0a33d0 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4010664266 0 0) 0x7fa7d8004eb0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7ec13aff0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7e0001d00 con 0x7fa7ec0694b0 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3035380830 0 0) 0x7fa7ec0a33d0 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7d8004eb0 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7dc003b00 con 0x7fa7ec0a46a0 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1880186589 0 0) 0x7fa7ec13ad10 con 0x7fa7ec0a6830 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7ec0a33d0 con 0x7fa7ec0a6830 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3797186628 0 0) 0x7fa7ec13aff0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.1:6793/0 conn(0x7fa7ec0a46a0 legacy=0x7fa7ec137150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.1:6790/0 conn(0x7fa7ec0a6830 legacy=0x7fa7ec1d6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 monclient: found mon.g 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec137860 con 0x7fa7ec0694b0 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.266+0000 7fa7e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7e0004c70 con 0x7fa7ec0694b0 2024-08-19T09:50:19.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7e8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7e0005160 con 0x7fa7ec0694b0 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7e8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7e8ff9640 10 monclient: dump: 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.312 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7ea7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 5 monclient: authenticate success, global_id 4296 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 10 monclient: _renew_subs 2024-08-19T09:50:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ec1384c0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 10 monclient: _renew_subs 2024-08-19T09:50:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7ec1392c0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7fa7f2533640 1 librados: init done 2024-08-19T09:50:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.267+0000 7f5696ffd640 10 client.?.objecter ms_handle_connect 0x7f56a009b460 2024-08-19T09:50:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa7e00115c0 con 0x7fa7ec0694b0 2024-08-19T09:50:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7f2533640 10 client.4296.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4061585887 0 0) 0x7f56a01ca6f0 con 0x7f56a009bc90 2024-08-19T09:50:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): my global_id is 4311 2024-08-19T09:50:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5684004eb0 con 0x7f56a009bc90 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213977587 0 0) 0x7f56a0073970 con 0x7f56a009b460 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): my global_id is 4252 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a01ca6f0 con 0x7f56a009b460 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 934296493 0 0) 0x7f56a00a33b0 con 0x7f56a009b840 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): my global_id is 4265 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a0073970 con 0x7f56a009b840 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 671498945 0 0) 0x7f5684004eb0 con 0x7f56a009bc90 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56a00a33b0 con 0x7f56a009bc90 2024-08-19T09:50:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5698003990 con 0x7f56a009bc90 2024-08-19T09:50:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): dump: 2024-08-19T09:50:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7f2533640 15 client.4296.objecter _session_command_op_assign -1 1 2024-08-19T09:50:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7f2533640 20 client.4296.objecter _get_session osd=-1 returning homeless 2024-08-19T09:50:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7f2533640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2831716078 0 0) 0x7f56a0073970 con 0x7f56a009b840 2024-08-19T09:50:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7f2533640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.82:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa7ec051b00 con 0x7fa7ec0694b0 2024-08-19T09:50:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5684004eb0 con 0x7f56a009b840 2024-08-19T09:50:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5690004d90 con 0x7f56a009b840 2024-08-19T09:50:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fa7e004da40 con 0x7fa7ec0694b0 2024-08-19T09:50:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7e8ff9640 10 client.4296.objecter ms_dispatch 0x7fa7ec000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7e8ff9640 3 client.4296.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7fa7e8ff9640 3 client.4296.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 10 client.4296.objecter checking command 1 2024-08-19T09:50:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 20 client.4296.objecter _get_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): dump: 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 125145143 0 0) 0x7f56a01ca6f0 con 0x7f56a009b460 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56a0073970 con 0x7f56a009b460 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f569c001d10 con 0x7f56a009b460 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient(hunting): dump: 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.325 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 352412654 0 0) 0x7f56a00a33b0 con 0x7f56a009bc90 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.1:6793/0 conn(0x7f56a009b840 legacy=0x7f56a01c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.1:6791/0 conn(0x7f56a009b460 legacy=0x7f56a01c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 1 monclient: found mon.a 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.268+0000 7f5696ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a00730d0 con 0x7f56a009bc90 2024-08-19T09:50:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 5 monclient: authenticate success, global_id 4311 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 10 monclient: _renew_subs 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56a00733b0 con 0x7f56a009bc90 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 10 monclient: _renew_subs 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f56a0074110 con 0x7f56a009bc90 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a839f640 1 librados: init done 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5698003e90 con 0x7f56a009bc90 2024-08-19T09:50:19.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f56980043c0 con 0x7f56a009bc90 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f5696ffd640 10 monclient: dump: 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.328 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.329 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7f56a4938640 4 set_mon_vals no callback set 2024-08-19T09:50:19.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 15 client.4296.objecter _session_command_op_remove -1 1 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 20 client.4296.objecter _get_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 20 client.4296.objecter get_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 15 client.4296.objecter _session_command_op_assign 2 1 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 4 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 10 client.4296.objecter _send_command 1 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.1:6808/207870648 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7d804c2c0 con 0x7fa7d8048610 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7e8ff9640 20 client.4296.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7f0ad0640 10 monclient: get_auth_request con 0x7fa7d8048610 auth_method 0 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.269+0000 7fa7eb7fe640 10 monclient: get_auth_request con 0x7fa7d80419c0 auth_method 0 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== mon.3 v1:172.21.15.82:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fa7e0013580 con 0x7fa7ec0694b0 2024-08-19T09:50:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7fa7e8ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5698004560 con 0x7f56a009bc90 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f56a839f640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f56a839f640 15 client.4311.objecter _session_command_op_assign -1 1 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f56a839f640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f56a839f640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f56a839f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.1:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f56a009eda0 con 0x7f56a009bc90 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f569804ded0 con 0x7f56a009bc90 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f5696ffd640 10 client.4311.objecter ms_dispatch 0x7f56a0000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f5696ffd640 3 client.4311.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f5696ffd640 3 client.4311.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.270+0000 7f5696ffd640 10 client.4311.objecter checking command 1 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 20 client.4311.objecter _get_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 15 client.4311.objecter _session_command_op_remove -1 1 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 20 client.4311.objecter _get_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 20 client.4311.objecter get_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 15 client.4311.objecter _session_command_op_assign 4 1 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 4 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 10 client.4311.objecter _send_command 1 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.82:6800/2865329823 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f568404c340 con 0x7f5684048690 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f569804e310 con 0x7f56a009bc90 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f5696ffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f56a613b640 10 monclient: get_auth_request con 0x7f56840419e0 auth_method 0 2024-08-19T09:50:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7f56a593a640 10 monclient: get_auth_request con 0x7f5684048690 auth_method 0 2024-08-19T09:50:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.271+0000 7fa7e8ff9640 10 client.4296.objecter ms_handle_connect 0x7fa7d80419c0 2024-08-19T09:50:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.273+0000 7f5696ffd640 10 client.4311.objecter ms_handle_connect 0x7f56840419e0 2024-08-19T09:50:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.273+0000 7fa7e8ff9640 10 client.4296.objecter ms_handle_connect 0x7fa7d8048610 2024-08-19T09:50:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.274+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== osd.2 v1:172.21.15.1:6808/207870648 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa7d804c2c0 con 0x7fa7d8048610 2024-08-19T09:50:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.274+0000 7fa7e8ff9640 10 client.4296.objecter ms_dispatch 0x7fa7ec000b90 command_reply(tid 1: 0 ) 2024-08-19T09:50:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.274+0000 7fa7e8ff9640 10 client.4296.objecter _finish_command 1 = system:0 2024-08-19T09:50:19.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.274+0000 7fa7e8ff9640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 4 2024-08-19T09:50:19.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.274+0000 7fa7e8ff9640 15 client.4296.objecter _session_command_op_remove 2 1 2024-08-19T09:50:19.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.275+0000 7f5696ffd640 10 client.4311.objecter ms_handle_connect 0x7f5684048690 2024-08-19T09:50:19.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.276+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== osd.4 v1:172.21.15.82:6800/2865329823 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f568404c340 con 0x7f5684048690 2024-08-19T09:50:19.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.276+0000 7f5696ffd640 10 client.4311.objecter ms_dispatch 0x7f56a0000b90 command_reply(tid 1: 0 ) 2024-08-19T09:50:19.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.276+0000 7f5696ffd640 10 client.4311.objecter _finish_command 1 = system:0 2024-08-19T09:50:19.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.276+0000 7f5696ffd640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 4 2024-08-19T09:50:19.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.276+0000 7f5696ffd640 15 client.4311.objecter _session_command_op_remove 4 1 2024-08-19T09:50:19.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7f13150cc640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7f13150cc640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7f13150cc640 10 monclient: monmap: 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.293212+0000 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.293212+0000 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.338 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.339 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 10 client.4296.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 15 client.4296.objecter _session_command_op_assign -1 2 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 20 client.4296.objecter _get_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 20 client.4296.objecter _get_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 15 client.4296.objecter _session_command_op_remove -1 2 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 20 client.4296.objecter get_session s=0x7fa7d8047f60 osd=2 3 2024-08-19T09:50:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 15 client.4296.objecter _session_command_op_assign 2 2 2024-08-19T09:50:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 4 2024-08-19T09:50:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 10 client.4296.objecter _send_command 2 2024-08-19T09:50:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.292+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 --> v1:172.21.15.1:6808/207870648 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7ec143180 con 0x7fa7d8048610 2024-08-19T09:50:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7e8ff9640 1 -- 172.21.15.1:0/1458090130 <== osd.2 v1:172.21.15.1:6808/207870648 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7fa7ec143180 con 0x7fa7d8048610 2024-08-19T09:50:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7f13150cc640 1 Processor -- start 2024-08-19T09:50:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7e8ff9640 10 client.4296.objecter ms_dispatch 0x7fa7ec000b90 command_reply(tid 2: 0 ) 2024-08-19T09:50:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7e8ff9640 10 client.4296.objecter _finish_command 2 = system:0 2024-08-19T09:50:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7e8ff9640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 4 2024-08-19T09:50:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7e8ff9640 15 client.4296.objecter _session_command_op_remove 2 2 2024-08-19T09:50:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7f13150cc640 1 -- start start 2024-08-19T09:50:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7f13150cc640 10 monclient: init 2024-08-19T09:50:19.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 10 librados: watch_flush enter 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 10 librados: watch_flush exit 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 10 client.4296.objecter close_session for osd.2 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.1:6808/207870648 conn(0x7fa7d8048610 legacy=0x7fa7d804aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 20 client.4296.objecter put_session s=0x7fa7d8047f60 osd=2 1 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 20 client.4296.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.293+0000 7fa7f2533640 10 client.4296.objecter successfully canceled tick 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa7d80419c0 legacy=0x7fa7d8043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 10 monclient: shutdown 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient: _add_conns ranks=[0,3,7,4,8,2,1,5,6] 2024-08-19T09:50:19.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 >> v1:172.21.15.82:6790/0 conn(0x7fa7ec0694b0 legacy=0x7fa7ec136a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f0ad0640 1 -- 172.21.15.1:0/1458090130 reap_dead start 2024-08-19T09:50:19.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient(hunting): picked mon.noname-a con 0x7f13100a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient(hunting): picked mon.noname-g con 0x7f13100694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 shutdown_connections 2024-08-19T09:50:19.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 >> 172.21.15.1:0/1458090130 conn(0x7fa7ec0a9020 msgr2=0x7fa7ec0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f130ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f13100a6830 0x7f13100a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37134/0 (socket says 172.21.15.1:37134) 2024-08-19T09:50:19.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient(hunting): picked mon.noname-i con 0x7f13100a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13100a3c70 con 0x7f13100a6830 2024-08-19T09:50:19.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13100a33d0 con 0x7f13100694b0 2024-08-19T09:50:19.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13100a36b0 con 0x7f13100a46a0 2024-08-19T09:50:19.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f130f59e640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f13100a46a0 0x7f1310141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:58470/0 (socket says 172.21.15.1:58470) 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f130e59c640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f13100694b0 0x7f13100a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48402/0 (socket says 172.21.15.1:48402) 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f130f59e640 1 -- 172.21.15.1:0/3146910588 learned_addr learned my addr 172.21.15.1:0/3146910588 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7f13150cc640 10 monclient(hunting): authenticate will time out at 675.370828s 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 shutdown_connections 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.294+0000 7fa7f2533640 1 -- 172.21.15.1:0/1458090130 wait complete. 2024-08-19T09:50:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.295+0000 7fa7f2533640 1 librados: shutdown 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 15 client.4311.objecter _session_command_op_assign -1 2 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 20 client.4311.objecter _get_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 20 client.4311.objecter _get_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 15 client.4311.objecter _session_command_op_remove -1 2 2024-08-19T09:50:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 20 client.4311.objecter get_session s=0x7f5684047fe0 osd=4 3 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 15 client.4311.objecter _session_command_op_assign 4 2 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 4 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 10 client.4311.objecter _send_command 2 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 --> v1:172.21.15.82:6800/2865329823 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f56a0051b00 con 0x7f5684048690 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f5696ffd640 1 -- 172.21.15.1:0/717025457 <== osd.4 v1:172.21.15.82:6800/2865329823 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4245302165) 0x7f56a0051b00 con 0x7f5684048690 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f5696ffd640 10 client.4311.objecter ms_dispatch 0x7f56a0000b90 command_reply(tid 2: 0 ) 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f5696ffd640 10 client.4311.objecter _finish_command 2 = system:0 2024-08-19T09:50:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f5696ffd640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 4 2024-08-19T09:50:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f5696ffd640 15 client.4311.objecter _session_command_op_remove 4 2 2024-08-19T09:50:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.296+0000 7f130e59c640 1 -- 172.21.15.1:0/3146910588 >> v1:172.21.15.82:6790/0 conn(0x7f13100694b0 legacy=0x7f13100a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130f59e640 1 -- 172.21.15.1:0/3146910588 >> v1:172.21.15.82:6792/0 conn(0x7f13100a46a0 legacy=0x7f1310141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f5ee30ec640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f5ee30ec640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f5ee30ec640 10 monclient: monmap: 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.298046+0000 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.298046+0000 2024-08-19T09:50:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.353 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704081122 0 0) 0x7f13100a36b0 con 0x7f13100a46a0 2024-08-19T09:50:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient(hunting): my global_id is 4273 2024-08-19T09:50:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12f4003520 con 0x7f13100a46a0 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 10 librados: watch_flush enter 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 10 librados: watch_flush exit 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 10 client.4311.objecter close_session for osd.4 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.82:6800/2865329823 conn(0x7f5684048690 legacy=0x7f568404aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7fb360694640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7fb360694640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7fb360694640 10 monclient: monmap: 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.298313+0000 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.298313+0000 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.356 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 20 client.4311.objecter put_session s=0x7f5684047fe0 osd=4 1 2024-08-19T09:50:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 20 client.4311.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 10 client.4311.objecter successfully canceled tick 2024-08-19T09:50:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.82:6812/2549104415 conn(0x7f56840419e0 legacy=0x7f5684043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 10 monclient: shutdown 2024-08-19T09:50:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 >> v1:172.21.15.1:6789/0 conn(0x7f56a009bc90 legacy=0x7f56a01d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a693c640 1 -- 172.21.15.1:0/717025457 reap_dead start 2024-08-19T09:50:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 shutdown_connections 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 940120196 0 0) 0x7f12f4003520 con 0x7f13100a46a0 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 >> v1:172.21.15.82:6790/0 conn(0x7f13100694b0 legacy=0x7f13100a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 >> v1:172.21.15.1:6789/0 conn(0x7f13100a6830 legacy=0x7f13100a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 monclient: found mon.noname-i 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131006dbb0 con 0x7f13100a46a0 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1308001e60 con 0x7f13100a46a0 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f130dd9b640 10 monclient: dump: 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f13150cc640 5 monclient: authenticate success, global_id 4273 2024-08-19T09:50:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.297+0000 7f13150cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 1 Processor -- start 2024-08-19T09:50:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 >> 172.21.15.1:0/717025457 conn(0x7f56a00a4cb0 msgr2=0x7f56a01bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f5ee30ec640 1 Processor -- start 2024-08-19T09:50:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 1 -- start start 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 10 monclient: init 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f5ee30ec640 1 -- start start 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f5ee30ec640 10 monclient: init 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1308004c20 con 0x7f13100a46a0 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f130dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f130dd9b640 1 -- 172.21.15.1:0/3146910588 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1308005190 con 0x7f13100a46a0 2024-08-19T09:50:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f130dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f130dd9b640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f130dd9b640 10 monclient: dump: 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f13150cc640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f13150cc640 4 set_mon_vals no callback set 2024-08-19T09:50:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 shutdown_connections 2024-08-19T09:50:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f56a839f640 1 -- 172.21.15.1:0/717025457 wait complete. 2024-08-19T09:50:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7f56a839f640 1 librados: shutdown 2024-08-19T09:50:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 10 monclient: _add_conns ranks=[0,8,2,1,6,4,5,3,7] 2024-08-19T09:50:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 10 monclient(hunting): picked mon.noname-a con 0x7fb3580a1be0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.298+0000 7fb360694640 10 monclient(hunting): picked mon.noname-e con 0x7fb358136410 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient: _add_conns ranks=[1,0,8,7,5,2,6,4,3] 2024-08-19T09:50:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb360694640 10 monclient(hunting): picked mon.noname-b con 0x7fb35813c990 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient(hunting): picked mon.noname-f con 0x7f5edc0a4590 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb360694640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb358068c60 con 0x7fb3580a1be0 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb360694640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3580a4160 con 0x7fb35813c990 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb35dc2f640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fb358136410 0x7fb35813b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44632/0 (socket says 172.21.15.1:44632) 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb35dc2f640 1 -- 172.21.15.1:0/1166412165 learned_addr learned my addr 172.21.15.1:0/1166412165 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb360694640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3580a26d0 con 0x7fb358136410 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb360694640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7fb360694640 10 monclient(hunting): authenticate will time out at 675.375294s 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient(hunting): picked mon.noname-a con 0x7f5edc0738d0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient(hunting): picked mon.noname-e con 0x7f5edc13ca90 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5edc0720a0 con 0x7f5edc0738d0 2024-08-19T09:50:19.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 10 monclient: shutdown 2024-08-19T09:50:19.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 1 -- 172.21.15.1:0/3146910588 >> v1:172.21.15.82:6792/0 conn(0x7f13100a46a0 legacy=0x7f1310141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5edc0a5430 con 0x7f5edc0a4590 2024-08-19T09:50:19.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5edc072d70 con 0x7f5edc13ca90 2024-08-19T09:50:19.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 1 -- 172.21.15.1:0/3146910588 shutdown_connections 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 1 -- 172.21.15.1:0/3146910588 >> 172.21.15.1:0/3146910588 conn(0x7f13100a9020 msgr2=0x7f1310068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee30ec640 10 monclient(hunting): authenticate will time out at 675.375136s 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee1689640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f5edc13ca90 0x7f5edc13ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44648/0 (socket says 172.21.15.1:44648) 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5ee1689640 1 -- 172.21.15.1:0/222968891 learned_addr learned my addr 172.21.15.1:0/222968891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f5edbfff640 1 --1- 172.21.15.1:0/222968891 >> v1:172.21.15.1:6789/0 conn(0x7f5edc0738d0 0x7f5edc13b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37142/0 (socket says 172.21.15.1:37142) 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 1 -- 172.21.15.1:0/3146910588 shutdown_connections 2024-08-19T09:50:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.299+0000 7f13150cc640 1 -- 172.21.15.1:0/3146910588 wait complete. 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.300+0000 7f0eb3a8b640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.300+0000 7f0eb3a8b640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.300+0000 7f0eb3a8b640 10 monclient: monmap: 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.301041+0000 2024-08-19T09:50:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.301041+0000 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.300+0000 7f0eb3a8b640 1 Processor -- start 2024-08-19T09:50:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 1 -- start start 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 10 monclient: init 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f13150cc640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f13150cc640 10 monclient: monmap: 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35dc2f640 1 -- 172.21.15.1:0/1166412165 >> v1:172.21.15.1:6793/0 conn(0x7fb358136410 legacy=0x7fb35813b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 10 monclient: _add_conns ranks=[6,7,0,3,1,5,8,2,4] 2024-08-19T09:50:19.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f13150cc640 1 librados: starting msgr at 2024-08-19T09:50:19.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 10 monclient(hunting): picked mon.noname-d con 0x7f0eac0a5610 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35ec31640 1 -- 172.21.15.1:0/1166412165 >> v1:172.21.15.1:6790/0 conn(0x7fb35813c990 legacy=0x7fb35813ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 10 monclient(hunting): picked mon.noname-i con 0x7f0eac073940 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f13150cc640 1 librados: starting objecter 2024-08-19T09:50:19.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb3a8b640 10 monclient(hunting): picked mon.noname-a con 0x7f0eac0a6100 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f5ee1689640 1 -- 172.21.15.1:0/222968891 >> v1:172.21.15.1:6793/0 conn(0x7f5edc13ca90 legacy=0x7f5edc13ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f5edbfff640 1 -- 172.21.15.1:0/222968891 >> v1:172.21.15.1:6789/0 conn(0x7f5edc0738d0 legacy=0x7f5edc13b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190492663 0 0) 0x7fb3580a4160 con 0x7fb35813c990 2024-08-19T09:50:19.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35d42e640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35d42e640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35d42e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35d42e640 10 monclient(hunting): my global_id is 4268 2024-08-19T09:50:19.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb344003520 con 0x7fb35813c990 2024-08-19T09:50:19.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.301+0000 7f0eb1827640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f0eac0a5610 0x7f0eac0a59f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52836/0 (socket says 172.21.15.1:52836) 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f0eb1827640 1 -- 172.21.15.1:0/2887425223 learned_addr learned my addr 172.21.15.1:0/2887425223 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac072f60 con 0x7f0eac0a6100 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac074340 con 0x7f0eac0a5610 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac070d20 con 0x7f0eac073940 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f0eb3a8b640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1518826684 0 0) 0x7f5edc072d70 con 0x7f5edc13ca90 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient(hunting): my global_id is 4283 2024-08-19T09:50:19.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ecc003520 con 0x7f5edc13ca90 2024-08-19T09:50:19.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f0eb3a8b640 10 monclient(hunting): authenticate will time out at 675.378262s 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2868713428 0 0) 0x7fb3580a26d0 con 0x7fb358136410 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient(hunting): my global_id is 4292 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3580a4160 con 0x7fb358136410 2024-08-19T09:50:19.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f13150cc640 1 Processor -- start 2024-08-19T09:50:19.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3922520492 0 0) 0x7fb344003520 con 0x7fb35813c990 2024-08-19T09:50:19.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5ee0e88640 1 -- 172.21.15.1:0/222968891 >> v1:172.21.15.82:6789/0 conn(0x7f5edc0a4590 legacy=0x7f5edc0731c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 >> v1:172.21.15.1:6793/0 conn(0x7fb358136410 legacy=0x7fb35813b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f13150cc640 1 -- start start 2024-08-19T09:50:19.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f13150cc640 1 librados: setting wanted keys 2024-08-19T09:50:19.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 >> v1:172.21.15.1:6789/0 conn(0x7fb3580a1be0 legacy=0x7fb3580a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T09:50:19.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f13150cc640 1 librados: calling monclient init 2024-08-19T09:50:19.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f13150cc640 10 monclient: init 2024-08-19T09:50:19.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4212974212 0 0) 0x7f5ecc003520 con 0x7f5edc13ca90 2024-08-19T09:50:19.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 monclient: found mon.noname-b 2024-08-19T09:50:19.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 >> v1:172.21.15.82:6789/0 conn(0x7f5edc0a4590 legacy=0x7f5edc0731c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:19.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3580a5b10 con 0x7fb35813c990 2024-08-19T09:50:19.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 >> v1:172.21.15.1:6789/0 conn(0x7f5edc0738d0 legacy=0x7f5edc13b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 monclient: found mon.noname-e 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc071800 con 0x7f5edc13ca90 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ed4001e60 con 0x7f5edc13ca90 2024-08-19T09:50:19.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5ee30ec640 5 monclient: authenticate success, global_id 4283 2024-08-19T09:50:19.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5ee30ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb360694640 5 monclient: authenticate success, global_id 4268 2024-08-19T09:50:19.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb360694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7f5edb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb350001eb0 con 0x7fb35813c990 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.302+0000 7fb35d42e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 10 monclient: dump: 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.394 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 10 monclient: dump: 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb360694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb350003990 con 0x7fb35813c990 2024-08-19T09:50:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5ee30ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ed4004c20 con 0x7f5edc13ca90 2024-08-19T09:50:19.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb360694640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb360694640 4 set_mon_vals no callback set 2024-08-19T09:50:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 1 -- 172.21.15.1:0/222968891 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ed4005110 con 0x7f5edc13ca90 2024-08-19T09:50:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5ee30ec640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5ee30ec640 4 set_mon_vals no callback set 2024-08-19T09:50:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 1 -- 172.21.15.1:0/1166412165 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb350003f00 con 0x7fb35813c990 2024-08-19T09:50:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5edb7fe640 10 monclient: dump: 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb35d42e640 10 monclient: dump: 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.402 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f13150cc640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f13150cc640 10 monclient: _add_conns ranks=[3,1,2,4,5,7,8,0,6] 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f13150cc640 10 monclient(hunting): picked mon.noname-g con 0x7f13100694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f13150cc640 10 monclient(hunting): picked mon.noname-f con 0x7f13100a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f13150cc640 10 monclient(hunting): picked mon.noname-b con 0x7f13100a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f130e59c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f13100a46a0 0x7f13101d2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:45526/0 (socket says 172.21.15.1:45526) 2024-08-19T09:50:19.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f130e59c640 1 -- 172.21.15.1:0/3693368840 learned_addr learned my addr 172.21.15.1:0/3693368840 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13101388b0 con 0x7f13100a46a0 2024-08-19T09:50:19.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5ee30ec640 10 monclient: shutdown 2024-08-19T09:50:19.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5ee30ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7f5ee30ec640 1 -- 172.21.15.1:0/222968891 >> v1:172.21.15.1:6793/0 conn(0x7f5edc13ca90 legacy=0x7f5edc13ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.303+0000 7fb360694640 10 monclient: shutdown 2024-08-19T09:50:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7fb360694640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7fb360694640 1 -- 172.21.15.1:0/1166412165 >> v1:172.21.15.1:6790/0 conn(0x7fb35813c990 legacy=0x7fb35813ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f5ee30ec640 1 -- 172.21.15.1:0/222968891 shutdown_connections 2024-08-19T09:50:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f5ee30ec640 1 -- 172.21.15.1:0/222968891 >> 172.21.15.1:0/222968891 conn(0x7f5edc0a5f60 msgr2=0x7f5edc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1310142c50 con 0x7f13100a6830 2024-08-19T09:50:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f5ee30ec640 1 -- 172.21.15.1:0/222968891 shutdown_connections 2024-08-19T09:50:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7fb360694640 1 -- 172.21.15.1:0/1166412165 shutdown_connections 2024-08-19T09:50:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7fb360694640 1 -- 172.21.15.1:0/1166412165 >> 172.21.15.1:0/1166412165 conn(0x7fb3580a4d50 msgr2=0x7fb3580a5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb1827640 1 -- 172.21.15.1:0/2887425223 >> v1:172.21.15.1:6792/0 conn(0x7f0eac0a5610 legacy=0x7f0eac0a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb1026640 1 -- 172.21.15.1:0/2887425223 >> v1:172.21.15.82:6792/0 conn(0x7f0eac073940 legacy=0x7f0eac13c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f5ee30ec640 1 -- 172.21.15.1:0/222968891 wait complete. 2024-08-19T09:50:19.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13100a9fc0 con 0x7f13100694b0 2024-08-19T09:50:19.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f13150cc640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3161759999 0 0) 0x7f0eac074340 con 0x7f0eac0a5610 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient(hunting): my global_id is 4254 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e98003520 con 0x7f0eac0a5610 2024-08-19T09:50:19.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f13150cc640 10 monclient(hunting): authenticate will time out at 675.380595s 2024-08-19T09:50:19.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 996814492 0 0) 0x7f0e98003520 con 0x7f0eac0a5610 2024-08-19T09:50:19.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7fb360694640 1 -- 172.21.15.1:0/1166412165 shutdown_connections 2024-08-19T09:50:19.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7fb360694640 1 -- 172.21.15.1:0/1166412165 wait complete. 2024-08-19T09:50:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 >> v1:172.21.15.82:6792/0 conn(0x7f0eac073940 legacy=0x7f0eac13c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 >> v1:172.21.15.1:6789/0 conn(0x7f0eac0a6100 legacy=0x7f0eac13ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 monclient: found mon.noname-d 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac071620 con 0x7f0eac0a5610 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0ea8001ae0 con 0x7f0eac0a5610 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb0825640 10 monclient: dump: 2024-08-19T09:50:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.414 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb3a8b640 5 monclient: authenticate success, global_id 4254 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.304+0000 7f0eb3a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ea80048a0 con 0x7f0eac0a5610 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb0825640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb0825640 1 -- 172.21.15.1:0/2887425223 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0ea8004d90 con 0x7f0eac0a5610 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb0825640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb0825640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb0825640 10 monclient: dump: 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.416 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb3a8b640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb3a8b640 4 set_mon_vals no callback set 2024-08-19T09:50:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f5ee30ec640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 10 monclient: monmap: 2024-08-19T09:50:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f5ee30ec640 10 monclient: monmap: 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.420 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 1 librados: starting msgr at 2024-08-19T09:50:19.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 1 librados: starting objecter 2024-08-19T09:50:19.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 1 Processor -- start 2024-08-19T09:50:19.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f5ee30ec640 1 librados: starting msgr at 2024-08-19T09:50:19.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f5ee30ec640 1 librados: starting objecter 2024-08-19T09:50:19.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 1 -- start start 2024-08-19T09:50:19.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 1 librados: setting wanted keys 2024-08-19T09:50:19.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 1 librados: calling monclient init 2024-08-19T09:50:19.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7fb360694640 10 monclient: init 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb3a8b640 10 monclient: shutdown 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb3a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 >> v1:172.21.15.1:6792/0 conn(0x7f0eac0a5610 legacy=0x7f0eac0a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.305+0000 7f5ee30ec640 1 Processor -- start 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 client.?.objecter ms_handle_connect 0x7f13100694b0 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 client.?.objecter ms_handle_connect 0x7f13100a46a0 2024-08-19T09:50:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f130ed9d640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.82:6790/0 conn(0x7f13100694b0 legacy=0x7f1310141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f130e59c640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.82:6789/0 conn(0x7f13100a46a0 legacy=0x7f13101d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f130f59e640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.1:6790/0 conn(0x7f13100a6830 legacy=0x7f13101d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 client.?.objecter ms_handle_connect 0x7f13100a6830 2024-08-19T09:50:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 shutdown_connections 2024-08-19T09:50:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f5ee30ec640 1 -- start start 2024-08-19T09:50:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f5ee30ec640 1 librados: setting wanted keys 2024-08-19T09:50:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f5ee30ec640 1 librados: calling monclient init 2024-08-19T09:50:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f5ee30ec640 10 monclient: init 2024-08-19T09:50:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 >> 172.21.15.1:0/2887425223 conn(0x7f0eac0aa2f0 msgr2=0x7f0eac0a6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb360694640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb360694640 10 monclient: _add_conns ranks=[4,2,5,7,1,6,3,8,0] 2024-08-19T09:50:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb360694640 10 monclient(hunting): picked mon.noname-c con 0x7fb3580a1be0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 shutdown_connections 2024-08-19T09:50:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb35e430640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb3580a1be0 0x7fb35806fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57346/0 (socket says 172.21.15.1:57346) 2024-08-19T09:50:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb35dc2f640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb358136410 0x7fb3580701d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52746/0 (socket says 172.21.15.1:52746) 2024-08-19T09:50:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090168925 0 0) 0x7f13101388b0 con 0x7f13100a46a0 2024-08-19T09:50:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): my global_id is 4249 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12f8004ee0 con 0x7f13100a46a0 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609372550 0 0) 0x7f1310142c50 con 0x7f13100a6830 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): my global_id is 4277 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13101388b0 con 0x7f13100a6830 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270501739 0 0) 0x7f13100a9fc0 con 0x7f13100694b0 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 10 monclient(hunting): my global_id is 4305 2024-08-19T09:50:19.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1310142c50 con 0x7f13100694b0 2024-08-19T09:50:19.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/2887425223 wait complete. 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb35e430640 1 -- 172.21.15.1:0/4132106584 learned_addr learned my addr 172.21.15.1:0/4132106584 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.306+0000 7fb360694640 10 monclient(hunting): picked mon.noname-b con 0x7fb358136410 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7fb360694640 10 monclient(hunting): picked mon.noname-h con 0x7fb35813c990 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35809f650 con 0x7fb358136410 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3580a7df0 con 0x7fb3580a1be0 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3580708e0 con 0x7fb35813c990 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7fb360694640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2594931726 0 0) 0x7f13101388b0 con 0x7f13100a6830 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13100a9fc0 con 0x7f13100a6830 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1308005110 con 0x7f13100a6830 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): dump: 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1772366878 0 0) 0x7f12f8004ee0 con 0x7f13100a46a0 2024-08-19T09:50:19.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7fb360694640 10 monclient(hunting): authenticate will time out at 675.383384s 2024-08-19T09:50:19.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13101388b0 con 0x7f13100a46a0 2024-08-19T09:50:19.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2421827289 0 0) 0x7f1310142c50 con 0x7f13100694b0 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12f8004ee0 con 0x7f13100694b0 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f12fc001d00 con 0x7f13100a46a0 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): dump: 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13000015f0 con 0x7f13100694b0 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient(hunting): dump: 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.438 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f0eb3a8b640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3939352948 0 0) 0x7f13101388b0 con 0x7f13100a46a0 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.82:6790/0 conn(0x7f13100694b0 legacy=0x7f1310141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.1:6790/0 conn(0x7f13100a6830 legacy=0x7f13101d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 monclient: found mon.f 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:19.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1310136550 con 0x7f13100a46a0 2024-08-19T09:50:19.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient: _add_conns ranks=[8,0,4,5,6,7,2,3,1] 2024-08-19T09:50:19.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient(hunting): picked mon.noname-e con 0x7f5edc0738d0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient(hunting): picked mon.noname-a con 0x7f5edc0a4590 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient(hunting): picked mon.noname-c con 0x7f5edc13ca90 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5edc120cc0 con 0x7f5edc0a4590 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5edc121290 con 0x7f5edc13ca90 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5edc13b9d0 con 0x7f5edc0738d0 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee30ec640 10 monclient(hunting): authenticate will time out at 675.383804s 2024-08-19T09:50:19.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13150cc640 5 monclient: authenticate success, global_id 4249 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f0eb3a8b640 10 monclient: monmap: 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.442 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13150cc640 10 monclient: _renew_subs 2024-08-19T09:50:19.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13150cc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:19.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13101385d0 con 0x7f13100a46a0 2024-08-19T09:50:19.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee0e88640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f5edc0738d0 0x7f5edc13abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44654/0 (socket says 172.21.15.1:44654) 2024-08-19T09:50:19.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee1689640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f5edc13ca90 0x7f5edc1d4560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57350/0 (socket says 172.21.15.1:57350) 2024-08-19T09:50:19.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.307+0000 7f5ee1689640 1 -- 172.21.15.1:0/311491944 learned_addr learned my addr 172.21.15.1:0/311491944 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12fc004bc0 con 0x7f13100a46a0 2024-08-19T09:50:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13077fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f12fc0050b0 con 0x7f13100a46a0 2024-08-19T09:50:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13077fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13077fe640 10 monclient: dump: 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f130d59a640 4 set_mon_vals no callback set 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13150cc640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13150cc640 10 monclient: _renew_subs 2024-08-19T09:50:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 1 librados: starting msgr at 2024-08-19T09:50:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 1 librados: starting objecter 2024-08-19T09:50:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13150cc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1310142550 con 0x7f13100a46a0 2024-08-19T09:50:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f13150cc640 1 librados: init done 2024-08-19T09:50:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 1 Processor -- start 2024-08-19T09:50:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 1 -- start start 2024-08-19T09:50:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 1 librados: setting wanted keys 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 1 librados: calling monclient init 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.308+0000 7f0eb3a8b640 10 monclient: init 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient: _add_conns ranks=[2,4,6,7,0,3,1,8,5] 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient(hunting): picked mon.noname-b con 0x7f0eac073940 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb35e430640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.1:6791/0 conn(0x7fb3580a1be0 legacy=0x7fb35806fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb35dc2f640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.1:6790/0 conn(0x7fb358136410 legacy=0x7fb3580701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 client.?.objecter ms_handle_connect 0x7fb3580a1be0 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 client.?.objecter ms_handle_connect 0x7fb358136410 2024-08-19T09:50:19.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient(hunting): picked mon.noname-c con 0x7f0eac0a5610 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient(hunting): picked mon.noname-d con 0x7f0eac0a6100 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac0a9140 con 0x7f0eac073940 2024-08-19T09:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac0a36d0 con 0x7f0eac0a5610 2024-08-19T09:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac06e390 con 0x7f0eac0a6100 2024-08-19T09:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692182841 0 0) 0x7fb3580a7df0 con 0x7fb3580a1be0 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 10 monclient(hunting): my global_id is 4261 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb35ec31640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.82:6791/0 conn(0x7fb35813c990 legacy=0x7fb3581dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb3a8b640 10 monclient(hunting): authenticate will time out at 675.385418s 2024-08-19T09:50:19.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb33c004eb0 con 0x7fb3580a1be0 2024-08-19T09:50:19.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb1827640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0eac073940 0x7f0eac137a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52754/0 (socket says 172.21.15.1:52754) 2024-08-19T09:50:19.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb2028640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f0eac0a6100 0x7f0eac1d6900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52846/0 (socket says 172.21.15.1:52846) 2024-08-19T09:50:19.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f0eb1827640 1 -- 172.21.15.1:0/1920146929 learned_addr learned my addr 172.21.15.1:0/1920146929 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:50:19.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.309+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f12fc005250 con 0x7f13100a46a0 2024-08-19T09:50:19.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5edbfff640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.1:6789/0 conn(0x7f5edc0a4590 legacy=0x7f5edc13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 client.?.objecter ms_handle_connect 0x7f5edc0a4590 2024-08-19T09:50:19.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13150cc640 10 client.4249.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f12fc04d900 con 0x7f13100a46a0 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13150cc640 15 client.4249.objecter _session_command_op_assign -1 1 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13150cc640 20 client.4249.objecter _get_session osd=-1 returning homeless 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 10 client.4249.objecter ms_dispatch 0x7f1310000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13150cc640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13150cc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ee1689640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.1:6791/0 conn(0x7f5edc13ca90 legacy=0x7f5edc1d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.82:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1310051b00 con 0x7f13100a46a0 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 client.?.objecter ms_handle_connect 0x7f5edc13ca90 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 client.?.objecter ms_handle_connect 0x7f5edc0738d0 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ee0e88640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.1:6793/0 conn(0x7f5edc0738d0 legacy=0x7f5edc13abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 3 client.4249.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 3 client.4249.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2960314638 0 0) 0x7f5edc120cc0 con 0x7f5edc0a4590 2024-08-19T09:50:19.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): my global_id is 4320 2024-08-19T09:50:19.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ec0004eb0 con 0x7f5edc0a4590 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 10 client.4249.objecter checking command 1 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 20 client.4249.objecter _get_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 15 client.4249.objecter _session_command_op_remove -1 1 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 20 client.4249.objecter _get_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 20 client.4249.objecter get_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 15 client.4249.objecter _session_command_op_assign 0 1 2024-08-19T09:50:19.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 4 2024-08-19T09:50:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518687902 0 0) 0x7f5edc13b9d0 con 0x7f5edc0738d0 2024-08-19T09:50:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): my global_id is 4301 2024-08-19T09:50:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5edc120cc0 con 0x7f5edc0738d0 2024-08-19T09:50:19.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 10 client.4249.objecter _send_command 1 2024-08-19T09:50:19.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.1:6804/3862255507 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12f804c320 con 0x7f12f8048670 2024-08-19T09:50:19.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 20 client.4249.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f130ed9d640 10 monclient: get_auth_request con 0x7f12f8041a90 auth_method 0 2024-08-19T09:50:19.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f130f59e640 10 monclient: get_auth_request con 0x7f12f8048670 auth_method 0 2024-08-19T09:50:19.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3233065331 0 0) 0x7f5edc071800 con 0x7f5edc13ca90 2024-08-19T09:50:19.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f12fc005540 con 0x7f13100a46a0 2024-08-19T09:50:19.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f13077fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T09:50:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): my global_id is 4270 2024-08-19T09:50:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5edc13b9d0 con 0x7f5edc13ca90 2024-08-19T09:50:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 752872113 0 0) 0x7f5ec0004eb0 con 0x7f5edc0a4590 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5edc071800 con 0x7f5edc0a4590 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ec8001b90 con 0x7f5edc0a4590 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): dump: 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.310+0000 7f5ed97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1700774026 0 0) 0x7f5edc071800 con 0x7f5edc0a4590 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.1:6793/0 conn(0x7f5edc0738d0 legacy=0x7f5edc13abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.1:6791/0 conn(0x7f5edc13ca90 legacy=0x7f5edc1d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 monclient: found mon.a 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc0a2330 con 0x7f5edc0a4590 2024-08-19T09:50:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 5 monclient: authenticate success, global_id 4320 2024-08-19T09:50:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 10 monclient: _renew_subs 2024-08-19T09:50:19.467 INFO:teuthology.orchestra.run.smithi001.stdout:51539607556 2024-08-19T09:50:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc121b30 con 0x7f5edc0a4590 2024-08-19T09:50:19.468 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 client.?.objecter ms_handle_connect 0x7fb35813c990 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479866133 0 0) 0x7fb3580a4160 con 0x7fb358136410 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): my global_id is 4286 2024-08-19T09:50:19.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3580a7df0 con 0x7fb358136410 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404694768 0 0) 0x7fb3580708e0 con 0x7fb35813c990 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): my global_id is 4325 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3580a4160 con 0x7fb35813c990 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3489862036 0 0) 0x7fb33c004eb0 con 0x7fb3580a1be0 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 10 monclient: _renew_subs 2024-08-19T09:50:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5edc120720 con 0x7f5edc0a4590 2024-08-19T09:50:19.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ee30ec640 1 librados: init done 2024-08-19T09:50:19.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3580708e0 con 0x7fb3580a1be0 2024-08-19T09:50:19.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb354000ed0 con 0x7fb3580a1be0 2024-08-19T09:50:19.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec8004a90 con 0x7f5edc0a4590 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): dump: 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.473 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.474 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.475 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.475 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ec8005000 con 0x7f5edc0a4590 2024-08-19T09:50:19.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5eda7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:19.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7fb34effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f5ed97fa640 10 monclient: dump: 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.476 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 577242446 0 0) 0x7fb3580a7df0 con 0x7fb358136410 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.311+0000 7f0eb2028640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6792/0 conn(0x7f0eac0a6100 legacy=0x7f0eac1d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 client.?.objecter ms_handle_connect 0x7f0eac0a6100 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018263780 0 0) 0x7f0eac06e390 con 0x7f0eac0a6100 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): my global_id is 4263 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e90004eb0 con 0x7f0eac0a6100 2024-08-19T09:50:19.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0eb1827640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6790/0 conn(0x7f0eac073940 legacy=0x7f0eac137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 client.?.objecter ms_handle_connect 0x7f0eac073940 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb33c004eb0 con 0x7fb358136410 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3480047e0 con 0x7fb358136410 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient(hunting): dump: 2024-08-19T09:50:19.479 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3606253377 0 0) 0x7f0e90004eb0 con 0x7f0eac0a6100 2024-08-19T09:50:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0eb1026640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6791/0 conn(0x7f0eac0a5610 legacy=0x7f0eac1d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0eac06e390 con 0x7f0eac0a6100 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 client.?.objecter ms_handle_connect 0x7f0eac0a5610 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0ea4004670 con 0x7f0eac0a6100 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3819987601 0 0) 0x7fb3580a4160 con 0x7fb35813c990 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3580a7df0 con 0x7fb35813c990 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 930067078 0 0) 0x7fb3580708e0 con 0x7fb3580a1be0 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.82:6791/0 conn(0x7fb35813c990 legacy=0x7fb3581dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.1:6790/0 conn(0x7fb358136410 legacy=0x7fb3580701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 monclient: found mon.c 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3580712b0 con 0x7fb3580a1be0 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7fb34effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f13077fe640 10 client.4249.objecter ms_handle_connect 0x7f12f8041a90 2024-08-19T09:50:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): dump: 2024-08-19T09:50:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1452124279 0 0) 0x7f0eac0a36d0 con 0x7f0eac0a5610 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient(hunting): my global_id is 4279 2024-08-19T09:50:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e90004eb0 con 0x7f0eac0a5610 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3535322026 0 0) 0x7f0eac06e390 con 0x7f0eac0a6100 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6791/0 conn(0x7f0eac0a5610 legacy=0x7f0eac1d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6790/0 conn(0x7f0eac073940 legacy=0x7f0eac137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 monclient: found mon.d 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.312+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac1273f0 con 0x7f0eac0a6100 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 5 monclient: authenticate success, global_id 4263 2024-08-19T09:50:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 10 monclient: _renew_subs 2024-08-19T09:50:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac126b50 con 0x7f0eac0a6100 2024-08-19T09:50:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 5 monclient: authenticate success, global_id 4261 2024-08-19T09:50:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 10 monclient: _renew_subs 2024-08-19T09:50:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f0eac125bf0 con 0x7f0eac0a6100 2024-08-19T09:50:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0eb3a8b640 1 librados: init done 2024-08-19T09:50:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ea4001500 con 0x7f0eac0a6100 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0ea40019f0 con 0x7f0eac0a6100 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea27fc640 10 monclient: dump: 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.491 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ee30ec640 10 client.4320.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ee30ec640 15 client.4320.objecter _session_command_op_assign -1 1 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ee30ec640 20 client.4320.objecter _get_session osd=-1 returning homeless 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ee30ec640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ee30ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.1:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5ebc002910 con 0x7f5edc0a4590 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5ec80051a0 con 0x7f5edc0a4590 2024-08-19T09:50:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f5ec804d950 con 0x7f5edc0a4590 2024-08-19T09:50:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f5ed97fa640 10 client.4320.objecter ms_dispatch 0x7f5edc000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7f0ea3fff640 4 set_mon_vals no callback set 2024-08-19T09:50:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0eb3a8b640 10 client.4263.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 10 monclient: _renew_subs 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb358073310 con 0x7fb3580a1be0 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 10 monclient: _renew_subs 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fb358072540 con 0x7fb3580a1be0 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb360694640 1 librados: init done 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb354001450 con 0x7fb3580a1be0 2024-08-19T09:50:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb34effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb354001980 con 0x7fb3580a1be0 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb34effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb34effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb34effd640 10 monclient: dump: 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.313+0000 7fb35cc2d640 4 set_mon_vals no callback set 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb360694640 10 client.4261.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb360694640 15 client.4261.objecter _session_command_op_assign -1 1 2024-08-19T09:50:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb360694640 20 client.4261.objecter _get_session osd=-1 returning homeless 2024-08-19T09:50:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb360694640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb360694640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.1:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fb340002910 con 0x7fb3580a1be0 2024-08-19T09:50:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ee0e88640 10 monclient: get_auth_request con 0x7f5ec0041220 auth_method 0 2024-08-19T09:50:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 3 client.4320.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 3 client.4320.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 10 client.4320.objecter checking command 1 2024-08-19T09:50:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb354001b20 con 0x7fb3580a1be0 2024-08-19T09:50:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 20 client.4320.objecter _get_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0ea4001c50 con 0x7f0eac0a6100 2024-08-19T09:50:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0eb3a8b640 15 client.4263.objecter _session_command_op_assign -1 1 2024-08-19T09:50:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0eb3a8b640 20 client.4263.objecter _get_session osd=-1 returning homeless 2024-08-19T09:50:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0eb3a8b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0eb3a8b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 15 client.4320.objecter _session_command_op_remove -1 1 2024-08-19T09:50:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0e94002910 con 0x7f0eac0a6100 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 20 client.4320.objecter _get_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 20 client.4320.objecter get_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 15 client.4320.objecter _session_command_op_assign 5 1 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 4 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 10 client.4320.objecter _send_command 1 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.82:6808/1813220032 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ec004ba60 con 0x7f5ec0047dd0 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 20 client.4320.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f5ec8016b20 con 0x7f5edc0a4590 2024-08-19T09:50:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ed97fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T09:50:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.314+0000 7f5ee1689640 10 monclient: get_auth_request con 0x7f5ec0047dd0 auth_method 0 2024-08-19T09:50:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fb35404d980 con 0x7fb3580a1be0 2024-08-19T09:50:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 10 client.4261.objecter ms_dispatch 0x7fb358000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 3 client.4261.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 3 client.4261.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f13077fe640 10 client.4249.objecter ms_handle_connect 0x7f12f8048670 2024-08-19T09:50:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f0ea404dd60 con 0x7f0eac0a6100 2024-08-19T09:50:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 10 client.4263.objecter ms_dispatch 0x7f0eac000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 3 client.4263.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 3 client.4263.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 10 client.4261.objecter checking command 1 2024-08-19T09:50:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 10 client.4263.objecter checking command 1 2024-08-19T09:50:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 20 client.4261.objecter _get_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 20 client.4263.objecter _get_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 15 client.4263.objecter _session_command_op_remove -1 1 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 20 client.4263.objecter _get_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 20 client.4263.objecter get_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 15 client.4263.objecter _session_command_op_assign 1 1 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 4 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 10 client.4263.objecter _send_command 1 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6800/1668590231 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e90053fd0 con 0x7f0e90050320 2024-08-19T09:50:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 15 client.4261.objecter _session_command_op_remove -1 1 2024-08-19T09:50:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 20 client.4263.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f0ea404e220 con 0x7f0eac0a6100 2024-08-19T09:50:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0ea27fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 20 client.4261.objecter _get_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 20 client.4261.objecter get_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 15 client.4261.objecter _session_command_op_assign 3 1 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 4 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 10 client.4261.objecter _send_command 1 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.82:6804/394826682 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb33c04c1e0 con 0x7fb33c048530 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 20 client.4261.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fb35404dc80 con 0x7fb3580a1be0 2024-08-19T09:50:19.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb34effd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T09:50:19.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f0eb1026640 10 monclient: get_auth_request con 0x7f0e90050320 auth_method 0 2024-08-19T09:50:19.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7fb35ec31640 10 monclient: get_auth_request con 0x7fb33c048530 auth_method 0 2024-08-19T09:50:19.509 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-08-19T09:50:19.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-19T09:50:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== osd.0 v1:172.21.15.1:6804/3862255507 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f12f804c320 con 0x7f12f8048670 2024-08-19T09:50:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f13077fe640 10 client.4249.objecter ms_dispatch 0x7f1310000b90 command_reply(tid 1: 0 ) 2024-08-19T09:50:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f13077fe640 10 client.4249.objecter _finish_command 1 = system:0 2024-08-19T09:50:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f13077fe640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 4 2024-08-19T09:50:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.315+0000 7f13077fe640 15 client.4249.objecter _session_command_op_remove 0 1 2024-08-19T09:50:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.316+0000 7f5ed97fa640 10 client.4320.objecter ms_handle_connect 0x7f5ec0041220 2024-08-19T09:50:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.316+0000 7fb35dc2f640 10 monclient: get_auth_request con 0x7fb33c041920 auth_method 0 2024-08-19T09:50:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.316+0000 7f0eb1827640 10 monclient: get_auth_request con 0x7f0e90049810 auth_method 0 2024-08-19T09:50:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.318+0000 7f0ea27fc640 10 client.4263.objecter ms_handle_connect 0x7f0e90049810 2024-08-19T09:50:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.318+0000 7fb34effd640 10 client.4261.objecter ms_handle_connect 0x7fb33c041920 2024-08-19T09:50:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.319+0000 7f5ed97fa640 10 client.4320.objecter ms_handle_connect 0x7f5ec0047dd0 2024-08-19T09:50:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.319+0000 7f0ea27fc640 10 client.4263.objecter ms_handle_connect 0x7f0e90050320 2024-08-19T09:50:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.319+0000 7fb34effd640 10 client.4261.objecter ms_handle_connect 0x7fb33c048530 2024-08-19T09:50:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== osd.5 v1:172.21.15.82:6808/1813220032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5ec004ba60 con 0x7f5ec0047dd0 2024-08-19T09:50:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f5ed97fa640 10 client.4320.objecter ms_dispatch 0x7f5edc000b90 command_reply(tid 1: 0 ) 2024-08-19T09:50:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f5ed97fa640 10 client.4320.objecter _finish_command 1 = system:0 2024-08-19T09:50:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f5ed97fa640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 4 2024-08-19T09:50:19.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f5ed97fa640 15 client.4320.objecter _session_command_op_remove 5 1 2024-08-19T09:50:19.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== osd.1 v1:172.21.15.1:6800/1668590231 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e90053fd0 con 0x7f0e90050320 2024-08-19T09:50:19.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f0ea27fc640 10 client.4263.objecter ms_dispatch 0x7f0eac000b90 command_reply(tid 1: 0 ) 2024-08-19T09:50:19.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f0ea27fc640 10 client.4263.objecter _finish_command 1 = system:0 2024-08-19T09:50:19.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== osd.3 v1:172.21.15.82:6804/394826682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb33c04c1e0 con 0x7fb33c048530 2024-08-19T09:50:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f0ea27fc640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 4 2024-08-19T09:50:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7f0ea27fc640 15 client.4263.objecter _session_command_op_remove 1 1 2024-08-19T09:50:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7fb34effd640 10 client.4261.objecter ms_dispatch 0x7fb358000b90 command_reply(tid 1: 0 ) 2024-08-19T09:50:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7fb34effd640 10 client.4261.objecter _finish_command 1 = system:0 2024-08-19T09:50:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7fb34effd640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 4 2024-08-19T09:50:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.320+0000 7fb34effd640 15 client.4261.objecter _session_command_op_remove 3 1 2024-08-19T09:50:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 10 client.4249.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T09:50:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 15 client.4249.objecter _session_command_op_assign -1 2 2024-08-19T09:50:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 20 client.4249.objecter _get_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 20 client.4249.objecter _get_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 15 client.4249.objecter _session_command_op_remove -1 2 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 20 client.4249.objecter get_session s=0x7f12f8047fc0 osd=0 3 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 15 client.4249.objecter _session_command_op_assign 0 2 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 4 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 10 client.4249.objecter _send_command 2 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.341+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 --> v1:172.21.15.1:6804/3862255507 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13101cd9f0 con 0x7f12f8048670 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13077fe640 1 -- 172.21.15.1:0/3693368840 <== osd.0 v1:172.21.15.1:6804/3862255507 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f13101cd9f0 con 0x7f12f8048670 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13077fe640 10 client.4249.objecter ms_dispatch 0x7f1310000b90 command_reply(tid 2: 0 ) 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13077fe640 10 client.4249.objecter _finish_command 2 = system:0 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13077fe640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 4 2024-08-19T09:50:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13077fe640 15 client.4249.objecter _session_command_op_remove 0 2 2024-08-19T09:50:19.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 10 librados: watch_flush enter 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 10 librados: watch_flush exit 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 10 client.4249.objecter close_session for osd.0 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.1:6804/3862255507 conn(0x7f12f8048670 legacy=0x7f12f804aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 20 client.4249.objecter put_session s=0x7f12f8047fc0 osd=0 1 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 20 client.4249.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 10 client.4249.objecter successfully canceled tick 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.82:6812/2549104415 conn(0x7f12f8041a90 legacy=0x7f12f8043f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 10 monclient: shutdown 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 >> v1:172.21.15.82:6789/0 conn(0x7f13100a46a0 legacy=0x7f13101d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f130f59e640 1 -- 172.21.15.1:0/3693368840 reap_dead start 2024-08-19T09:50:19.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 shutdown_connections 2024-08-19T09:50:19.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.342+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 >> 172.21.15.1:0/3693368840 conn(0x7f13100a9020 msgr2=0x7f13100a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.343+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 shutdown_connections 2024-08-19T09:50:19.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.343+0000 7f13150cc640 1 -- 172.21.15.1:0/3693368840 wait complete. 2024-08-19T09:50:19.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.343+0000 7f13150cc640 1 librados: shutdown 2024-08-19T09:50:19.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 10 client.4320.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 15 client.4320.objecter _session_command_op_assign -1 2 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 20 client.4320.objecter _get_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 20 client.4320.objecter _get_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 15 client.4320.objecter _session_command_op_remove -1 2 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 20 client.4320.objecter get_session s=0x7f5ec0047720 osd=5 3 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 15 client.4320.objecter _session_command_op_assign 5 2 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 4 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 10 client.4320.objecter _send_command 2 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.349+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 --> v1:172.21.15.82:6808/1813220032 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5ebc001220 con 0x7f5ec0047dd0 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.350+0000 7f5ed97fa640 1 -- 172.21.15.1:0/311491944 <== osd.5 v1:172.21.15.82:6808/1813220032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7f5ebc001220 con 0x7f5ec0047dd0 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.350+0000 7f5ed97fa640 10 client.4320.objecter ms_dispatch 0x7f5edc000b90 command_reply(tid 2: 0 ) 2024-08-19T09:50:19.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.350+0000 7f5ed97fa640 10 client.4320.objecter _finish_command 2 = system:0 2024-08-19T09:50:19.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.350+0000 7f5ed97fa640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 4 2024-08-19T09:50:19.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.350+0000 7f5ed97fa640 15 client.4320.objecter _session_command_op_remove 5 2 2024-08-19T09:50:19.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.350+0000 7f5ee30ec640 10 librados: watch_flush enter 2024-08-19T09:50:19.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 10 librados: watch_flush exit 2024-08-19T09:50:19.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 10 client.4320.objecter close_session for osd.5 2024-08-19T09:50:19.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.82:6808/1813220032 conn(0x7f5ec0047dd0 legacy=0x7f5ec004a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 20 client.4320.objecter put_session s=0x7f5ec0047720 osd=5 1 2024-08-19T09:50:19.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 20 client.4320.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 10 client.4320.objecter successfully canceled tick 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5ec0041220 legacy=0x7f5ec00436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 10 monclient: shutdown 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 >> v1:172.21.15.1:6789/0 conn(0x7f5edc0a4590 legacy=0x7f5edc13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee1689640 1 -- 172.21.15.1:0/311491944 reap_dead start 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 shutdown_connections 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 >> 172.21.15.1:0/311491944 conn(0x7f5edc0a5f60 msgr2=0x7f5edc06c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 shutdown_connections 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 -- 172.21.15.1:0/311491944 wait complete. 2024-08-19T09:50:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.351+0000 7f5ee30ec640 1 librados: shutdown 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 10 client.4263.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 15 client.4263.objecter _session_command_op_assign -1 2 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 20 client.4263.objecter _get_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 20 client.4263.objecter _get_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 15 client.4263.objecter _session_command_op_remove -1 2 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 20 client.4263.objecter get_session s=0x7f0e9004fc70 osd=1 3 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 15 client.4263.objecter _session_command_op_assign 1 2 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 4 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 10 client.4263.objecter _send_command 2 2024-08-19T09:50:19.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 --> v1:172.21.15.1:6800/1668590231 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e94001220 con 0x7f0e90050320 2024-08-19T09:50:19.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 10 client.4261.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T09:50:19.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 15 client.4261.objecter _session_command_op_assign -1 2 2024-08-19T09:50:19.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 20 client.4261.objecter _get_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 20 client.4261.objecter _get_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 15 client.4261.objecter _session_command_op_remove -1 2 2024-08-19T09:50:19.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 20 client.4261.objecter get_session s=0x7fb33c047e80 osd=3 3 2024-08-19T09:50:19.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 15 client.4261.objecter _session_command_op_assign 3 2 2024-08-19T09:50:19.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 4 2024-08-19T09:50:19.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 10 client.4261.objecter _send_command 2 2024-08-19T09:50:19.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 --> v1:172.21.15.82:6804/394826682 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb340001150 con 0x7fb33c048530 2024-08-19T09:50:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0ea27fc640 1 -- 172.21.15.1:0/1920146929 <== osd.1 v1:172.21.15.1:6800/1668590231 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7f0e94001220 con 0x7f0e90050320 2024-08-19T09:50:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0ea27fc640 10 client.4263.objecter ms_dispatch 0x7f0eac000b90 command_reply(tid 2: 0 ) 2024-08-19T09:50:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0ea27fc640 10 client.4263.objecter _finish_command 2 = system:0 2024-08-19T09:50:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0ea27fc640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 4 2024-08-19T09:50:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.352+0000 7f0ea27fc640 15 client.4263.objecter _session_command_op_remove 1 2 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 10 librados: watch_flush enter 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 10 librados: watch_flush exit 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 10 client.4263.objecter close_session for osd.1 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6800/1668590231 conn(0x7f0e90050320 legacy=0x7f0e90052760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 20 client.4263.objecter put_session s=0x7f0e9004fc70 osd=1 1 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 20 client.4263.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 10 client.4263.objecter successfully canceled tick 2024-08-19T09:50:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb34effd640 1 -- 172.21.15.1:0/4132106584 <== osd.3 v1:172.21.15.82:6804/394826682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 258058390) 0x7fb340001150 con 0x7fb33c048530 2024-08-19T09:50:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb34effd640 10 client.4261.objecter ms_dispatch 0x7fb358000b90 command_reply(tid 2: 0 ) 2024-08-19T09:50:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb34effd640 10 client.4261.objecter _finish_command 2 = system:0 2024-08-19T09:50:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb34effd640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 4 2024-08-19T09:50:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb34effd640 15 client.4261.objecter _session_command_op_remove 3 2 2024-08-19T09:50:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0e90049810 legacy=0x7f0e9004bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 10 monclient: shutdown 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 >> v1:172.21.15.1:6792/0 conn(0x7f0eac0a6100 legacy=0x7f0eac1d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb2028640 1 -- 172.21.15.1:0/1920146929 reap_dead start 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 shutdown_connections 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 >> 172.21.15.1:0/1920146929 conn(0x7f0eac0aa2f0 msgr2=0x7f0eac0a7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 shutdown_connections 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 -- 172.21.15.1:0/1920146929 wait complete. 2024-08-19T09:50:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7f0eb3a8b640 1 librados: shutdown 2024-08-19T09:50:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 10 librados: watch_flush enter 2024-08-19T09:50:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 10 librados: watch_flush exit 2024-08-19T09:50:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 10 client.4261.objecter close_session for osd.3 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.82:6804/394826682 conn(0x7fb33c048530 legacy=0x7fb33c04a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 20 client.4261.objecter put_session s=0x7fb33c047e80 osd=3 1 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 20 client.4261.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 10 client.4261.objecter successfully canceled tick 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb33c041920 legacy=0x7fb33c043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 10 monclient: shutdown 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 >> v1:172.21.15.1:6791/0 conn(0x7fb3580a1be0 legacy=0x7fb35806fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb35ec31640 1 -- 172.21.15.1:0/4132106584 reap_dead start 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 shutdown_connections 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.353+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 >> 172.21.15.1:0/4132106584 conn(0x7fb3580a4d50 msgr2=0x7fb3580a6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.354+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 shutdown_connections 2024-08-19T09:50:19.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.354+0000 7fb360694640 1 -- 172.21.15.1:0/4132106584 wait complete. 2024-08-19T09:50:19.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.354+0000 7fb360694640 1 librados: shutdown 2024-08-19T09:50:19.535 INFO:teuthology.orchestra.run.smithi001.stdout:51539607556 2024-08-19T09:50:19.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-19T09:50:19.536 INFO:teuthology.orchestra.run.smithi001.stdout:51539607557 2024-08-19T09:50:19.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-19T09:50:19.537 INFO:teuthology.orchestra.run.smithi001.stdout:47244640261 2024-08-19T09:50:19.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-19T09:50:19.539 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-08-19T09:50:19.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.657+0000 7f84920de640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.657+0000 7f84920de640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.657+0000 7f84920de640 10 monclient: monmap: 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.658787+0000 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.658787+0000 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.660 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.658+0000 7f84920de640 1 Processor -- start 2024-08-19T09:50:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.658+0000 7f84920de640 1 -- start start 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.658+0000 7f84920de640 10 monclient: init 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient: _add_conns ranks=[7,1,5,3,8,2,4,0,6] 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient(hunting): picked mon.noname-i con 0x7f848c0a4f80 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient(hunting): picked mon.noname-f con 0x7f848c0a4650 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient(hunting): picked mon.noname-h con 0x7f848c13fcd0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a4a60 con 0x7f848c0a4650 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c069270 con 0x7f848c13fcd0 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a3c70 con 0x7f848c0a4f80 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f84920de640 10 monclient(hunting): authenticate will time out at 675.735528s 2024-08-19T09:50:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f848b7fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f848c0a4f80 0x7f848c0a5360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:58482/0 (socket says 172.21.15.1:58482) 2024-08-19T09:50:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.659+0000 7f848b7fe640 1 -- 172.21.15.1:0/1050878582 learned_addr learned my addr 172.21.15.1:0/1050878582 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.661+0000 7fa556da0640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.661+0000 7fa556da0640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.661+0000 7fa556da0640 10 monclient: monmap: 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.662743+0000 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.662743+0000 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.665 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848bfff640 1 -- 172.21.15.1:0/1050878582 >> v1:172.21.15.82:6791/0 conn(0x7f848c13fcd0 legacy=0x7f848c1420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848b7fe640 1 -- 172.21.15.1:0/1050878582 >> v1:172.21.15.82:6792/0 conn(0x7f848c0a4f80 legacy=0x7f848c0a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848affd640 1 -- 172.21.15.1:0/1050878582 >> v1:172.21.15.82:6789/0 conn(0x7f848c0a4650 legacy=0x7f848c13e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517844857 0 0) 0x7f848c0a3c70 con 0x7f848c0a4f80 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): my global_id is 4291 2024-08-19T09:50:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8474003520 con 0x7f848c0a4f80 2024-08-19T09:50:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7fa556da0640 1 Processor -- start 2024-08-19T09:50:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3813957277 0 0) 0x7f848c0a4a60 con 0x7f848c0a4650 2024-08-19T09:50:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): my global_id is 4258 2024-08-19T09:50:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7fa556da0640 1 -- start start 2024-08-19T09:50:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7fa556da0640 10 monclient: init 2024-08-19T09:50:19.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f848c0a3c70 con 0x7f848c0a4650 2024-08-19T09:50:19.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310817048 0 0) 0x7f848c069270 con 0x7f848c13fcd0 2024-08-19T09:50:19.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient(hunting): my global_id is 4334 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f848c0a4a60 con 0x7f848c13fcd0 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 157592204 0 0) 0x7f848c0a3c70 con 0x7f848c0a4650 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 >> v1:172.21.15.82:6792/0 conn(0x7f848c0a4f80 legacy=0x7f848c0a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 >> v1:172.21.15.82:6791/0 conn(0x7f848c13fcd0 legacy=0x7f848c1420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 monclient: found mon.noname-f 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:50:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c0a3420 con 0x7f848c0a4650 2024-08-19T09:50:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8480001a40 con 0x7f848c0a4650 2024-08-19T09:50:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7fa556da0640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:50:19.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7fa556da0640 10 monclient: _add_conns ranks=[7,3,6,1,4,0,5,2,8] 2024-08-19T09:50:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7fa556da0640 10 monclient(hunting): picked mon.noname-i con 0x7fa55006cdb0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7fa556da0640 10 monclient(hunting): picked mon.noname-g con 0x7fa550135d50 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7fa556da0640 10 monclient(hunting): picked mon.noname-d con 0x7fa55013f860 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: dump: 2024-08-19T09:50:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.676 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f84920de640 5 monclient: authenticate success, global_id 4258 2024-08-19T09:50:19.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f84920de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa555d9e640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7fa55006cdb0 0x7fa5500a57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:58486/0 (socket says 172.21.15.1:58486) 2024-08-19T09:50:19.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa555d9e640 1 -- 172.21.15.1:0/1978406005 learned_addr learned my addr 172.21.15.1:0/1978406005 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8480001d10 con 0x7f848c0a4650 2024-08-19T09:50:19.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5500a5ee0 con 0x7fa550135d50 2024-08-19T09:50:19.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 1 -- 172.21.15.1:0/1050878582 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8480003950 con 0x7f848c0a4650 2024-08-19T09:50:19.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5500a7bf0 con 0x7fa55013f860 2024-08-19T09:50:19.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55006dbe0 con 0x7fa55006cdb0 2024-08-19T09:50:19.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa556da0640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7fa556da0640 10 monclient(hunting): authenticate will time out at 675.740417s 2024-08-19T09:50:19.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f84920de640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f84920de640 4 set_mon_vals no callback set 2024-08-19T09:50:19.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:19.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.662+0000 7f848a7fc640 10 monclient: dump: 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.682 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7f84920de640 10 monclient: shutdown 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7f84920de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.663+0000 7f84920de640 1 -- 172.21.15.1:0/1050878582 >> v1:172.21.15.82:6789/0 conn(0x7f848c0a4650 legacy=0x7f848c13e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7f84920de640 1 -- 172.21.15.1:0/1050878582 shutdown_connections 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7f84920de640 1 -- 172.21.15.1:0/1050878582 >> 172.21.15.1:0/1050878582 conn(0x7f848c0a8ef0 msgr2=0x7f848c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7f84920de640 1 -- 172.21.15.1:0/1050878582 shutdown_connections 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.664+0000 7f84920de640 1 -- 172.21.15.1:0/1050878582 wait complete. 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 10 monclient: monmap: 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.683 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 1 librados: starting msgr at 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 1 librados: starting objecter 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 1 Processor -- start 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 1 -- start start 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 1 librados: setting wanted keys 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 1 librados: calling monclient init 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.665+0000 7f84920de640 10 monclient: init 2024-08-19T09:50:19.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient: _add_conns ranks=[7,8,1,2,6,5,4,0,3] 2024-08-19T09:50:19.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient(hunting): picked mon.noname-i con 0x7f848c0a4650 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient(hunting): picked mon.noname-e con 0x7f848c0a4f80 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient(hunting): picked mon.noname-f con 0x7f848c13fcd0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a2460 con 0x7f848c13fcd0 2024-08-19T09:50:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa555d9e640 1 -- 172.21.15.1:0/1978406005 >> v1:172.21.15.82:6792/0 conn(0x7fa55006cdb0 legacy=0x7fa5500a57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c13e9d0 con 0x7f848c0a4650 2024-08-19T09:50:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a2a30 con 0x7f848c0a4f80 2024-08-19T09:50:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa55659f640 1 -- 172.21.15.1:0/1978406005 >> v1:172.21.15.1:6792/0 conn(0x7fa55013f860 legacy=0x7fa550141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f84920de640 10 monclient(hunting): authenticate will time out at 675.742218s 2024-08-19T09:50:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f848affd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f848c0a4f80 0x7f848c13e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44660/0 (socket says 172.21.15.1:44660) 2024-08-19T09:50:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7f848affd640 1 -- 172.21.15.1:0/13610143 learned_addr learned my addr 172.21.15.1:0/13610143 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853342813 0 0) 0x7fa55006dbe0 con 0x7fa55006cdb0 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa55559d640 1 -- 172.21.15.1:0/1978406005 >> v1:172.21.15.82:6790/0 conn(0x7fa550135d50 legacy=0x7fa55013e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): my global_id is 4300 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa53c003520 con 0x7fa55006cdb0 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 366292808 0 0) 0x7fa5500a7bf0 con 0x7fa55013f860 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): my global_id is 4272 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa55006dbe0 con 0x7fa55013f860 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654984197 0 0) 0x7fa5500a5ee0 con 0x7fa550135d50 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient(hunting): my global_id is 4314 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5500a7bf0 con 0x7fa550135d50 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773775277 0 0) 0x7fa55006dbe0 con 0x7fa55013f860 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 >> v1:172.21.15.82:6792/0 conn(0x7fa55006cdb0 legacy=0x7fa5500a57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 >> v1:172.21.15.82:6790/0 conn(0x7fa550135d50 legacy=0x7fa55013e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.666+0000 7fa554d9c640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 1 monclient: found mon.noname-d 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55009dca0 con 0x7fa55013f860 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 5 monclient: authenticate success, global_id 4272 2024-08-19T09:50:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa548001e90 con 0x7fa55013f860 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: dump: 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa548003990 con 0x7fa55013f860 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 1 -- 172.21.15.1:0/1978406005 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa548003e80 con 0x7fa55013f860 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 4 set_mon_vals no callback set 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa554d9c640 10 monclient: dump: 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.697 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 10 monclient: shutdown 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.667+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 >> v1:172.21.15.1:6792/0 conn(0x7fa55013f860 legacy=0x7fa550141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 shutdown_connections 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 >> 172.21.15.1:0/1978406005 conn(0x7fa5500abd80 msgr2=0x7fa5500ac190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 shutdown_connections 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7fa556da0640 1 -- 172.21.15.1:0/1978406005 wait complete. 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f848affd640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.1:6793/0 conn(0x7f848c0a4f80 legacy=0x7f848c13e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f8473fff640 10 client.?.objecter ms_handle_connect 0x7f848c0a4f80 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f8473fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f8473fff640 10 client.?.objecter ms_handle_connect 0x7f848c13fcd0 2024-08-19T09:50:19.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f8473fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f848bfff640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.82:6789/0 conn(0x7f848c13fcd0 legacy=0x7f848c1dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f848b7fe640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.82:6792/0 conn(0x7f848c0a4650 legacy=0x7f848c13dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:50:19.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7fa556da0640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.668+0000 7fa556da0640 10 monclient: monmap: 2024-08-19T09:50:19.701 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.701 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.701 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.701 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 1 librados: starting msgr at 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 1 librados: starting objecter 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 1 Processor -- start 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 1 -- start start 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 1 librados: setting wanted keys 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 1 librados: calling monclient init 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 10 monclient: init 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 10 monclient: _add_conns ranks=[2,4,7,5,3,8,0,1,6] 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 10 monclient(hunting): picked mon.noname-b con 0x7fa55006cdb0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.669+0000 7fa556da0640 10 monclient(hunting): picked mon.noname-c con 0x7fa550135d50 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa556da0640 10 monclient(hunting): picked mon.noname-i con 0x7fa55013f860 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa556da0640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5501372e0 con 0x7fa55006cdb0 2024-08-19T09:50:19.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa55559d640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fa550135d50 0x7fa5500a25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57376/0 (socket says 172.21.15.1:57376) 2024-08-19T09:50:19.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa55559d640 1 -- 172.21.15.1:0/611088592 learned_addr learned my addr 172.21.15.1:0/611088592 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5500a5450 con 0x7fa550135d50 2024-08-19T09:50:19.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5500a3990 con 0x7fa55013f860 2024-08-19T09:50:19.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa556da0640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7fa556da0640 10 monclient(hunting): authenticate will time out at 675.746236s 2024-08-19T09:50:19.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 client.?.objecter ms_handle_connect 0x7f848c0a4650 2024-08-19T09:50:19.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700510282 0 0) 0x7f848c0a2a30 con 0x7f848c0a4f80 2024-08-19T09:50:19.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): my global_id is 4310 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8478004eb0 con 0x7f848c0a4f80 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3646503484 0 0) 0x7f848c0a2460 con 0x7f848c13fcd0 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): my global_id is 4267 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f848c0a2a30 con 0x7f848c13fcd0 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667521116 0 0) 0x7f848c13e9d0 con 0x7f848c0a4650 2024-08-19T09:50:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 10 monclient(hunting): my global_id is 4309 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.670+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f848c0a2460 con 0x7f848c0a4650 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1690555007 0 0) 0x7f8478004eb0 con 0x7f848c0a4f80 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f848c13e9d0 con 0x7f848c0a4f80 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8480003950 con 0x7f848c0a4f80 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): dump: 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.711 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1509219649 0 0) 0x7f848c0a2a30 con 0x7f848c13fcd0 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8478004eb0 con 0x7f848c13fcd0 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8484002090 con 0x7f848c13fcd0 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient(hunting): dump: 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.712 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.713 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1601733082 0 0) 0x7f848c13e9d0 con 0x7f848c0a4f80 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.82:6792/0 conn(0x7f848c0a4650 legacy=0x7f848c13dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.82:6789/0 conn(0x7f848c13fcd0 legacy=0x7f848c1dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 monclient: found mon.e 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c06cff0 con 0x7f848c0a4f80 2024-08-19T09:50:19.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 5 monclient: authenticate success, global_id 4310 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 10 monclient: _renew_subs 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f848c0a0f70 con 0x7f848c0a4f80 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 10 monclient: _renew_subs 2024-08-19T09:50:19.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f848c13b6c0 con 0x7f848c0a4f80 2024-08-19T09:50:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f84920de640 1 librados: init done 2024-08-19T09:50:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8480001c90 con 0x7f848c0a4f80 2024-08-19T09:50:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f84800042c0 con 0x7f848c0a4f80 2024-08-19T09:50:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa55659f640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.82:6792/0 conn(0x7fa55013f860 legacy=0x7fa5500a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 client.?.objecter ms_handle_connect 0x7fa55013f860 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580006996 0 0) 0x7fa5500a3990 con 0x7fa55013f860 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 monclient(hunting): my global_id is 4318 2024-08-19T09:50:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa534004eb0 con 0x7fa55013f860 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa55559d640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.1:6791/0 conn(0x7fa550135d50 legacy=0x7fa5500a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 client.?.objecter ms_handle_connect 0x7fa550135d50 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa555d9e640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.1:6790/0 conn(0x7fa55006cdb0 legacy=0x7fa5500a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 client.?.objecter ms_handle_connect 0x7fa55006cdb0 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.672+0000 7fa5467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8473fff640 10 monclient: dump: 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.719 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266816241 0 0) 0x7fa5500a7bf0 con 0x7fa550135d50 2024-08-19T09:50:19.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.671+0000 7f8489ffb640 4 set_mon_vals no callback set 2024-08-19T09:50:19.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8480011a80 con 0x7f848c0a4f80 2024-08-19T09:50:19.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f84920de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): my global_id is 4288 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5500a3990 con 0x7fa550135d50 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 275713718 0 0) 0x7fa534004eb0 con 0x7fa55013f860 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5500a7bf0 con 0x7fa55013f860 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 216368773 0 0) 0x7fa53c003520 con 0x7fa55006cdb0 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): my global_id is 4304 2024-08-19T09:50:19.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa534004eb0 con 0x7fa55006cdb0 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa548003e80 con 0x7fa55013f860 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): dump: 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f84920de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f84920de640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846c005080 con 0x7f848c0a4f80 2024-08-19T09:50:19.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f848004daf0 con 0x7f848c0a4f80 2024-08-19T09:50:19.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f8473fff640 10 client.4310.objecter ms_dispatch 0x7f848c000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3622583065 0 0) 0x7fa5500a7bf0 con 0x7fa55013f860 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.1:6791/0 conn(0x7fa550135d50 legacy=0x7fa5500a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.1:6790/0 conn(0x7fa55006cdb0 legacy=0x7fa5500a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 monclient: found mon.i 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa550136be0 con 0x7fa55013f860 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa5467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 5 monclient: authenticate success, global_id 4318 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 10 monclient: _renew_subs 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa55013e510 con 0x7fa55013f860 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 10 monclient: _renew_subs 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa55006eab0 con 0x7fa55013f860 2024-08-19T09:50:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7fa556da0640 1 librados: init done 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa548003990 con 0x7fa55013f860 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa5467fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa548004610 con 0x7fa55013f860 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa5467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa5467fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f8473fff640 3 client.4310.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f8473fff640 3 client.4310.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.673+0000 7f8473fff640 20 client.4310.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7f848b7fe640 10 monclient: get_auth_request con 0x7f8478041640 auth_method 0 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa5467fc640 10 monclient: dump: 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.730 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.674+0000 7fa547fff640 4 set_mon_vals no callback set 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa548011e00 con 0x7fa55013f860 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa556da0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa556da0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa556da0640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa538005080 con 0x7fa55013f860 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fa54804dc40 con 0x7fa55013f860 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa5467fc640 10 client.4318.objecter ms_dispatch 0x7fa550000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa5467fc640 3 client.4318.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa5467fc640 3 client.4318.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa5467fc640 20 client.4318.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.675+0000 7fa555d9e640 10 monclient: get_auth_request con 0x7fa5340411d0 auth_method 0 2024-08-19T09:50:19.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.676+0000 7f8473fff640 10 client.4310.objecter ms_handle_connect 0x7f8478041640 2024-08-19T09:50:19.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.678+0000 7fa5467fc640 10 client.4318.objecter ms_handle_connect 0x7fa5340411d0 2024-08-19T09:50:19.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.679+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f84800176b0 con 0x7f848c0a4f80 2024-08-19T09:50:19.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.679+0000 7f8473fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.679+0000 7f8473fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.680+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa5480176e0 con 0x7fa55013f860 2024-08-19T09:50:19.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.681+0000 7fa5467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.681+0000 7fa5467fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.702+0000 7f7e99e48640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.702+0000 7f7e99e48640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.702+0000 7f7e99e48640 10 monclient: monmap: 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.703824+0000 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.703824+0000 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.703+0000 7f7e99e48640 1 Processor -- start 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.703+0000 7f7e99e48640 1 -- start start 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.703+0000 7f7e99e48640 10 monclient: init 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient: _add_conns ranks=[1,2,4,7,3,0,6,8,5] 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient(hunting): picked mon.noname-f con 0x7f7e940a4f80 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient(hunting): picked mon.noname-b con 0x7f7e940a4650 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient(hunting): picked mon.noname-c con 0x7f7e9413fcd0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e940a4a60 con 0x7f7e940a4f80 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e94069270 con 0x7f7e940a4650 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e940a3c70 con 0x7f7e9413fcd0 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e99e48640 10 monclient(hunting): authenticate will time out at 675.780688s 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e92ffd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7e940a4650 0x7f7e9413e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52766/0 (socket says 172.21.15.1:52766) 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e92ffd640 1 -- 172.21.15.1:0/2001376705 learned_addr learned my addr 172.21.15.1:0/2001376705 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.704+0000 7f7e937fe640 1 --1- 172.21.15.1:0/2001376705 >> v1:172.21.15.82:6789/0 conn(0x7f7e940a4f80 0x7f7e940a5360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:45550/0 (socket says 172.21.15.1:45550) 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.706+0000 7fa3a24e5640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.706+0000 7fa3a24e5640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.706+0000 7fa3a24e5640 10 monclient: monmap: 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.707554+0000 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.707554+0000 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.740 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.706+0000 7f7e92ffd640 1 -- 172.21.15.1:0/2001376705 >> v1:172.21.15.1:6790/0 conn(0x7f7e940a4650 legacy=0x7f7e9413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.706+0000 7f7e937fe640 1 -- 172.21.15.1:0/2001376705 >> v1:172.21.15.82:6789/0 conn(0x7f7e940a4f80 legacy=0x7f7e940a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e93fff640 1 -- 172.21.15.1:0/2001376705 >> v1:172.21.15.1:6791/0 conn(0x7f7e9413fcd0 legacy=0x7f7e941420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502465867 0 0) 0x7f7e94069270 con 0x7f7e940a4650 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): my global_id is 4313 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e7c003520 con 0x7f7e940a4650 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383376760 0 0) 0x7f7e940a3c70 con 0x7f7e9413fcd0 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient(hunting): my global_id is 4297 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e94069270 con 0x7f7e9413fcd0 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1872620981 0 0) 0x7f7e7c003520 con 0x7f7e940a4650 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 >> v1:172.21.15.1:6791/0 conn(0x7f7e9413fcd0 legacy=0x7f7e941420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 >> v1:172.21.15.82:6789/0 conn(0x7f7e940a4f80 legacy=0x7f7e940a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 monclient: found mon.noname-b 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e940a3420 con 0x7f7e940a4650 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7e88001ae0 con 0x7f7e940a4650 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e927fc640 10 monclient: dump: 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.746 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e99e48640 5 monclient: authenticate success, global_id 4313 2024-08-19T09:50:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.707+0000 7f7e99e48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e880048a0 con 0x7f7e940a4650 2024-08-19T09:50:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 1 Processor -- start 2024-08-19T09:50:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e927fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e927fc640 1 -- 172.21.15.1:0/2001376705 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7e88004e10 con 0x7f7e940a4650 2024-08-19T09:50:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e99e48640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e99e48640 4 set_mon_vals no callback set 2024-08-19T09:50:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 1 -- start start 2024-08-19T09:50:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient: init 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e927fc640 10 monclient: dump: 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.749 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient: _add_conns ranks=[0,3,1,6,7,5,2,8,4] 2024-08-19T09:50:19.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient(hunting): picked mon.noname-a con 0x7fa39c070b80 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient(hunting): picked mon.noname-g con 0x7fa39c0a3e00 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient(hunting): picked mon.noname-f con 0x7fa39c071670 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c0727c0 con 0x7fa39c070b80 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c072aa0 con 0x7fa39c071670 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c072d80 con 0x7fa39c0a3e00 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa3a24e5640 10 monclient(hunting): authenticate will time out at 675.784742s 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa39bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa39c070b80 0x7fa39c070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37176/0 (socket says 172.21.15.1:37176) 2024-08-19T09:50:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7fa39bfff640 1 -- 172.21.15.1:0/1811041832 learned_addr learned my addr 172.21.15.1:0/1811041832 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e99e48640 10 monclient: shutdown 2024-08-19T09:50:19.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.708+0000 7f7e99e48640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 1 -- 172.21.15.1:0/2001376705 >> v1:172.21.15.1:6790/0 conn(0x7f7e940a4650 legacy=0x7f7e9413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 1 -- 172.21.15.1:0/2001376705 shutdown_connections 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 1 -- 172.21.15.1:0/2001376705 >> 172.21.15.1:0/2001376705 conn(0x7f7e940a8ef0 msgr2=0x7f7e94068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 1 -- 172.21.15.1:0/2001376705 shutdown_connections 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 1 -- 172.21.15.1:0/2001376705 wait complete. 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.709+0000 7f7e99e48640 10 monclient: monmap: 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.753 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 1 librados: starting msgr at 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 1 librados: starting objecter 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 1 Processor -- start 2024-08-19T09:50:19.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 1 -- start start 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 1 librados: setting wanted keys 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 1 librados: calling monclient init 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f7e99e48640 10 monclient: init 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f9e953aa640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f9e953aa640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.710+0000 7f9e953aa640 10 monclient: monmap: 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.711469+0000 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.711469+0000 2024-08-19T09:50:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.756 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient: _add_conns ranks=[8,6,5,1,7,0,2,3,4] 2024-08-19T09:50:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39bfff640 1 -- 172.21.15.1:0/1811041832 >> v1:172.21.15.1:6789/0 conn(0x7fa39c070b80 legacy=0x7fa39c070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f6fd4c27640 10 monclient: get_monmap_and_config 2024-08-19T09:50:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f6fd4c27640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient(hunting): picked mon.noname-e con 0x7f7e940a4650 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f6fd4c27640 10 monclient: monmap: 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:19.712067+0000 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:19.712067+0000 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f9e953aa640 1 Processor -- start 2024-08-19T09:50:19.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient(hunting): picked mon.noname-d con 0x7f7e940a4f80 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient(hunting): picked mon.noname-h con 0x7f7e9413fcd0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e940a1c10 con 0x7f7e9413fcd0 2024-08-19T09:50:19.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e94139540 con 0x7f7e940a4f80 2024-08-19T09:50:19.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e94139820 con 0x7f7e940a4650 2024-08-19T09:50:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e99e48640 10 monclient(hunting): authenticate will time out at 675.787373s 2024-08-19T09:50:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa3a0a82640 1 -- 172.21.15.1:0/1811041832 >> v1:172.21.15.82:6789/0 conn(0x7fa39c071670 legacy=0x7fa39c13ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39b7fe640 1 -- 172.21.15.1:0/1811041832 >> v1:172.21.15.82:6790/0 conn(0x7fa39c0a3e00 legacy=0x7fa39c13c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e93fff640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f7e9413fcd0 0x7f7e940a1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:34716/0 (socket says 172.21.15.1:34716) 2024-08-19T09:50:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f7e93fff640 1 -- 172.21.15.1:0/3319189926 learned_addr learned my addr 172.21.15.1:0/3319189926 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f9e953aa640 1 -- start start 2024-08-19T09:50:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7f9e953aa640 10 monclient: init 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617297773 0 0) 0x7fa39c0727c0 con 0x7fa39c070b80 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): my global_id is 4329 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa384003520 con 0x7fa39c070b80 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3108978448 0 0) 0x7fa39c072aa0 con 0x7fa39c071670 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 10 monclient(hunting): my global_id is 4285 2024-08-19T09:50:19.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa39c0727c0 con 0x7fa39c071670 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1917351933 0 0) 0x7fa39c0727c0 con 0x7fa39c071670 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 >> v1:172.21.15.82:6790/0 conn(0x7fa39c0a3e00 legacy=0x7fa39c13c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.711+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 >> v1:172.21.15.1:6789/0 conn(0x7fa39c070b80 legacy=0x7fa39c070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 1 monclient: found mon.noname-f 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39c0a58a0 con 0x7fa39c071670 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa394001e90 con 0x7fa39c071670 2024-08-19T09:50:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: dump: 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.766 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa3a24e5640 5 monclient: authenticate success, global_id 4285 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa3a24e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 493538184 0 0) 0x7fa384003520 con 0x7fa39c070b80 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa38c001ae0 con 0x7fa39c070b80 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa3a0a82640 1 --1- 172.21.15.1:0/1811041832 >> v1:172.21.15.82:6789/0 conn(0x7fa39c071670 0x7fa39c13ff80 :-1 s=OPENED pgs=62 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fa394003f00 mon_map magic: 0 2024-08-19T09:50:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa394003990 con 0x7fa39c071670 2024-08-19T09:50:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa39affd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa3a24e5640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7fa3a24e5640 4 set_mon_vals no callback set 2024-08-19T09:50:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f6fd4c27640 1 Processor -- start 2024-08-19T09:50:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f6fd4c27640 1 -- start start 2024-08-19T09:50:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f6fd4c27640 10 monclient: init 2024-08-19T09:50:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient: _add_conns ranks=[3,4,0,6,8,1,2,5,7] 2024-08-19T09:50:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient(hunting): picked mon.noname-g con 0x7f9e900a1700 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9009f410 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9013c0d0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e900a3ec0 con 0x7f9e9013c0d0 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e8effd640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f9e900a1700 0x7f9e900a1ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48426/0 (socket says 172.21.15.1:48426) 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e8effd640 1 -- 172.21.15.1:0/1335182817 learned_addr learned my addr 172.21.15.1:0/1335182817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e900a3c40 con 0x7f9e900a1700 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 1 -- 172.21.15.1:0/1335182817 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9009f930 con 0x7f9e9009f410 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.712+0000 7f9e953aa640 10 monclient(hunting): authenticate will time out at 675.788820s 2024-08-19T09:50:19.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 10 monclient: shutdown 2024-08-19T09:50:19.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 1 -- 172.21.15.1:0/1811041832 >> v1:172.21.15.82:6789/0 conn(0x7fa39c071670 legacy=0x7fa39c13ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 1 -- 172.21.15.1:0/1811041832 shutdown_connections 2024-08-19T09:50:19.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 1 -- 172.21.15.1:0/1811041832 >> 172.21.15.1:0/1811041832 conn(0x7fa39c068650 msgr2=0x7fa39c068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa39affd640 1 -- 172.21.15.1:0/1811041832 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa394003f00 con 0x7fa39c071670 2024-08-19T09:50:19.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa39affd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:50:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient: _add_conns ranks=[4,6,7,0,1,3,2,5,8] 2024-08-19T09:50:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd006fae0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient(hunting): picked mon.noname-d con 0x7f6fd00705d0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 1 -- 172.21.15.1:0/1811041832 shutdown_connections 2024-08-19T09:50:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient(hunting): picked mon.noname-i con 0x7f6fd0072370 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd0071c70 con 0x7f6fd006fae0 2024-08-19T09:50:19.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7fa3a24e5640 1 -- 172.21.15.1:0/1811041832 wait complete. 2024-08-19T09:50:19.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fce59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f6fd006fae0 0x7f6fd006fec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57404/0 (socket says 172.21.15.1:57404) 2024-08-19T09:50:19.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fce59c640 1 -- 172.21.15.1:0/3549188471 learned_addr learned my addr 172.21.15.1:0/3549188471 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd0071f50 con 0x7f6fd00705d0 2024-08-19T09:50:19.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd00a46b0 con 0x7f6fd0072370 2024-08-19T09:50:19.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f6fd4c27640 10 monclient(hunting): authenticate will time out at 675.789414s 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e93fff640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.82:6791/0 conn(0x7f7e9413fcd0 legacy=0x7f7e940a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 client.?.objecter ms_handle_connect 0x7f7e9413fcd0 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e937fe640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.1:6793/0 conn(0x7f7e940a4650 legacy=0x7f7e940a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 client.?.objecter ms_handle_connect 0x7f7e940a4650 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 client.?.objecter ms_handle_connect 0x7f7e940a4f80 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e92ffd640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.1:6792/0 conn(0x7f7e940a4f80 legacy=0x7f7e940a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230125267 0 0) 0x7f7e940a1c10 con 0x7f7e9413fcd0 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 10 monclient(hunting): my global_id is 4343 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.713+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e80004eb0 con 0x7f7e9413fcd0 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748897578 0 0) 0x7f7e94139820 con 0x7f7e940a4650 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): my global_id is 4319 2024-08-19T09:50:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e940a1c10 con 0x7f7e940a4650 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 10 monclient: monmap: 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.777 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.778 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2382326817 0 0) 0x7f7e80004eb0 con 0x7f7e9413fcd0 2024-08-19T09:50:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e94139820 con 0x7f7e9413fcd0 2024-08-19T09:50:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7e8c002090 con 0x7f7e9413fcd0 2024-08-19T09:50:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:50:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): dump: 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3290010339 0 0) 0x7f7e94139820 con 0x7f7e9413fcd0 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.1:6793/0 conn(0x7f7e940a4650 legacy=0x7f7e940a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.1:6792/0 conn(0x7f7e940a4f80 legacy=0x7f7e940a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 monclient: found mon.h 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9413f740 con 0x7f7e9413fcd0 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 1 librados: starting msgr at 2024-08-19T09:50:19.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 1 librados: starting objecter 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 5 monclient: authenticate success, global_id 4343 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 10 monclient: _renew_subs 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e94138910 con 0x7f7e9413fcd0 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 10 monclient: _renew_subs 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e941d79a0 con 0x7f7e9413fcd0 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e99e48640 1 librados: init done 2024-08-19T09:50:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 1 Processor -- start 2024-08-19T09:50:19.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e8c003cc0 con 0x7f7e9413fcd0 2024-08-19T09:50:19.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7e8c0041f0 con 0x7f7e9413fcd0 2024-08-19T09:50:19.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f9e8effd640 1 -- 172.21.15.1:0/1335182817 >> v1:172.21.15.82:6790/0 conn(0x7f9e900a1700 legacy=0x7f9e900a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 1 -- start start 2024-08-19T09:50:19.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 1 librados: setting wanted keys 2024-08-19T09:50:19.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 1 librados: calling monclient init 2024-08-19T09:50:19.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7fa3a24e5640 10 monclient: init 2024-08-19T09:50:19.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/1335182817 >> v1:172.21.15.1:6789/0 conn(0x7f9e9013c0d0 legacy=0x7f9e9013e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2657507564 0 0) 0x7f9e900a3c40 con 0x7f9e900a1700 2024-08-19T09:50:19.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): my global_id is 4323 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e78003520 con 0x7f9e900a1700 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319772219 0 0) 0x7f9e900a3ec0 con 0x7f9e9013c0d0 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8e7fc640 1 -- 172.21.15.1:0/1335182817 >> v1:172.21.15.1:6791/0 conn(0x7f9e9009f410 legacy=0x7f9e9013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f6fce59c640 1 -- 172.21.15.1:0/3549188471 >> v1:172.21.15.1:6791/0 conn(0x7f6fd006fae0 legacy=0x7f6fd006fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f6fced9d640 1 -- 172.21.15.1:0/3549188471 >> v1:172.21.15.82:6792/0 conn(0x7f6fd0072370 legacy=0x7f6fd013eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e91ffb640 4 set_mon_vals no callback set 2024-08-19T09:50:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.714+0000 7f7e7bfff640 10 monclient: dump: 2024-08-19T09:50:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.789 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient(hunting): my global_id is 4338 2024-08-19T09:50:19.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e900a3c40 con 0x7f9e9013c0d0 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7e8c004390 con 0x7f7e9413fcd0 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient: _add_conns ranks=[5,6,7,1,4,3,8,2,0] 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient(hunting): picked mon.noname-h con 0x7fa39c070b80 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient(hunting): picked mon.noname-d con 0x7fa39c0a3e00 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient(hunting): picked mon.noname-i con 0x7fa39c1c9ac0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c06d3b0 con 0x7fa39c070b80 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c126220 con 0x7fa39c0a3e00 2024-08-19T09:50:19.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c0a93b0 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 9621888 0 0) 0x7f9e78003520 con 0x7f9e900a1700 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 >> v1:172.21.15.1:6791/0 conn(0x7f9e9009f410 legacy=0x7f9e9013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 >> v1:172.21.15.1:6789/0 conn(0x7f9e9013c0d0 legacy=0x7f9e9013e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.715+0000 7f9e8dffb640 1 monclient: found mon.noname-g 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9009fc10 con 0x7f9e900a1700 2024-08-19T09:50:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9e88001e60 con 0x7f9e900a1700 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: dump: 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.793 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e953aa640 5 monclient: authenticate success, global_id 4323 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e953aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e88004c20 con 0x7f9e900a1700 2024-08-19T09:50:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 1 -- 172.21.15.1:0/1335182817 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9e88005110 con 0x7f9e900a1700 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e8dffb640 10 monclient: dump: 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa39b7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa39c0a3e00 0x7fa39c1cee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52866/0 (socket says 172.21.15.1:52866) 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa39b7fe640 1 -- 172.21.15.1:0/2526719260 learned_addr learned my addr 172.21.15.1:0/2526719260 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7fa3a24e5640 10 monclient(hunting): authenticate will time out at 675.792474s 2024-08-19T09:50:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e953aa640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f9e953aa640 4 set_mon_vals no callback set 2024-08-19T09:50:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f7e8c04dda0 con 0x7f7e9413fcd0 2024-08-19T09:50:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995261195 0 0) 0x7f6fd00a46b0 con 0x7f6fd0072370 2024-08-19T09:50:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): my global_id is 4327 2024-08-19T09:50:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fbc003520 con 0x7f6fd0072370 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fc7fff640 1 -- 172.21.15.1:0/3549188471 >> v1:172.21.15.1:6792/0 conn(0x7f6fd00705d0 legacy=0x7f6fd013b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3478187775 0 0) 0x7f6fd0071c70 con 0x7f6fd006fae0 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient(hunting): my global_id is 4315 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fbc003880 con 0x7f6fd006fae0 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 994334479 0 0) 0x7f6fbc003520 con 0x7f6fd0072370 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 >> v1:172.21.15.1:6792/0 conn(0x7f6fd00705d0 legacy=0x7f6fd013b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 >> v1:172.21.15.1:6791/0 conn(0x7f6fd006fae0 legacy=0x7f6fd006fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 monclient: found mon.noname-i 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd00a4880 con 0x7f6fd0072370 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8001e90 con 0x7f6fd0072370 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: dump: 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fd4c27640 5 monclient: authenticate success, global_id 4327 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fd4c27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:19.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fc8003990 con 0x7f6fd0072370 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fd4c27640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fd4c27640 4 set_mon_vals no callback set 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 1 -- 172.21.15.1:0/3549188471 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8001e90 con 0x7f6fd0072370 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f6fcdd9b640 10 monclient: dump: 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.802 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.803 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e7bfff640 10 client.4343.objecter ms_dispatch 0x7f7e94000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e7bfff640 3 client.4343.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e7bfff640 3 client.4343.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e99e48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e7bfff640 20 client.4343.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e99e48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e99e48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.716+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e74005080 con 0x7f7e9413fcd0 2024-08-19T09:50:19.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f7e937fe640 10 monclient: get_auth_request con 0x7f7e80040e10 auth_method 0 2024-08-19T09:50:19.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 10 monclient: shutdown 2024-08-19T09:50:19.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 1 -- 172.21.15.1:0/1335182817 >> v1:172.21.15.82:6790/0 conn(0x7f9e900a1700 legacy=0x7f9e900a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 1 -- 172.21.15.1:0/1335182817 shutdown_connections 2024-08-19T09:50:19.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 1 -- 172.21.15.1:0/1335182817 >> 172.21.15.1:0/1335182817 conn(0x7f9e900a8210 msgr2=0x7f9e900a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 1 -- 172.21.15.1:0/1335182817 shutdown_connections 2024-08-19T09:50:19.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f9e953aa640 1 -- 172.21.15.1:0/1335182817 wait complete. 2024-08-19T09:50:19.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 10 monclient: shutdown 2024-08-19T09:50:19.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 >> v1:172.21.15.82:6792/0 conn(0x7f6fd0072370 legacy=0x7f6fd013eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 shutdown_connections 2024-08-19T09:50:19.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 >> 172.21.15.1:0/3549188471 conn(0x7f6fd0068650 msgr2=0x7f6fd0068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 shutdown_connections 2024-08-19T09:50:19.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.717+0000 7f6fd4c27640 1 -- 172.21.15.1:0/3549188471 wait complete. 2024-08-19T09:50:19.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 10 monclient: monmap: 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.808 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 1 librados: starting msgr at 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 1 librados: starting objecter 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 1 Processor -- start 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 1 -- start start 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 1 librados: setting wanted keys 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 1 librados: calling monclient init 2024-08-19T09:50:19.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f9e953aa640 10 monclient: init 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f6fd4c27640 10 monclient: build_initial_monmap 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7f6fd4c27640 10 monclient: monmap: 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.810 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:19.811 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7fa3a0a82640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.82:6792/0 conn(0x7fa39c1c9ac0 legacy=0x7fa39c1d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7fa398ff9640 10 client.?.objecter ms_handle_connect 0x7fa39c1c9ac0 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7fa398ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7fa39b7fe640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.1:6792/0 conn(0x7fa39c0a3e00 legacy=0x7fa39c1cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7fa398ff9640 10 client.?.objecter ms_handle_connect 0x7fa39c0a3e00 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.718+0000 7fa398ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7fa39bfff640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.82:6791/0 conn(0x7fa39c070b80 legacy=0x7fa39c137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:50:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 1 librados: starting msgr at 2024-08-19T09:50:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 1 librados: starting objecter 2024-08-19T09:50:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 1 Processor -- start 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient: _add_conns ranks=[8,5,1,3,7,6,2,4,0] 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient(hunting): picked mon.noname-e con 0x7f9e9009f410 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient(hunting): picked mon.noname-h con 0x7f9e900a1700 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient(hunting): picked mon.noname-f con 0x7f9e9013c0d0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e90071650 con 0x7f9e9013c0d0 2024-08-19T09:50:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 1 -- start start 2024-08-19T09:50:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e90071930 con 0x7f9e900a1700 2024-08-19T09:50:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e8effd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f9e9009f410 0x7f9e90139f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44684/0 (socket says 172.21.15.1:44684) 2024-08-19T09:50:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e8effd640 1 -- 172.21.15.1:0/3213220123 learned_addr learned my addr 172.21.15.1:0/3213220123 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9013adb0 con 0x7f9e9009f410 2024-08-19T09:50:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f9e953aa640 10 monclient(hunting): authenticate will time out at 675.795644s 2024-08-19T09:50:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 1 librados: setting wanted keys 2024-08-19T09:50:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 1 librados: calling monclient init 2024-08-19T09:50:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f6fd4c27640 10 monclient: init 2024-08-19T09:50:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.719+0000 7f7e7bfff640 10 client.4343.objecter ms_handle_connect 0x7f7e80040e10 2024-08-19T09:50:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient: _add_conns ranks=[4,7,2,0,6,3,8,5,1] 2024-08-19T09:50:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd006fae0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient(hunting): picked mon.noname-i con 0x7f6fd00705d0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fce59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f6fd006fae0 0x7f6fd01366c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57408/0 (socket says 172.21.15.1:57408) 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fce59c640 1 -- 172.21.15.1:0/1469429431 learned_addr learned my addr 172.21.15.1:0/1469429431 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd0072370 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd00a7fb0 con 0x7f6fd0072370 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd0121d20 con 0x7f6fd006fae0 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd0123da0 con 0x7f6fd00705d0 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.720+0000 7f6fd4c27640 10 monclient(hunting): authenticate will time out at 675.796615s 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 client.?.objecter ms_handle_connect 0x7fa39c070b80 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1245693438 0 0) 0x7fa39c0a93b0 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): my global_id is 4336 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa388004ee0 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588505830 0 0) 0x7fa39c126220 con 0x7fa39c0a3e00 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): my global_id is 4290 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa39c0a93b0 con 0x7fa39c0a3e00 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1584560340 0 0) 0x7fa388004ee0 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa39c126220 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa394003e80 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4039145577 0 0) 0x7fa39c0a93b0 con 0x7fa39c0a3e00 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa388004ee0 con 0x7fa39c0a3e00 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa390001c00 con 0x7fa39c0a3e00 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3127959169 0 0) 0x7fa39c126220 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.1:6792/0 conn(0x7fa39c0a3e00 legacy=0x7fa39c1cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.82:6791/0 conn(0x7fa39c070b80 legacy=0x7fa39c137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e8effd640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.1:6793/0 conn(0x7f9e9009f410 legacy=0x7f9e90139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e777fe640 10 client.?.objecter ms_handle_connect 0x7f9e9009f410 2024-08-19T09:50:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 monclient: found mon.i 2024-08-19T09:50:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39c0a8f60 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa398ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.82:6789/0 conn(0x7f9e9013c0d0 legacy=0x7f9e901d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e8e7fc640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.82:6791/0 conn(0x7f9e900a1700 legacy=0x7f9e9013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e777fe640 10 client.?.objecter ms_handle_connect 0x7f9e9013c0d0 2024-08-19T09:50:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e777fe640 10 client.?.objecter ms_handle_connect 0x7f9e900a1700 2024-08-19T09:50:19.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7f9e777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa3a24e5640 5 monclient: authenticate success, global_id 4336 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa3a24e5640 10 monclient: _renew_subs 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa3a24e5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.721+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa39c124ba0 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa3a24e5640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa394001eb0 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa3a24e5640 10 monclient: _renew_subs 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa3a24e5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa39c125f40 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa3a24e5640 1 librados: init done 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa398ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa394004710 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa398ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa398ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa398ff9640 10 monclient: dump: 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7fa39a7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71174499 0 0) 0x7f9e9013adb0 con 0x7f9e9009f410 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): my global_id is 4328 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e7c004ee0 con 0x7f9e9009f410 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407528758 0 0) 0x7f9e90071650 con 0x7f9e9013c0d0 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): my global_id is 4294 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e9013adb0 con 0x7f9e9013c0d0 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3335564963 0 0) 0x7f9e90071930 con 0x7f9e900a1700 2024-08-19T09:50:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): my global_id is 4352 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e90071650 con 0x7f9e900a1700 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f7e8c017840 con 0x7f7e9413fcd0 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f7e7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f7e7bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fced9d640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.1:6790/0 conn(0x7f6fd0072370 legacy=0x7f6fd01d66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd0072370 2024-08-19T09:50:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1395543781 0 0) 0x7f9e90071650 con 0x7f9e900a1700 2024-08-19T09:50:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e90071930 con 0x7f9e900a1700 2024-08-19T09:50:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2633853747 0 0) 0x7f9e9013adb0 con 0x7f9e9013c0d0 2024-08-19T09:50:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e90071650 con 0x7f9e9013c0d0 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9e84001d10 con 0x7f9e900a1700 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): dump: 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd006fae0 2024-08-19T09:50:19.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fce59c640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.1:6791/0 conn(0x7f6fd006fae0 legacy=0x7f6fd01366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc7fff640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.82:6792/0 conn(0x7f6fd00705d0 legacy=0x7f6fd01d2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd00705d0 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9e80001cf0 con 0x7f9e9013c0d0 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f9e777fe640 10 monclient(hunting): dump: 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.836 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.837 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719468216 0 0) 0x7f6fd00a4880 con 0x7f6fd0072370 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 4322 2024-08-19T09:50:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.722+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fb0004eb0 con 0x7f6fd0072370 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4088856839 0 0) 0x7f9e7c004ee0 con 0x7f9e9009f410 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e9013adb0 con 0x7f9e9009f410 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9e88001de0 con 0x7f9e9009f410 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient(hunting): dump: 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.839 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.840 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1131781636 0 0) 0x7f6fd0121d20 con 0x7f6fd006fae0 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3389188592 0 0) 0x7f9e90071930 con 0x7f9e900a1700 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.1:6793/0 conn(0x7f9e9009f410 legacy=0x7f9e90139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.82:6789/0 conn(0x7f9e9013c0d0 legacy=0x7f9e901d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 monclient: found mon.h 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e900a6240 con 0x7f9e900a1700 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 4324 2024-08-19T09:50:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e840038c0 con 0x7f9e900a1700 2024-08-19T09:50:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9e84003df0 con 0x7f9e900a1700 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e777fe640 10 monclient: dump: 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 5 monclient: authenticate success, global_id 4352 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 10 monclient: _renew_subs 2024-08-19T09:50:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd00a4880 con 0x7f6fd006fae0 2024-08-19T09:50:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3613575028 0 0) 0x7f6fb0004eb0 con 0x7f6fd0072370 2024-08-19T09:50:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd0121d20 con 0x7f6fd0072370 2024-08-19T09:50:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8001e90 con 0x7f6fd0072370 2024-08-19T09:50:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): dump: 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa3a24e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 826185893 0 0) 0x7f6fd0121d20 con 0x7f6fd0072370 2024-08-19T09:50:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa3a24e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa3a24e5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.82:6792/0 conn(0x7f6fd00705d0 legacy=0x7f6fd01d2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa39c050c10 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.1:6791/0 conn(0x7f6fd006fae0 legacy=0x7f6fd01366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 monclient: found mon.b 2024-08-19T09:50:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa394011e00 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd0124080 con 0x7f6fd0072370 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 5 monclient: authenticate success, global_id 4322 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 10 monclient: _renew_subs 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e8d7fa640 4 set_mon_vals no callback set 2024-08-19T09:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9e90071370 con 0x7f9e900a1700 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 10 monclient: _renew_subs 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9e9013feb0 con 0x7f9e900a1700 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f9e953aa640 1 librados: init done 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd01237e0 con 0x7f6fd0072370 2024-08-19T09:50:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fa39404dc30 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa398ff9640 10 client.4336.objecter ms_dispatch 0x7fa39c000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa398ff9640 3 client.4336.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa398ff9640 3 client.4336.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 10 monclient: _renew_subs 2024-08-19T09:50:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.723+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6fd01cdee0 con 0x7f6fd0072370 2024-08-19T09:50:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7fa398ff9640 20 client.4336.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7f6fd4c27640 1 librados: init done 2024-08-19T09:50:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fc8003990 con 0x7f6fd0072370 2024-08-19T09:50:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7f6fc6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.724+0000 7f6fcd59a640 4 set_mon_vals no callback set 2024-08-19T09:50:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8004550 con 0x7f6fd0072370 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f9e953aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f9e953aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f9e953aa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9013f750 con 0x7f9e900a1700 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9e84003f90 con 0x7f9e900a1700 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f6fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f6fc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7fa39bfff640 10 monclient: get_auth_request con 0x7fa388041550 auth_method 0 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.725+0000 7f6fc6ffd640 10 monclient: dump: 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6fc8011e00 con 0x7f6fd0072370 2024-08-19T09:50:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fd4c27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fd4c27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f9e8404d890 con 0x7f9e900a1700 2024-08-19T09:50:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fd4c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fb4005080 con 0x7f6fd0072370 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f9e777fe640 10 client.4352.objecter ms_dispatch 0x7f9e90000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f9e777fe640 3 client.4352.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f6fc804dd00 con 0x7f6fd0072370 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fc6ffd640 10 client.4322.objecter ms_dispatch 0x7f6fd0000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fc6ffd640 3 client.4322.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fc6ffd640 3 client.4322.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f6fc6ffd640 20 client.4322.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f9e777fe640 3 client.4352.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f9e777fe640 20 client.4352.objecter dump_active .. 0 homeless 2024-08-19T09:50:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.726+0000 7f9e8effd640 10 monclient: get_auth_request con 0x7f9e7c041a30 auth_method 0 2024-08-19T09:50:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.727+0000 7f6fce59c640 10 monclient: get_auth_request con 0x7f6fb0041060 auth_method 0 2024-08-19T09:50:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.728+0000 7fa398ff9640 10 client.4336.objecter ms_handle_connect 0x7fa388041550 2024-08-19T09:50:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.729+0000 7f9e777fe640 10 client.4352.objecter ms_handle_connect 0x7f9e7c041a30 2024-08-19T09:50:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.729+0000 7f6fc6ffd640 10 client.4322.objecter ms_handle_connect 0x7f6fb0041060 2024-08-19T09:50:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.729+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa394017850 con 0x7fa39c1c9ac0 2024-08-19T09:50:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.729+0000 7fa398ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.729+0000 7fa398ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.730+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f9e84017390 con 0x7f9e900a1700 2024-08-19T09:50:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.730+0000 7f9e777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.730+0000 7f9e777fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.736+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6fd00a7fb0 con 0x7f6fd0072370 2024-08-19T09:50:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.736+0000 7f6fc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:19.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.736+0000 7f6fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:19.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f84920de640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T09:50:19.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f84920de640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T09:50:19.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f84920de640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:19.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f846c005370 con 0x7f848c0a4f80 2024-08-19T09:50:19.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f8473fff640 1 -- 172.21.15.1:0/13610143 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f8480016fb0 con 0x7f848c0a4f80 2024-08-19T09:50:19.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f8473fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T09:50:19.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.888+0000 7f8473fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:19.890 INFO:teuthology.orchestra.run.smithi001.stdout:51539607555 2024-08-19T09:50:19.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 10 librados: watch_flush enter 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 10 librados: watch_flush exit 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 20 client.4310.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 10 client.4310.objecter successfully canceled tick 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8478041640 legacy=0x7f8478043b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 10 monclient: shutdown 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 >> v1:172.21.15.1:6793/0 conn(0x7f848c0a4f80 legacy=0x7f848c13e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 shutdown_connections 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 >> 172.21.15.1:0/13610143 conn(0x7f848c0a8ef0 msgr2=0x7f848c0ab560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 shutdown_connections 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 -- 172.21.15.1:0/13610143 wait complete. 2024-08-19T09:50:19.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.891+0000 7f84920de640 1 librados: shutdown 2024-08-19T09:50:19.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-08-19T09:50:19.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e953aa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T09:50:19.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e953aa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T09:50:19.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e953aa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:19.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f9e9006ce70 con 0x7f9e900a1700 2024-08-19T09:50:19.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e777fe640 1 -- 172.21.15.1:0/3213220123 <== mon.5 v1:172.21.15.82:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1444633209) 0x7f9e84016c90 con 0x7f9e900a1700 2024-08-19T09:50:19.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T09:50:19.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.929+0000 7f9e777fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:19.931 INFO:teuthology.orchestra.run.smithi001.stdout:47244640260 2024-08-19T09:50:19.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 10 librados: watch_flush enter 2024-08-19T09:50:19.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 10 librados: watch_flush exit 2024-08-19T09:50:19.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 20 client.4352.objecter shutdown clearing up homeless session... 2024-08-19T09:50:19.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 10 client.4352.objecter successfully canceled tick 2024-08-19T09:50:19.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9e7c041a30 legacy=0x7f9e7c043ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 10 monclient: shutdown 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 >> v1:172.21.15.82:6791/0 conn(0x7f9e900a1700 legacy=0x7f9e9013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.931+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 shutdown_connections 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.932+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 >> 172.21.15.1:0/3213220123 conn(0x7f9e900a8210 msgr2=0x7f9e900a6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.932+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 shutdown_connections 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.932+0000 7f9e953aa640 1 -- 172.21.15.1:0/3213220123 wait complete. 2024-08-19T09:50:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:19.932+0000 7f9e953aa640 1 librados: shutdown 2024-08-19T09:50:19.943 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.1 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.013+0000 7fa556da0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.014+0000 7fa556da0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.014+0000 7fa556da0640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.014+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa538005370 con 0x7fa55013f860 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.014+0000 7fa5467fc640 1 -- 172.21.15.1:0/611088592 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 956180462) 0x7fa54801b580 con 0x7fa55013f860 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.014+0000 7fa5467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T09:50:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.014+0000 7fa5467fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:20.016 INFO:teuthology.orchestra.run.smithi001.stdout:51539607555 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 10 librados: watch_flush enter 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 10 librados: watch_flush exit 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 20 client.4318.objecter shutdown clearing up homeless session... 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 10 client.4318.objecter successfully canceled tick 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa5340411d0 legacy=0x7fa534043690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 10 monclient: shutdown 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:20.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 >> v1:172.21.15.82:6792/0 conn(0x7fa55013f860 legacy=0x7fa5500a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 shutdown_connections 2024-08-19T09:50:20.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 >> 172.21.15.1:0/611088592 conn(0x7fa5500abd80 msgr2=0x7fa55013ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:20.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 shutdown_connections 2024-08-19T09:50:20.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 -- 172.21.15.1:0/611088592 wait complete. 2024-08-19T09:50:20.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.017+0000 7fa556da0640 1 librados: shutdown 2024-08-19T09:50:20.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.0 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.052+0000 7f6fd4c27640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.052+0000 7f6fd4c27640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.052+0000 7f6fd4c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.052+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6fb4005370 con 0x7f6fd0072370 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.053+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/1469429431 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 2777258794) 0x7f6fc8015290 con 0x7f6fd0072370 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.053+0000 7f6fc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.053+0000 7f6fc6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:20.054 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-08-19T09:50:20.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 10 librados: watch_flush enter 2024-08-19T09:50:20.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 10 librados: watch_flush exit 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 20 client.4322.objecter shutdown clearing up homeless session... 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 10 client.4322.objecter successfully canceled tick 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6fb0041060 legacy=0x7f6fb0043520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 10 monclient: shutdown 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 >> v1:172.21.15.1:6790/0 conn(0x7f6fd0072370 legacy=0x7f6fd01d66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 shutdown_connections 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 >> 172.21.15.1:0/1469429431 conn(0x7f6fd0068650 msgr2=0x7f6fd00a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 shutdown_connections 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 -- 172.21.15.1:0/1469429431 wait complete. 2024-08-19T09:50:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.055+0000 7f6fd4c27640 1 librados: shutdown 2024-08-19T09:50:20.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.064+0000 7f7e99e48640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T09:50:20.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.064+0000 7f7e99e48640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T09:50:20.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.064+0000 7f7e99e48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:50:20.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.064+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7e74005370 con 0x7f7e9413fcd0 2024-08-19T09:50:20.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.065+0000 7f7e7bfff640 1 -- 172.21.15.1:0/3319189926 <== mon.5 v1:172.21.15.82:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 3756282479) 0x7f7e8c017140 con 0x7f7e9413fcd0 2024-08-19T09:50:20.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.065+0000 7f7e7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T09:50:20.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.065+0000 7f7e7bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:20.066 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-08-19T09:50:20.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-08-19T09:50:20.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.066+0000 7fa3a24e5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T09:50:20.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.066+0000 7f7e99e48640 10 librados: watch_flush enter 2024-08-19T09:50:20.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.066+0000 7fa3a24e5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T09:50:20.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7fa3a24e5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:20.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa39c0698e0 con 0x7fa39c1c9ac0 2024-08-19T09:50:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 10 librados: watch_flush exit 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 20 client.4343.objecter shutdown clearing up homeless session... 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 10 client.4343.objecter successfully canceled tick 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7e80040e10 legacy=0x7f7e800432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 10 monclient: shutdown 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 >> v1:172.21.15.82:6791/0 conn(0x7f7e9413fcd0 legacy=0x7f7e940a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 shutdown_connections 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 >> 172.21.15.1:0/3319189926 conn(0x7f7e940a8ef0 msgr2=0x7f7e94136d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 shutdown_connections 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 -- 172.21.15.1:0/3319189926 wait complete. 2024-08-19T09:50:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7f7e99e48640 1 librados: shutdown 2024-08-19T09:50:20.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7fa398ff9640 1 -- 172.21.15.1:0/2526719260 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 203001719) 0x7fa394017150 con 0x7fa39c1c9ac0 2024-08-19T09:50:20.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7fa398ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T09:50:20.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.067+0000 7fa398ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:51539607556 2024-08-19T09:50:20.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 10 librados: watch_flush enter 2024-08-19T09:50:20.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 10 librados: watch_flush exit 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 20 client.4336.objecter shutdown clearing up homeless session... 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 10 client.4336.objecter successfully canceled tick 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa388041550 legacy=0x7fa388043a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 10 monclient: shutdown 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 >> v1:172.21.15.82:6792/0 conn(0x7fa39c1c9ac0 legacy=0x7fa39c1d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 shutdown_connections 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 >> 172.21.15.1:0/2526719260 conn(0x7fa39c068650 msgr2=0x7fa39c09fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 shutdown_connections 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 -- 172.21.15.1:0/2526719260 wait complete. 2024-08-19T09:50:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:20.069+0000 7fa3a24e5640 1 librados: shutdown 2024-08-19T09:50:20.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-08-19T09:50:20.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.5 2024-08-19T09:50:20.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-19T09:50:20.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-19T09:50:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.017+0000 7f247f3f7640 10 monclient: get_monmap_and_config 2024-08-19T09:50:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.017+0000 7f247f3f7640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.017+0000 7f247f3f7640 10 monclient: monmap: 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:21.018475+0000 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:21.018475+0000 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.020 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 1 Processor -- start 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 1 -- start start 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient: init 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient: _add_conns ranks=[3,5,2,7,0,6,8,4,1] 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient(hunting): picked mon.noname-g con 0x7f24780a4f80 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient(hunting): picked mon.noname-h con 0x7f24780a4650 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient(hunting): picked mon.noname-b con 0x7f247813fcd0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24780a4a60 con 0x7f247813fcd0 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2478069270 con 0x7f24780a4f80 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24780a3c70 con 0x7f24780a4650 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.018+0000 7f247f3f7640 10 monclient(hunting): authenticate will time out at 677.094942s 2024-08-19T09:50:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.019+0000 7f247d994640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f247813fcd0 0x7f24781420c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52774/0 (socket says 172.21.15.1:52774) 2024-08-19T09:50:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.019+0000 7f247d994640 1 -- 172.21.15.1:0/3854023510 learned_addr learned my addr 172.21.15.1:0/3854023510 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.021+0000 7f247d994640 1 -- 172.21.15.1:0/3854023510 >> v1:172.21.15.1:6790/0 conn(0x7f247813fcd0 legacy=0x7f24781420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.021+0000 7f247d193640 1 -- 172.21.15.1:0/3854023510 >> v1:172.21.15.82:6790/0 conn(0x7f24780a4f80 legacy=0x7f24780a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.021+0000 7f247c992640 1 -- 172.21.15.1:0/3854023510 >> v1:172.21.15.82:6791/0 conn(0x7f24780a4650 legacy=0x7f247813e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-19T09:50:21.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.054+0000 7fccaee3f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:21.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.054+0000 7fccaee3f640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.054+0000 7fccaee3f640 10 monclient: monmap: 2024-08-19T09:50:21.056 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.056 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.056 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:21.055558+0000 2024-08-19T09:50:21.056 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:21.055558+0000 2024-08-19T09:50:21.056 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.057 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 1 Processor -- start 2024-08-19T09:50:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 1 -- start start 2024-08-19T09:50:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient: init 2024-08-19T09:50:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient: _add_conns ranks=[8,5,1,2,4,3,7,0,6] 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient(hunting): picked mon.noname-e con 0x7fcca80a4f80 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient(hunting): picked mon.noname-h con 0x7fcca80a4650 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient(hunting): picked mon.noname-f con 0x7fcca813fcd0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca80a4a60 con 0x7fcca813fcd0 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca8069270 con 0x7fcca80a4650 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca80a3c70 con 0x7fcca80a4f80 2024-08-19T09:50:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.055+0000 7fccaee3f640 10 monclient(hunting): authenticate will time out at 677.131938s 2024-08-19T09:50:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.056+0000 7fccacbdb640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcca80a4f80 0x7fcca80a5360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44696/0 (socket says 172.21.15.1:44696) 2024-08-19T09:50:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.056+0000 7fccacbdb640 1 -- 172.21.15.1:0/3346659304 learned_addr learned my addr 172.21.15.1:0/3346659304 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.058+0000 7fccacbdb640 1 -- 172.21.15.1:0/3346659304 >> v1:172.21.15.1:6793/0 conn(0x7fcca80a4f80 legacy=0x7fcca80a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.058+0000 7fcca7fff640 1 -- 172.21.15.1:0/3346659304 >> v1:172.21.15.82:6791/0 conn(0x7fcca80a4650 legacy=0x7fcca813e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.058+0000 7fccad3dc640 1 -- 172.21.15.1:0/3346659304 >> v1:172.21.15.82:6789/0 conn(0x7fcca813fcd0 legacy=0x7fcca81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-19T09:50:21.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-19T09:50:21.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.132+0000 7f9c25c94640 10 monclient: get_monmap_and_config 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.132+0000 7f9c25c94640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.132+0000 7f9c25c94640 10 monclient: monmap: 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:21.133259+0000 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:21.133259+0000 2024-08-19T09:50:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.134 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 1 Processor -- start 2024-08-19T09:50:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 1 -- start start 2024-08-19T09:50:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient: init 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient: _add_conns ranks=[8,3,5,2,0,6,7,1,4] 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient(hunting): picked mon.noname-e con 0x7f9c20068650 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient(hunting): picked mon.noname-g con 0x7f9c2009ec40 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient(hunting): picked mon.noname-h con 0x7f9c20069140 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c200a7e90 con 0x7f9c2009ec40 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2009dca0 con 0x7f9c20069140 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2009df80 con 0x7f9c20068650 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c25c94640 10 monclient(hunting): authenticate will time out at 677.209864s 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c1f7fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f9c20068650 0x7f9c20068a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44698/0 (socket says 172.21.15.1:44698) 2024-08-19T09:50:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.133+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1683585196 learned_addr learned my addr 172.21.15.1:0/1683585196 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.136+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1683585196 >> v1:172.21.15.1:6793/0 conn(0x7f9c20068650 legacy=0x7f9c20068a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.136+0000 7f9c1effd640 1 -- 172.21.15.1:0/1683585196 >> v1:172.21.15.82:6790/0 conn(0x7f9c2009ec40 legacy=0x7f9c2013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.136+0000 7f9c1ffff640 1 -- 172.21.15.1:0/1683585196 >> v1:172.21.15.82:6791/0 conn(0x7f9c20069140 legacy=0x7f9c20141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.169+0000 7fe08810d640 10 monclient: get_monmap_and_config 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.169+0000 7fe08810d640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.169+0000 7fe08810d640 10 monclient: monmap: 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:21.170841+0000 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:21.170841+0000 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.172 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.170+0000 7fe08810d640 1 Processor -- start 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.170+0000 7fe08810d640 1 -- start start 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.170+0000 7fe08810d640 10 monclient: init 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient: _add_conns ranks=[1,8,6,4,7,2,5,3,0] 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient(hunting): picked mon.noname-f con 0x7fe0800a4f80 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient(hunting): picked mon.noname-e con 0x7fe0800a4650 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient(hunting): picked mon.noname-d con 0x7fe08013fcd0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0800a4a60 con 0x7fe0800a4f80 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe080069270 con 0x7fe08013fcd0 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0800a3c70 con 0x7fe0800a4650 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe08810d640 10 monclient(hunting): authenticate will time out at 677.247270s 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe0866aa640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe08013fcd0 0x7fe0801420c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52876/0 (socket says 172.21.15.1:52876) 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe0866aa640 1 -- 172.21.15.1:0/1174645074 learned_addr learned my addr 172.21.15.1:0/1174645074 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.171+0000 7fe0856a8640 1 --1- 172.21.15.1:0/1174645074 >> v1:172.21.15.1:6793/0 conn(0x7fe0800a4650 0x7fe08013e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44702/0 (socket says 172.21.15.1:44702) 2024-08-19T09:50:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.173+0000 7fe0866aa640 1 -- 172.21.15.1:0/1174645074 >> v1:172.21.15.1:6792/0 conn(0x7fe08013fcd0 legacy=0x7fe0801420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.173+0000 7fe0856a8640 1 -- 172.21.15.1:0/1174645074 >> v1:172.21.15.1:6793/0 conn(0x7fe0800a4650 legacy=0x7fe08013e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.173+0000 7fe085ea9640 1 -- 172.21.15.1:0/1174645074 >> v1:172.21.15.82:6789/0 conn(0x7fe0800a4f80 legacy=0x7fe0800a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7fed1c581640 10 monclient: get_monmap_and_config 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7fed1c581640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7fed1c581640 10 monclient: monmap: 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:21.213252+0000 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:21.213252+0000 2024-08-19T09:50:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.214 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7f0fec7e9640 10 monclient: get_monmap_and_config 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7f0fec7e9640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7f0fec7e9640 10 monclient: monmap: 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:21.213260+0000 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:21.213260+0000 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7f0fec7e9640 1 Processor -- start 2024-08-19T09:50:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7fed1c581640 1 Processor -- start 2024-08-19T09:50:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7f0fec7e9640 1 -- start start 2024-08-19T09:50:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7f0fec7e9640 10 monclient: init 2024-08-19T09:50:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.212+0000 7fed1c581640 1 -- start start 2024-08-19T09:50:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient: init 2024-08-19T09:50:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient: _add_conns ranks=[3,7,8,0,5,6,2,4,1] 2024-08-19T09:50:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient(hunting): picked mon.noname-g con 0x7f0fe40a6720 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient: _add_conns ranks=[0,1,6,7,3,2,5,4,8] 2024-08-19T09:50:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient(hunting): picked mon.noname-i con 0x7f0fe40a7210 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient(hunting): picked mon.noname-a con 0x7fed140a4f80 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient(hunting): picked mon.noname-e con 0x7f0fe40a4500 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe406ed50 con 0x7f0fe40a6720 2024-08-19T09:50:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient(hunting): picked mon.noname-f con 0x7fed140a4650 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe40a3b20 con 0x7f0fe40a7210 2024-08-19T09:50:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient(hunting): picked mon.noname-d con 0x7fed1413fcd0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe406f190 con 0x7f0fe40a4500 2024-08-19T09:50:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fec7e9640 10 monclient(hunting): authenticate will time out at 677.289744s 2024-08-19T09:50:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed140a4a60 con 0x7fed140a4f80 2024-08-19T09:50:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fead86640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f0fe40a4500 0x7f0fe4141ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44708/0 (socket says 172.21.15.1:44708) 2024-08-19T09:50:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7f0fead86640 1 -- 172.21.15.1:0/2570912371 learned_addr learned my addr 172.21.15.1:0/2570912371 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1a31d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fed140a4f80 0x7fed140a5360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37186/0 (socket says 172.21.15.1:37186) 2024-08-19T09:50:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1a31d640 1 -- 172.21.15.1:0/3010330822 learned_addr learned my addr 172.21.15.1:0/3010330822 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed14069270 con 0x7fed140a4650 2024-08-19T09:50:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed140a3c70 con 0x7fed1413fcd0 2024-08-19T09:50:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.213+0000 7fed1c581640 10 monclient(hunting): authenticate will time out at 677.289953s 2024-08-19T09:50:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7f0fead86640 1 -- 172.21.15.1:0/2570912371 >> v1:172.21.15.1:6793/0 conn(0x7f0fe40a4500 legacy=0x7f0fe4141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1a31d640 1 -- 172.21.15.1:0/3010330822 >> v1:172.21.15.1:6789/0 conn(0x7fed140a4f80 legacy=0x7fed140a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1ab1e640 1 -- 172.21.15.1:0/3010330822 >> v1:172.21.15.1:6792/0 conn(0x7fed1413fcd0 legacy=0x7fed141420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7f0fe9d84640 1 -- 172.21.15.1:0/2570912371 >> v1:172.21.15.82:6792/0 conn(0x7f0fe40a7210 legacy=0x7f0fe413e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7f0fea585640 1 -- 172.21.15.1:0/2570912371 >> v1:172.21.15.82:6790/0 conn(0x7f0fe40a6720 legacy=0x7f0fe40a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed19b1c640 1 -- 172.21.15.1:0/3010330822 >> v1:172.21.15.82:6789/0 conn(0x7fed140a4650 legacy=0x7fed1413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865092770 0 0) 0x7fed140a4a60 con 0x7fed140a4f80 2024-08-19T09:50:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient(hunting): my global_id is 4347 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed00003520 con 0x7fed140a4f80 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4071938439 0 0) 0x7fed00003520 con 0x7fed140a4f80 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 >> v1:172.21.15.1:6792/0 conn(0x7fed1413fcd0 legacy=0x7fed141420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 >> v1:172.21.15.82:6789/0 conn(0x7fed140a4650 legacy=0x7fed1413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 monclient: found mon.noname-a 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed140a3420 con 0x7fed140a4f80 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed10001ae0 con 0x7fed140a4f80 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1c581640 5 monclient: authenticate success, global_id 4347 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1c581640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: dump: 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.231 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1c581640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed100048a0 con 0x7fed140a4f80 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1c581640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1c581640 4 set_mon_vals no callback set 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 1 -- 172.21.15.1:0/3010330822 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed10004dd0 con 0x7fed140a4f80 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.216+0000 7fed1931b640 10 monclient: dump: 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 10 monclient: shutdown 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 >> v1:172.21.15.1:6789/0 conn(0x7fed140a4f80 legacy=0x7fed140a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 shutdown_connections 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 >> 172.21.15.1:0/3010330822 conn(0x7fed140a8ef0 msgr2=0x7fed14068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 shutdown_connections 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.217+0000 7fed1c581640 1 -- 172.21.15.1:0/3010330822 wait complete. 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient: monmap: 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 librados: starting msgr at 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 librados: starting objecter 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 Processor -- start 2024-08-19T09:50:21.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 -- start start 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 librados: setting wanted keys 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 librados: calling monclient init 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient: init 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient: _add_conns ranks=[5,3,6,8,1,0,7,2,4] 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient(hunting): picked mon.noname-h con 0x7fed140a4650 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient(hunting): picked mon.noname-g con 0x7fed140a4f80 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient(hunting): picked mon.noname-d con 0x7fed1413fcd0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed140a1d50 con 0x7fed140a4f80 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed140a2630 con 0x7fed140a4650 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed14139810 con 0x7fed1413fcd0 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.218+0000 7fed1c581640 10 monclient(hunting): authenticate will time out at 677.294964s 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.219+0000 7fed1ab1e640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fed1413fcd0 0x7fed141d67b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52890/0 (socket says 172.21.15.1:52890) 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.219+0000 7fed1ab1e640 1 -- 172.21.15.1:0/1565907259 learned_addr learned my addr 172.21.15.1:0/1565907259 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed1ab1e640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.1:6792/0 conn(0x7fed1413fcd0 legacy=0x7fed141d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed0affd640 10 client.?.objecter ms_handle_connect 0x7fed1413fcd0 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed0affd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed0affd640 10 client.?.objecter ms_handle_connect 0x7fed140a4f80 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed0affd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed19b1c640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.82:6790/0 conn(0x7fed140a4f80 legacy=0x7fed140a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed0affd640 10 client.?.objecter ms_handle_connect 0x7fed140a4650 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed0affd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.221+0000 7fed1a31d640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.82:6791/0 conn(0x7fed140a4650 legacy=0x7fed140a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415321119 0 0) 0x7f24780a4a60 con 0x7f247813fcd0 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): my global_id is 4331 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2464003520 con 0x7f247813fcd0 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993382923 0 0) 0x7f2478069270 con 0x7f24780a4f80 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): my global_id is 4332 2024-08-19T09:50:21.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24780a4a60 con 0x7f24780a4f80 2024-08-19T09:50:21.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836412119 0 0) 0x7f9c200a7e90 con 0x7f9c2009ec40 2024-08-19T09:50:21.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): my global_id is 4341 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c08003520 con 0x7f9c2009ec40 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2792956277 0 0) 0x7f9c2009df80 con 0x7f9c20068650 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): my global_id is 4346 2024-08-19T09:50:21.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c200a7e90 con 0x7f9c20068650 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791171590 0 0) 0x7fe080069270 con 0x7fe08013fcd0 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): my global_id is 4299 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe06c003520 con 0x7fe08013fcd0 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197191406 0 0) 0x7fe0800a4a60 con 0x7fe0800a4f80 2024-08-19T09:50:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 10 monclient(hunting): my global_id is 4312 2024-08-19T09:50:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe080069270 con 0x7fe0800a4f80 2024-08-19T09:50:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305662805 0 0) 0x7f9c2009dca0 con 0x7f9c20069140 2024-08-19T09:50:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 10 monclient(hunting): my global_id is 4379 2024-08-19T09:50:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c2009df80 con 0x7f9c20069140 2024-08-19T09:50:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714803611 0 0) 0x7fed140a1d50 con 0x7fed140a4f80 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fed0affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fed0affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fed0affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fed0affd640 10 monclient(hunting): my global_id is 4359 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf8004ee0 con 0x7fed140a4f80 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441229222 0 0) 0x7f24780a3c70 con 0x7f24780a4650 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient(hunting): my global_id is 4361 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2464003a10 con 0x7f24780a4650 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3567235805 0 0) 0x7f2464003520 con 0x7f247813fcd0 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 >> v1:172.21.15.82:6791/0 conn(0x7f24780a4650 legacy=0x7f247813e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 >> v1:172.21.15.82:6790/0 conn(0x7f24780a4f80 legacy=0x7f24780a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 monclient: found mon.noname-b 2024-08-19T09:50:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24780a3420 con 0x7f247813fcd0 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2470001f20 con 0x7f247813fcd0 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: dump: 2024-08-19T09:50:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f247f3f7640 5 monclient: authenticate success, global_id 4331 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f247f3f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2470004c20 con 0x7f247813fcd0 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f246ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f246ffff640 1 -- 172.21.15.1:0/3854023510 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2470005210 con 0x7f247813fcd0 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f246ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f246ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f246ffff640 10 monclient: dump: 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.612 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f247f3f7640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f247f3f7640 4 set_mon_vals no callback set 2024-08-19T09:50:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283282593 0 0) 0x7fcca80a4a60 con 0x7fcca813fcd0 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): my global_id is 4303 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc98003520 con 0x7fcca813fcd0 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591770986 0 0) 0x7fcca80a3c70 con 0x7fcca80a4f80 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): my global_id is 4337 2024-08-19T09:50:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcca80a4a60 con 0x7fcca80a4f80 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2521043962 0 0) 0x7fcca8069270 con 0x7fcca80a4650 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 10 monclient(hunting): my global_id is 4370 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc98003a10 con 0x7fcca80a4650 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1579408434 0 0) 0x7fcca80a4a60 con 0x7fcca80a4f80 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 >> v1:172.21.15.82:6791/0 conn(0x7fcca80a4650 legacy=0x7fcca813e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 >> v1:172.21.15.82:6789/0 conn(0x7fcca813fcd0 legacy=0x7fcca81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 monclient: found mon.noname-e 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca80a3420 con 0x7fcca80a4f80 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc9c001a40 con 0x7fcca80a4f80 2024-08-19T09:50:21.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2541335745 0 0) 0x7f9c200a7e90 con 0x7f9c20068650 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 >> v1:172.21.15.82:6791/0 conn(0x7f9c20069140 legacy=0x7f9c20141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 >> v1:172.21.15.82:6790/0 conn(0x7f9c2009ec40 legacy=0x7f9c2013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 monclient: found mon.noname-e 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c20077710 con 0x7f9c20068650 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c10001ac0 con 0x7f9c20068650 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: dump: 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.618 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.619 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1846971297 0 0) 0x7fe06c003520 con 0x7fe08013fcd0 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 >> v1:172.21.15.1:6793/0 conn(0x7fe0800a4650 legacy=0x7fe08013e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 >> v1:172.21.15.82:6789/0 conn(0x7fe0800a4f80 legacy=0x7fe0800a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 1 monclient: found mon.noname-d 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0800a3420 con 0x7fe08013fcd0 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe078001f20 con 0x7fe08013fcd0 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe084ea7640 10 monclient: dump: 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe08810d640 5 monclient: authenticate success, global_id 4299 2024-08-19T09:50:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fe08810d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009419016 0 0) 0x7f0fe40a3b20 con 0x7f0fe40a7210 2024-08-19T09:50:21.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): my global_id is 4354 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd0003520 con 0x7f0fe40a7210 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 658273483 0 0) 0x7f0fe406f190 con 0x7f0fe40a4500 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 10 monclient(hunting): my global_id is 4364 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.599+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fe40a3b20 con 0x7f0fe40a4500 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788881780 0 0) 0x7f0fe406ed50 con 0x7f0fe40a6720 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient(hunting): my global_id is 4350 2024-08-19T09:50:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fe406f190 con 0x7f0fe40a6720 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: dump: 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fccaee3f640 5 monclient: authenticate success, global_id 4337 2024-08-19T09:50:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fccaee3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c25c94640 5 monclient: authenticate success, global_id 4346 2024-08-19T09:50:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c25c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3640714188 0 0) 0x7f0fe40a3b20 con 0x7f0fe40a4500 2024-08-19T09:50:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 >> v1:172.21.15.82:6792/0 conn(0x7f0fe40a7210 legacy=0x7f0fe413e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc9c0048a0 con 0x7fcca80a4f80 2024-08-19T09:50:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 >> v1:172.21.15.82:6790/0 conn(0x7f0fe40a6720 legacy=0x7f0fe40a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 monclient: found mon.noname-e 2024-08-19T09:50:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 1 -- 172.21.15.1:0/3346659304 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc9c001a40 con 0x7fcca80a4f80 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fcca77fe640 10 monclient: dump: 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.631 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.632 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.632 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.632 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:21.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe40a9260 con 0x7f0fe40a4500 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0fdc001dc0 con 0x7f0fe40a4500 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fe9583640 10 monclient: dump: 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.633 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fec7e9640 5 monclient: authenticate success, global_id 4364 2024-08-19T09:50:21.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f0fec7e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333040975 0 0) 0x7fed14139810 con 0x7fed1413fcd0 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fed0affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fed0affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fed0affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fed0affd640 10 monclient(hunting): my global_id is 4308 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed140a1d50 con 0x7fed1413fcd0 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f247f3f7640 10 monclient: shutdown 2024-08-19T09:50:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f247f3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f247f3f7640 1 -- 172.21.15.1:0/3854023510 >> v1:172.21.15.1:6790/0 conn(0x7f247813fcd0 legacy=0x7f24781420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f247f3f7640 1 -- 172.21.15.1:0/3854023510 shutdown_connections 2024-08-19T09:50:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f247f3f7640 1 -- 172.21.15.1:0/3854023510 >> 172.21.15.1:0/3854023510 conn(0x7f24780a8ef0 msgr2=0x7f2478068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f247f3f7640 1 -- 172.21.15.1:0/3854023510 shutdown_connections 2024-08-19T09:50:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f247f3f7640 1 -- 172.21.15.1:0/3854023510 wait complete. 2024-08-19T09:50:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fccaee3f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7fccaee3f640 4 set_mon_vals no callback set 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c100048a0 con 0x7f9c20068650 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 1 -- 172.21.15.1:0/1683585196 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c10004e10 con 0x7f9c20068650 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.600+0000 7f9c1e7fc640 10 monclient: dump: 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.638 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f9c25c94640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f9c25c94640 4 set_mon_vals no callback set 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe078004c20 con 0x7fe08013fcd0 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe084ea7640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe084ea7640 1 -- 172.21.15.1:0/1174645074 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe078005110 con 0x7fe08013fcd0 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe084ea7640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe084ea7640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe084ea7640 10 monclient: dump: 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe08810d640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fe08810d640 4 set_mon_vals no callback set 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fdc004c20 con 0x7f0fe40a4500 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fe9583640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fe9583640 1 -- 172.21.15.1:0/2570912371 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0fdc005110 con 0x7f0fe40a4500 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fec7e9640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fec7e9640 4 set_mon_vals no callback set 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fe9583640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fe9583640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f0fe9583640 10 monclient: dump: 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.642 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031104160 0 0) 0x7fed140a2630 con 0x7fed140a4650 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fed0affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fed0affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fed0affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fed0affd640 10 monclient(hunting): my global_id is 4388 2024-08-19T09:50:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed14139810 con 0x7fed140a4650 2024-08-19T09:50:21.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fccaee3f640 10 monclient: shutdown 2024-08-19T09:50:21.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fccaee3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7fccaee3f640 1 -- 172.21.15.1:0/3346659304 >> v1:172.21.15.1:6793/0 conn(0x7fcca80a4f80 legacy=0x7fcca80a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.601+0000 7f9c25c94640 10 monclient: shutdown 2024-08-19T09:50:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f9c25c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f9c25c94640 1 -- 172.21.15.1:0/1683585196 >> v1:172.21.15.1:6793/0 conn(0x7f9c20068650 legacy=0x7f9c20068a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f9c25c94640 1 -- 172.21.15.1:0/1683585196 shutdown_connections 2024-08-19T09:50:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f9c25c94640 1 -- 172.21.15.1:0/1683585196 >> 172.21.15.1:0/1683585196 conn(0x7f9c200a80b0 msgr2=0x7f9c200a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 10 monclient: shutdown 2024-08-19T09:50:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 1 -- 172.21.15.1:0/1174645074 >> v1:172.21.15.1:6792/0 conn(0x7fe08013fcd0 legacy=0x7fe0801420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f9c25c94640 1 -- 172.21.15.1:0/1683585196 shutdown_connections 2024-08-19T09:50:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 10 monclient: shutdown 2024-08-19T09:50:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f9c25c94640 1 -- 172.21.15.1:0/1683585196 wait complete. 2024-08-19T09:50:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 1 -- 172.21.15.1:0/1174645074 shutdown_connections 2024-08-19T09:50:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 1 -- 172.21.15.1:0/1174645074 >> 172.21.15.1:0/1174645074 conn(0x7fe0800a8ef0 msgr2=0x7fe080068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 1 -- 172.21.15.1:0/2570912371 >> v1:172.21.15.1:6793/0 conn(0x7f0fe40a4500 legacy=0x7f0fe4141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fccaee3f640 1 -- 172.21.15.1:0/3346659304 shutdown_connections 2024-08-19T09:50:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fccaee3f640 1 -- 172.21.15.1:0/3346659304 >> 172.21.15.1:0/3346659304 conn(0x7fcca80a8ef0 msgr2=0x7fcca8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 1 -- 172.21.15.1:0/1174645074 shutdown_connections 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fe08810d640 1 -- 172.21.15.1:0/1174645074 wait complete. 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 10 monclient: monmap: 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 1 -- 172.21.15.1:0/2570912371 shutdown_connections 2024-08-19T09:50:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 1 -- 172.21.15.1:0/2570912371 >> 172.21.15.1:0/2570912371 conn(0x7f0fe40ac440 msgr2=0x7f0fe40a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 1 -- 172.21.15.1:0/2570912371 shutdown_connections 2024-08-19T09:50:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fccaee3f640 1 -- 172.21.15.1:0/3346659304 shutdown_connections 2024-08-19T09:50:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f0fec7e9640 1 -- 172.21.15.1:0/2570912371 wait complete. 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7fccaee3f640 1 -- 172.21.15.1:0/3346659304 wait complete. 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 1 librados: starting msgr at 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 1 librados: starting objecter 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 1 Processor -- start 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 1 -- start start 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 1 librados: setting wanted keys 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 1 librados: calling monclient init 2024-08-19T09:50:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.602+0000 7f247f3f7640 10 monclient: init 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f9c25c94640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f9c25c94640 10 monclient: monmap: 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f9c25c94640 1 librados: starting msgr at 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 10 monclient: monmap: 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f0fec7e9640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f0fec7e9640 10 monclient: monmap: 2024-08-19T09:50:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.658 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f0fec7e9640 1 librados: starting msgr at 2024-08-19T09:50:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f0fec7e9640 1 librados: starting objecter 2024-08-19T09:50:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f9c25c94640 1 librados: starting objecter 2024-08-19T09:50:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 1 librados: starting msgr at 2024-08-19T09:50:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 1 librados: starting objecter 2024-08-19T09:50:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 1 Processor -- start 2024-08-19T09:50:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f0fec7e9640 1 Processor -- start 2024-08-19T09:50:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f9c25c94640 1 Processor -- start 2024-08-19T09:50:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 1 -- start start 2024-08-19T09:50:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 1 librados: setting wanted keys 2024-08-19T09:50:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 1 librados: calling monclient init 2024-08-19T09:50:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7fe08810d640 10 monclient: init 2024-08-19T09:50:21.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f0fec7e9640 1 -- start start 2024-08-19T09:50:21.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f9c25c94640 1 -- start start 2024-08-19T09:50:21.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f9c25c94640 1 librados: setting wanted keys 2024-08-19T09:50:21.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f9c25c94640 1 librados: calling monclient init 2024-08-19T09:50:21.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f9c25c94640 10 monclient: init 2024-08-19T09:50:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f0fec7e9640 1 librados: setting wanted keys 2024-08-19T09:50:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f0fec7e9640 1 librados: calling monclient init 2024-08-19T09:50:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f0fec7e9640 10 monclient: init 2024-08-19T09:50:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f247f3f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f247f3f7640 10 monclient: _add_conns ranks=[5,7,4,6,1,8,2,3,0] 2024-08-19T09:50:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.603+0000 7f247f3f7640 10 monclient(hunting): picked mon.noname-h con 0x7f24780a4650 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7fe08810d640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 10 monclient(hunting): picked mon.noname-i con 0x7f24780a4f80 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 10 monclient(hunting): picked mon.noname-c con 0x7f247813fcd0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247813b4f0 con 0x7f247813fcd0 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247806d430 con 0x7f24780a4650 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247813d850 con 0x7f24780a4f80 2024-08-19T09:50:21.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f247f3f7640 10 monclient(hunting): authenticate will time out at 677.680745s 2024-08-19T09:50:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7f9c25c94640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.604+0000 7fe08810d640 10 monclient: _add_conns ranks=[3,2,7,5,4,8,6,0,1] 2024-08-19T09:50:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe08810d640 10 monclient(hunting): picked mon.noname-g con 0x7fe0800a4650 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient: _add_conns ranks=[0,1,2,7,4,6,8,5,3] 2024-08-19T09:50:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f247d193640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f24780a4650 0x7f2478138910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:34788/0 (socket says 172.21.15.1:34788) 2024-08-19T09:50:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f247d193640 1 -- 172.21.15.1:0/4195150106 learned_addr learned my addr 172.21.15.1:0/4195150106 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe40a4500 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient(hunting): picked mon.noname-f con 0x7f0fe40a6720 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe40a7210 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe4125740 con 0x7f0fe40a4500 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe4123190 con 0x7f0fe40a6720 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe4125a20 con 0x7f0fe40a7210 2024-08-19T09:50:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 10 monclient: _add_conns ranks=[4,8,3,5,2,1,0,7,6] 2024-08-19T09:50:21.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fec7e9640 10 monclient(hunting): authenticate will time out at 677.681335s 2024-08-19T09:50:21.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fead86640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0fe40a7210 0x7f0fe41da980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52790/0 (socket says 172.21.15.1:52790) 2024-08-19T09:50:21.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f0fead86640 1 -- 172.21.15.1:0/3964635550 learned_addr learned my addr 172.21.15.1:0/3964635550 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 10 monclient(hunting): picked mon.noname-c con 0x7f9c20068650 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:21.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c1f7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f9c20068650 0x7f9c201ca8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57428/0 (socket says 172.21.15.1:57428) 2024-08-19T09:50:21.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1478202240 learned_addr learned my addr 172.21.15.1:0/1478202240 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe085ea9640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fe0800a4650 0x7fe0800726b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48480/0 (socket says 172.21.15.1:48480) 2024-08-19T09:50:21.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe0856a8640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fe0800a4f80 0x7fe0801d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52780/0 (socket says 172.21.15.1:52780) 2024-08-19T09:50:21.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 10 monclient(hunting): picked mon.noname-e con 0x7f9c20069140 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:21.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 10 monclient(hunting): picked mon.noname-g con 0x7f9c2009ec40 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:21.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2013d5a0 con 0x7f9c2009ec40 2024-08-19T09:50:21.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe085ea9640 1 -- 172.21.15.1:0/152356919 learned_addr learned my addr 172.21.15.1:0/152356919 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2013e9d0 con 0x7f9c20068650 2024-08-19T09:50:21.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c200a5370 con 0x7f9c20069140 2024-08-19T09:50:21.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7f9c25c94640 10 monclient(hunting): authenticate will time out at 677.681826s 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe08810d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0800a4f80 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe08810d640 10 monclient(hunting): picked mon.noname-i con 0x7fe08013fcd0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.605+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0800a1fd0 con 0x7fe0800a4f80 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe08006feb0 con 0x7fe0800a4650 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0800a28b0 con 0x7fe08013fcd0 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fe08810d640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fe08810d640 10 monclient(hunting): authenticate will time out at 677.682095s 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 10 monclient: build_initial_monmap 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 10 monclient: monmap: 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:21.672 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 1 librados: starting msgr at 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 1 librados: starting objecter 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 1 Processor -- start 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 1 -- start start 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 1 librados: setting wanted keys 2024-08-19T09:50:21.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 1 librados: calling monclient init 2024-08-19T09:50:21.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.606+0000 7fccaee3f640 10 monclient: init 2024-08-19T09:50:21.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f247d193640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.82:6791/0 conn(0x7f24780a4650 legacy=0x7f2478138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f246dffb640 10 client.?.objecter ms_handle_connect 0x7f24780a4650 2024-08-19T09:50:21.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f246dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fead86640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.1:6790/0 conn(0x7f0fe40a7210 legacy=0x7f0fe41da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fdaffd640 10 client.?.objecter ms_handle_connect 0x7f0fe40a7210 2024-08-19T09:50:21.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fdaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient: _add_conns ranks=[4,0,2,3,6,7,8,1,5] 2024-08-19T09:50:21.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient(hunting): picked mon.noname-c con 0x7fcca80a4650 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:21.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient(hunting): picked mon.noname-a con 0x7fcca80a4f80 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f247d994640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.1:6791/0 conn(0x7f247813fcd0 legacy=0x7f24781d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f247c992640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.82:6792/0 conn(0x7f24780a4f80 legacy=0x7f24781cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f246dffb640 10 client.?.objecter ms_handle_connect 0x7f247813fcd0 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f246dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f246dffb640 10 client.?.objecter ms_handle_connect 0x7f24780a4f80 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f246dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient(hunting): picked mon.noname-b con 0x7fcca813fcd0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca806fc20 con 0x7fcca80a4f80 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca80a22e0 con 0x7fcca813fcd0 2024-08-19T09:50:21.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca8135660 con 0x7fcca80a4650 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccaee3f640 10 monclient(hunting): authenticate will time out at 677.683792s 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fe9d84640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6789/0 conn(0x7f0fe40a6720 legacy=0x7f0fe4122a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fdaffd640 10 client.?.objecter ms_handle_connect 0x7f0fe40a6720 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fdaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccacbdb640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fcca80a4650 0x7fcca8072720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57438/0 (socket says 172.21.15.1:57438) 2024-08-19T09:50:21.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7fccacbdb640 1 -- 172.21.15.1:0/196222874 learned_addr learned my addr 172.21.15.1:0/196222874 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:21.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fe9d84640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6789/0 conn(0x7f0fe40a6720 legacy=0x7f0fe4122a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T09:50:21.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fe9d84640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6789/0 conn(0x7f0fe40a6720 legacy=0x7f0fe4122a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:50:21.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fe9d84640 1 --1- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6789/0 conn(0x7f0fe40a6720 0x7f0fe4122a80 :-1 s=OPENED pgs=69 cs=1 l=1).handle_message read tag failed 2024-08-19T09:50:21.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fe9d84640 1 --1- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6789/0 conn(0x7f0fe40a6720 0x7f0fe4122a80 :-1 s=OPENED pgs=69 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:50:21.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.1:6791/0 conn(0x7f9c20068650 legacy=0x7f9c201ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 client.?.objecter ms_handle_connect 0x7f9c20068650 2024-08-19T09:50:21.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.607+0000 7f0fdaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6789/0 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fe085ea9640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.82:6790/0 conn(0x7fe0800a4650 legacy=0x7fe0800726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe0800a4650 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe0800a4f80 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fe0856a8640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.1:6790/0 conn(0x7fe0800a4f80 legacy=0x7fe0801d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c1ffff640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.82:6790/0 conn(0x7f9c2009ec40 legacy=0x7f9c201d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 client.?.objecter ms_handle_connect 0x7f9c2009ec40 2024-08-19T09:50:21.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814028714 0 0) 0x7f9c2013e9d0 con 0x7f9c20068650 2024-08-19T09:50:21.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c1effd640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.1:6793/0 conn(0x7f9c20069140 legacy=0x7f9c201d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): my global_id is 4342 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c0c004eb0 con 0x7f9c20068650 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 client.?.objecter ms_handle_connect 0x7f9c20069140 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 206691694 0 0) 0x7fecf8004ee0 con 0x7fed140a4f80 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed140a2630 con 0x7fed140a4f80 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed04003610 con 0x7fed140a4f80 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): dump: 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.683 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.684 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:21.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1220529723 0 0) 0x7f9c0c004eb0 con 0x7f9c20068650 2024-08-19T09:50:21.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1245842052 0 0) 0x7fed14139810 con 0x7fed140a4650 2024-08-19T09:50:21.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecf8004ee0 con 0x7fed140a4650 2024-08-19T09:50:21.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c2013e9d0 con 0x7f9c20068650 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c10004d90 con 0x7f9c20068650 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed10004dd0 con 0x7fed140a4650 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): dump: 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.686 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): dump: 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.688 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:21.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2004304320 0 0) 0x7f9c2013e9d0 con 0x7f9c20068650 2024-08-19T09:50:21.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f0fdaffd640 10 client.?.objecter ms_handle_connect 0x7f0fe40a4500 2024-08-19T09:50:21.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f0fdaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f0fea585640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.1:6789/0 conn(0x7f0fe40a4500 legacy=0x7f0fe4122370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4275830152 0 0) 0x7fed140a1d50 con 0x7fed1413fcd0 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed14139810 con 0x7fed1413fcd0 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed0c002050 con 0x7fed1413fcd0 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.1:6793/0 conn(0x7f9c20069140 legacy=0x7f9c201d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.82:6790/0 conn(0x7f9c2009ec40 legacy=0x7f9c201d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 monclient: found mon.c 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c200a5710 con 0x7f9c20068650 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.608+0000 7fed0affd640 10 monclient(hunting): dump: 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.692 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.693 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.694 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 5 monclient: authenticate success, global_id 4342 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 10 monclient: _renew_subs 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c2013c670 con 0x7f9c20068650 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 10 monclient: _renew_subs 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c2013d1d0 con 0x7f9c20068650 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c25c94640 1 librados: init done 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c10001d10 con 0x7f9c20068650 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c10005740 con 0x7f9c20068650 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c1d7fa640 4 set_mon_vals no callback set 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.609+0000 7f9c07fff640 10 monclient: dump: 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fe0866aa640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.82:6792/0 conn(0x7fe08013fcd0 legacy=0x7fe0801dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe08013fcd0 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fcca57fa640 10 client.?.objecter ms_handle_connect 0x7fcca80a4650 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fcca57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fcca57fa640 10 client.?.objecter ms_handle_connect 0x7fcca813fcd0 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fcca57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fccad3dc640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6790/0 conn(0x7fcca813fcd0 legacy=0x7fcca81dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7fccacbdb640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6791/0 conn(0x7fcca80a4650 legacy=0x7fcca8072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932958288 0 0) 0x7f247813b4f0 con 0x7f247813fcd0 2024-08-19T09:50:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f246dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f246dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f246dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f246dffb640 10 monclient(hunting): my global_id is 4333 2024-08-19T09:50:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f245c004eb0 con 0x7f247813fcd0 2024-08-19T09:50:21.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c25c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c25c94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c25c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9c10011ba0 con 0x7f9c20068650 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c00005080 con 0x7f9c20068650 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f9c1004dba0 con 0x7f9c20068650 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c07fff640 10 client.4342.objecter ms_dispatch 0x7f9c20000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c07fff640 3 client.4342.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c07fff640 3 client.4342.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c07fff640 20 client.4342.objecter dump_active .. 0 homeless 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.610+0000 7f9c1effd640 10 monclient: get_auth_request con 0x7f9c0c040f30 auth_method 0 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 10 client.?.objecter ms_handle_connect 0x7fcca80a4f80 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:21.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca7fff640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6789/0 conn(0x7fcca80a4f80 legacy=0x7fcca81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458300834 0 0) 0x7f0fe4125740 con 0x7f0fe40a4500 2024-08-19T09:50:21.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): my global_id is 4356 2024-08-19T09:50:21.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc8004ee0 con 0x7f0fe40a4500 2024-08-19T09:50:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133945754 0 0) 0x7fcca806fc20 con 0x7fcca80a4f80 2024-08-19T09:50:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 10 monclient(hunting): my global_id is 4365 2024-08-19T09:50:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc8c004eb0 con 0x7fcca80a4f80 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1084963271 0 0) 0x7f0fc8004ee0 con 0x7f0fe40a4500 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fe4125740 con 0x7f0fe40a4500 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0fe0001c40 con 0x7f0fe40a4500 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): dump: 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.704 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.705 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.706 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.706 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.706 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.706 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.706 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.611+0000 7f0fdaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.613+0000 7f9c07fff640 10 client.4342.objecter ms_handle_connect 0x7f9c0c040f30 2024-08-19T09:50:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4030122049 0 0) 0x7f0fe4125740 con 0x7f0fe40a4500 2024-08-19T09:50:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.1:6790/0 conn(0x7f0fe40a7210 legacy=0x7f0fe41da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6789/0 conn(0x7f0fe40a6720 legacy=0x7f0fe4122a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 1 monclient: found mon.a 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe40a2410 con 0x7f0fe40a4500 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 570334586 0 0) 0x7fcc8c004eb0 con 0x7fcca80a4f80 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcca806fc20 con 0x7fcca80a4f80 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc94003610 con 0x7fcca80a4f80 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 10 monclient(hunting): dump: 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7fcca57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 5 monclient: authenticate success, global_id 4356 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 10 monclient: _renew_subs 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe4125460 con 0x7f0fe40a4500 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 10 monclient: _renew_subs 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0fe41244f0 con 0x7f0fe40a4500 2024-08-19T09:50:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.615+0000 7f0fec7e9640 1 librados: init done 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618261360 0 0) 0x7fcca80a22e0 con 0x7fcca813fcd0 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient(hunting): my global_id is 4358 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc8c004eb0 con 0x7fcca813fcd0 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2028038432 0 0) 0x7fcca806fc20 con 0x7fcca80a4f80 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6791/0 conn(0x7fcca80a4650 legacy=0x7fcca8072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6790/0 conn(0x7fcca813fcd0 legacy=0x7fcca81dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 monclient: found mon.a 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca806f940 con 0x7fcca80a4f80 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fcca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 5 monclient: authenticate success, global_id 4365 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 10 monclient: _renew_subs 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcca813f060 con 0x7fcca80a4f80 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 10 monclient: _renew_subs 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcca81dc580 con 0x7fcca80a4f80 2024-08-19T09:50:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fccaee3f640 1 librados: init done 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc94003ba0 con 0x7fcca80a4f80 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc94004110 con 0x7fcca80a4f80 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca57fa640 10 monclient: dump: 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.715 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fcca6ffd640 4 set_mon_vals no callback set 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223728269 0 0) 0x7fe0800a1fd0 con 0x7fe0800a4f80 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fe0767fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fe0767fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fe0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fe0767fc640 10 monclient(hunting): my global_id is 4349 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.616+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe064004eb0 con 0x7fe0800a4f80 2024-08-19T09:50:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fec7e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fec7e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fec7e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe4051b00 con 0x7f0fe40a4500 2024-08-19T09:50:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2419108565 0 0) 0x7fed14139810 con 0x7fed1413fcd0 2024-08-19T09:50:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.82:6791/0 conn(0x7fed140a4650 legacy=0x7fed140a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.82:6790/0 conn(0x7fed140a4f80 legacy=0x7fed140a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 1 monclient: found mon.d 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106899708 0 0) 0x7f247806d430 con 0x7f24780a4650 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f246dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f246dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f246dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f246dffb640 10 monclient(hunting): my global_id is 4397 2024-08-19T09:50:21.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f247813b4f0 con 0x7f24780a4650 2024-08-19T09:50:21.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:21.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1413a0b0 con 0x7fed1413fcd0 2024-08-19T09:50:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fccaee3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1124737790 0 0) 0x7fe0800a28b0 con 0x7fe08013fcd0 2024-08-19T09:50:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fe0003a00 con 0x7f0fe40a4500 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): my global_id is 4372 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0800a1fd0 con 0x7fe08013fcd0 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3020185441 0 0) 0x7fe08006feb0 con 0x7fe0800a4650 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 10 monclient(hunting): my global_id is 4368 2024-08-19T09:50:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0800a28b0 con 0x7fe0800a4650 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0fe0003f30 con 0x7f0fe40a4500 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 10 monclient: dump: 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdbfff640 4 set_mon_vals no callback set 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0fe0004150 con 0x7f0fe40a4500 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f0fe004daf0 con 0x7f0fe40a4500 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7f0fdaffd640 10 client.4356.objecter ms_dispatch 0x7f0fe4000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7f0fdaffd640 3 client.4356.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7f0fdaffd640 3 client.4356.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7f0fdaffd640 20 client.4356.objecter dump_active .. 0 homeless 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f0fe9d84640 10 monclient: get_auth_request con 0x7f0fc8040f80 auth_method 0 2024-08-19T09:50:21.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197581658 0 0) 0x7f247813d850 con 0x7f24780a4f80 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): my global_id is 4363 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f247806d430 con 0x7f24780a4f80 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3114226010 0 0) 0x7fe0800a28b0 con 0x7fe0800a4650 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe08006feb0 con 0x7fe0800a4650 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe07c003610 con 0x7fe0800a4650 2024-08-19T09:50:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): dump: 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3120266097 0 0) 0x7fe064004eb0 con 0x7fe0800a4f80 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0800a28b0 con 0x7fe0800a4f80 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe070001c30 con 0x7fe0800a4f80 2024-08-19T09:50:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): dump: 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.729 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fccaee3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fccaee3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc88005080 con 0x7fcca80a4f80 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 268591247 0 0) 0x7fe0800a1fd0 con 0x7fe08013fcd0 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe064004eb0 con 0x7fe08013fcd0 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe078001ee0 con 0x7fe08013fcd0 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient(hunting): dump: 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.731 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 103429681 0 0) 0x7fe08006feb0 con 0x7fe0800a4650 2024-08-19T09:50:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.82:6792/0 conn(0x7fe08013fcd0 legacy=0x7fe0801dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.1:6790/0 conn(0x7fe0800a4f80 legacy=0x7fe0801d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 monclient: found mon.g 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0800a2470 con 0x7fe0800a4650 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fe0767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2285370272 0 0) 0x7f247806d430 con 0x7f24780a4f80 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f247813d850 con 0x7f24780a4f80 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2468001d00 con 0x7f24780a4f80 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): dump: 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.734 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.735 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7f246dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 5 monclient: authenticate success, global_id 4308 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 10 monclient: _renew_subs 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed140a2a40 con 0x7fed1413fcd0 2024-08-19T09:50:21.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 10 monclient: _renew_subs 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fed140a1370 con 0x7fed1413fcd0 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.617+0000 7fed1c581640 1 librados: init done 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed0c003c40 con 0x7fed1413fcd0 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed0affd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed0c004130 con 0x7fed1413fcd0 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1355776347 0 0) 0x7f247813d850 con 0x7f24780a4f80 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.82:6791/0 conn(0x7f24780a4650 legacy=0x7f2478138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.1:6791/0 conn(0x7f247813fcd0 legacy=0x7f24781d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 1 monclient: found mon.i 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247813c050 con 0x7f24780a4f80 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 5 monclient: authenticate success, global_id 4363 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 10 monclient: _renew_subs 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f247813e230 con 0x7f24780a4f80 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcc940042b0 con 0x7fcca80a4f80 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fcc9404daf0 con 0x7fcca80a4f80 2024-08-19T09:50:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fcca57fa640 10 client.4365.objecter ms_dispatch 0x7fcca8000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fcca57fa640 3 client.4365.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fcca57fa640 3 client.4365.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 20 client.4365.objecter dump_active .. 0 homeless 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe07c003b10 con 0x7fe0800a4650 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe0767fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe07c004040 con 0x7fe0800a4650 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe0767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe0767fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe0767fc640 10 monclient: dump: 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7fe077fff640 4 set_mon_vals no callback set 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed1c581640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed0affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed0affd640 10 monclient: dump: 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.618+0000 7fed18b1a640 4 set_mon_vals no callback set 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fed0c0042d0 con 0x7fed1413fcd0 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fed1c581640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fed1c581640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:21.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.619+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed140a16b0 con 0x7fed1413fcd0 2024-08-19T09:50:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca7fff640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6789/0 conn(0x7fcca80a4f80 legacy=0x7fcca81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T09:50:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca7fff640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6789/0 conn(0x7fcca80a4f80 legacy=0x7fcca81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:50:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca7fff640 1 --1- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6789/0 conn(0x7fcca80a4f80 0x7fcca81d7730 :-1 s=OPENED pgs=87 cs=1 l=1).handle_message read tag failed 2024-08-19T09:50:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca7fff640 1 --1- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6789/0 conn(0x7fcca80a4f80 0x7fcca81d7730 :-1 s=OPENED pgs=87 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:50:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.1:6789/0 2024-08-19T09:50:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6789/0 conn(0x7fcca80a4f80 legacy=0x7fcca81d7730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient: _add_conns ranks=[6,4,8,0,2,3,7,1,5] 2024-08-19T09:50:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 5 monclient: authenticate success, global_id 4368 2024-08-19T09:50:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient(hunting): picked mon.d con 0x7fcc8c0476a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 10 monclient: _renew_subs 2024-08-19T09:50:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f9c100177d0 con 0x7f9c20068650 2024-08-19T09:50:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe080070190 con 0x7fe0800a4650 2024-08-19T09:50:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f9c07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f9c07fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient(hunting): picked mon.c con 0x7fcca80a4650 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient(hunting): picked mon.e con 0x7fcc8c04f080 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcca806fc20 con 0x7fcca80a4650 2024-08-19T09:50:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcca80a22e0 con 0x7fcc8c0476a0 2024-08-19T09:50:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcc8c0552a0 con 0x7fcc8c04f080 2024-08-19T09:50:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fcca57fa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 10 monclient: _renew_subs 2024-08-19T09:50:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe08006fbd0 con 0x7fe0800a4650 2024-08-19T09:50:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fe08810d640 1 librados: init done 2024-08-19T09:50:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f0fdaffd640 10 client.4356.objecter ms_handle_connect 0x7f0fc8040f80 2024-08-19T09:50:21.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fed0c04dd00 con 0x7fed1413fcd0 2024-08-19T09:50:21.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fccacbdb640 10 monclient(hunting): get_auth_request con 0x7fcc8c040f70 auth_method 0 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 10 monclient: _renew_subs 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f247813aeb0 con 0x7f24780a4f80 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f247f3f7640 1 librados: init done 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24680038c0 con 0x7f24780a4f80 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.620+0000 7f246f7fe640 4 set_mon_vals no callback set 2024-08-19T09:50:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2468003db0 con 0x7f24780a4f80 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f246dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f246dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7f246dffb640 10 monclient: dump: 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.754 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fed0affd640 10 client.4308.objecter ms_dispatch 0x7fed14000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fed0affd640 3 client.4308.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.621+0000 7fed0affd640 3 client.4308.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.622+0000 7fed0affd640 20 client.4308.objecter dump_active .. 0 homeless 2024-08-19T09:50:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.622+0000 7fed1a31d640 10 monclient: get_auth_request con 0x7fecf80419c0 auth_method 0 2024-08-19T09:50:21.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.622+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe07c0041e0 con 0x7fe0800a4650 2024-08-19T09:50:21.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.622+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2468003f50 con 0x7f24780a4f80 2024-08-19T09:50:21.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.622+0000 7f247f3f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.622+0000 7f247f3f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f247f3f7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:21.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2458005080 con 0x7f24780a4f80 2024-08-19T09:50:21.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fe07c04d9f0 con 0x7fe0800a4650 2024-08-19T09:50:21.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe0767fc640 10 client.4368.objecter ms_dispatch 0x7fe080000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:21.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe0767fc640 3 client.4368.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:21.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe0767fc640 3 client.4368.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:21.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f246804d8e0 con 0x7f24780a4f80 2024-08-19T09:50:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f246dffb640 10 client.4363.objecter ms_dispatch 0x7f2478000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f246dffb640 3 client.4363.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f246dffb640 3 client.4363.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe0767fc640 20 client.4368.objecter dump_active .. 0 homeless 2024-08-19T09:50:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f246dffb640 20 client.4363.objecter dump_active .. 0 homeless 2024-08-19T09:50:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7f247d193640 10 monclient: get_auth_request con 0x7f245c0411e0 auth_method 0 2024-08-19T09:50:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe08810d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe08810d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe08810d640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0800a5860 con 0x7fe0800a4650 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fcca57fa640 10 client.4365.objecter ms_handle_connect 0x7fcca80a4650 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fcca57fa640 10 client.4365.objecter resend_mon_ops 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.623+0000 7fccad3dc640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6791/0 conn(0x7fcca80a4650 legacy=0x7fcc8c04d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca7fff640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6792/0 conn(0x7fcc8c0476a0 legacy=0x7fcc8c049a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 client.4365.objecter ms_handle_connect 0x7fcc8c0476a0 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 client.4365.objecter resend_mon_ops 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407125032 0 0) 0x7fcca806fc20 con 0x7fcca80a4650 2024-08-19T09:50:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:50:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fcc8c054220 con 0x7fcca80a4650 2024-08-19T09:50:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 client.4365.objecter ms_handle_connect 0x7fcc8c040f70 2024-08-19T09:50:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fe0856a8640 10 monclient: get_auth_request con 0x7fe064041aa0 auth_method 0 2024-08-19T09:50:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fccacbdb640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6793/0 conn(0x7fcc8c04f080 legacy=0x7fcc8c051540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 client.4365.objecter ms_handle_connect 0x7fcc8c04f080 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 client.4365.objecter resend_mon_ops 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 357970469 0 0) 0x7fcc8c054220 con 0x7fcca80a4650 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6793/0 conn(0x7fcc8c04f080 legacy=0x7fcc8c051540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6792/0 conn(0x7fcc8c0476a0 legacy=0x7fcc8c049a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 monclient: found mon.c 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7fcc8c0565a0 con 0x7fcca80a4650 2024-08-19T09:50:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca806fc20 con 0x7fcca80a4650 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcca0001ee0 con 0x7fcca80a4650 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.624+0000 7fcca57fa640 10 monclient: dump: 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:21.767 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.625+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcca0003990 con 0x7fcca80a4650 2024-08-19T09:50:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.625+0000 7fcca57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.625+0000 7fcca67fc640 4 set_mon_vals no callback set 2024-08-19T09:50:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.625+0000 7fed0affd640 10 client.4308.objecter ms_handle_connect 0x7fecf80419c0 2024-08-19T09:50:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.625+0000 7f246dffb640 10 client.4363.objecter ms_handle_connect 0x7f245c0411e0 2024-08-19T09:50:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.625+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcca0001ee0 con 0x7fcca80a4650 2024-08-19T09:50:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.626+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f0fe00175f0 con 0x7f0fe40a4500 2024-08-19T09:50:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.626+0000 7f0fdaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.626+0000 7f0fdaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.626+0000 7fe0767fc640 10 client.4368.objecter ms_handle_connect 0x7fe064041aa0 2024-08-19T09:50:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.628+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe0800a5860 con 0x7fe0800a4650 2024-08-19T09:50:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.628+0000 7fe0767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.628+0000 7fe0767fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.628+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f24680173e0 con 0x7f24780a4f80 2024-08-19T09:50:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.628+0000 7f246dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.628+0000 7f246dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.629+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fed140a16b0 con 0x7fed1413fcd0 2024-08-19T09:50:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.629+0000 7fed0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.629+0000 7fed0affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.631+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fcca0016540 con 0x7fcca80a4650 2024-08-19T09:50:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.631+0000 7fcca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.631+0000 7fcca57fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:21.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fec7e9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fec7e9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fec7e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0fe40aa590 con 0x7f0fe40a4500 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fdaffd640 1 -- 172.21.15.1:0/3964635550 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 3427191320) 0x7f0fe0016ef0 con 0x7f0fe40a4500 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fdaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.817+0000 7f0fdaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:21.819 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-08-19T09:50:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.818+0000 7fe08810d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T09:50:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.818+0000 7fe08810d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T09:50:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.818+0000 7fe08810d640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:50:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.818+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe0800a77d0 con 0x7fe0800a4650 2024-08-19T09:50:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.819+0000 7fe0767fc640 1 -- 172.21.15.1:0/152356919 <== mon.3 v1:172.21.15.82:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 3756282479) 0x7fe07c017510 con 0x7fe0800a4650 2024-08-19T09:50:21.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.819+0000 7fe0767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T09:50:21.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.819+0000 7fe0767fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:21.821 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-08-19T09:50:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 10 librados: watch_flush enter 2024-08-19T09:50:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 10 librados: watch_flush exit 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 20 client.4356.objecter shutdown clearing up homeless session... 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 10 client.4356.objecter successfully canceled tick 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0fc8040f80 legacy=0x7f0fc8043440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 10 monclient: shutdown 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 >> v1:172.21.15.1:6789/0 conn(0x7f0fe40a4500 legacy=0x7f0fe4122370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 shutdown_connections 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 >> 172.21.15.1:0/3964635550 conn(0x7f0fe40ac440 msgr2=0x7f0fe40a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 shutdown_connections 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 -- 172.21.15.1:0/3964635550 wait complete. 2024-08-19T09:50:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.820+0000 7f0fec7e9640 1 librados: shutdown 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 10 librados: watch_flush enter 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 10 librados: watch_flush exit 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 20 client.4368.objecter shutdown clearing up homeless session... 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 10 client.4368.objecter successfully canceled tick 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe064041aa0 legacy=0x7fe064043f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 10 monclient: shutdown 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 >> v1:172.21.15.82:6790/0 conn(0x7fe0800a4650 legacy=0x7fe0800726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.821+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 shutdown_connections 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.822+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 >> 172.21.15.1:0/152356919 conn(0x7fe0800a8ef0 msgr2=0x7fe0801420c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.822+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 shutdown_connections 2024-08-19T09:50:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.822+0000 7fe08810d640 1 -- 172.21.15.1:0/152356919 wait complete. 2024-08-19T09:50:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.822+0000 7fe08810d640 1 librados: shutdown 2024-08-19T09:50:21.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-08-19T09:50:21.833 DEBUG:teuthology.parallel:result is None 2024-08-19T09:50:21.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-08-19T09:50:21.834 DEBUG:teuthology.parallel:result is None 2024-08-19T09:50:21.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c25c94640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T09:50:21.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c25c94640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T09:50:21.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c25c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:21.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9c00005370 con 0x7f9c20068650 2024-08-19T09:50:21.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c07fff640 1 -- 172.21.15.1:0/1478202240 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 203001719) 0x7f9c100170d0 con 0x7f9c20068650 2024-08-19T09:50:21.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T09:50:21.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.938+0000 7f9c07fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:21.940 INFO:teuthology.orchestra.run.smithi001.stdout:51539607556 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 10 librados: watch_flush enter 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 10 librados: watch_flush exit 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 20 client.4342.objecter shutdown clearing up homeless session... 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 10 client.4342.objecter successfully canceled tick 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9c0c040f30 legacy=0x7f9c0c0433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 10 monclient: shutdown 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 >> v1:172.21.15.1:6791/0 conn(0x7f9c20068650 legacy=0x7f9c201ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 shutdown_connections 2024-08-19T09:50:21.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 >> 172.21.15.1:0/1478202240 conn(0x7f9c200a80b0 msgr2=0x7f9c200a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 shutdown_connections 2024-08-19T09:50:21.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 -- 172.21.15.1:0/1478202240 wait complete. 2024-08-19T09:50:21.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.940+0000 7f9c25c94640 1 librados: shutdown 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed1c581640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed1c581640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed1c581640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fed140a77d0 con 0x7fed1413fcd0 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed0affd640 1 -- 172.21.15.1:0/1565907259 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 532354816) 0x7fed0c017740 con 0x7fed1413fcd0 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T09:50:21.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.943+0000 7fed0affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:21.945 INFO:teuthology.orchestra.run.smithi001.stdout:51539607557 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 10 librados: watch_flush enter 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 10 librados: watch_flush exit 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 20 client.4308.objecter shutdown clearing up homeless session... 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 10 client.4308.objecter successfully canceled tick 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.82:6812/2549104415 conn(0x7fecf80419c0 legacy=0x7fecf8043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 10 monclient: shutdown 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 >> v1:172.21.15.1:6792/0 conn(0x7fed1413fcd0 legacy=0x7fed141d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 shutdown_connections 2024-08-19T09:50:21.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 >> 172.21.15.1:0/1565907259 conn(0x7fed140a8ef0 msgr2=0x7fed140aa890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:21.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 shutdown_connections 2024-08-19T09:50:21.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 -- 172.21.15.1:0/1565907259 wait complete. 2024-08-19T09:50:21.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:21.945+0000 7fed1c581640 1 librados: shutdown 2024-08-19T09:50:21.953 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.0 2024-08-19T09:50:21.953 DEBUG:teuthology.parallel:result is None 2024-08-19T09:50:21.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-08-19T09:50:21.958 DEBUG:teuthology.parallel:result is None 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.011+0000 7f247f3f7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.011+0000 7f247f3f7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7f247f3f7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2458005370 con 0x7f24780a4f80 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fccaee3f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fccaee3f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T09:50:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fccaee3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcc88005370 con 0x7fcca80a4650 2024-08-19T09:50:22.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7f246dffb640 1 -- 172.21.15.1:0/4195150106 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 203001719) 0x7f2468016ce0 con 0x7f24780a4f80 2024-08-19T09:50:22.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7f246dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T09:50:22.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7f246dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:22.014 INFO:teuthology.orchestra.run.smithi001.stdout:51539607556 2024-08-19T09:50:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fcca57fa640 1 -- 172.21.15.1:0/196222874 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1169801742) 0x7fcca0048f80 con 0x7fcca80a4650 2024-08-19T09:50:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fcca57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T09:50:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.012+0000 7fcca57fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:22.015 INFO:teuthology.orchestra.run.smithi001.stdout:47244640261 2024-08-19T09:50:22.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 10 librados: watch_flush enter 2024-08-19T09:50:22.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 10 librados: watch_flush exit 2024-08-19T09:50:22.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 20 client.4363.objecter shutdown clearing up homeless session... 2024-08-19T09:50:22.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 10 client.4363.objecter successfully canceled tick 2024-08-19T09:50:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 10 librados: watch_flush enter 2024-08-19T09:50:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 10 librados: watch_flush exit 2024-08-19T09:50:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 20 client.4365.objecter shutdown clearing up homeless session... 2024-08-19T09:50:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 10 client.4365.objecter successfully canceled tick 2024-08-19T09:50:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcc8c040f70 legacy=0x7fcc8c043430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.82:6812/2549104415 conn(0x7f245c0411e0 legacy=0x7f245c0436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 10 monclient: shutdown 2024-08-19T09:50:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 10 monclient: shutdown 2024-08-19T09:50:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 >> v1:172.21.15.82:6792/0 conn(0x7f24780a4f80 legacy=0x7f24781cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 >> v1:172.21.15.1:6791/0 conn(0x7fcca80a4650 legacy=0x7fcc8c04d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccad3dc640 1 -- 172.21.15.1:0/196222874 reap_dead start 2024-08-19T09:50:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.015+0000 7fccad3dc640 1 -- 172.21.15.1:0/196222874 reap_dead start 2024-08-19T09:50:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 shutdown_connections 2024-08-19T09:50:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 >> 172.21.15.1:0/4195150106 conn(0x7f24780a8ef0 msgr2=0x7f2478071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 shutdown_connections 2024-08-19T09:50:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 >> 172.21.15.1:0/196222874 conn(0x7fcca80a8ef0 msgr2=0x7fcca80aac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 shutdown_connections 2024-08-19T09:50:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7f247f3f7640 1 -- 172.21.15.1:0/4195150106 wait complete. 2024-08-19T09:50:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 shutdown_connections 2024-08-19T09:50:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7fccaee3f640 1 -- 172.21.15.1:0/196222874 wait complete. 2024-08-19T09:50:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7f247f3f7640 1 librados: shutdown 2024-08-19T09:50:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.016+0000 7fccaee3f640 1 librados: shutdown 2024-08-19T09:50:22.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.2 2024-08-19T09:50:22.028 DEBUG:teuthology.parallel:result is None 2024-08-19T09:50:22.029 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2024-08-19T09:50:22.029 DEBUG:teuthology.parallel:result is None 2024-08-19T09:50:22.029 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-19T09:50:22.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T09:50:22.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.126+0000 7f65707bc640 10 monclient: get_monmap_and_config 2024-08-19T09:50:22.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.126+0000 7f65707bc640 10 monclient: build_initial_monmap 2024-08-19T09:50:22.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.126+0000 7f65707bc640 10 monclient: monmap: 2024-08-19T09:50:22.127 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:22.127402+0000 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:22.127402+0000 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:22.128 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:22.129 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 1 Processor -- start 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 1 -- start start 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient: init 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient: _add_conns ranks=[8,0,6,3,5,2,7,1,4] 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient(hunting): picked mon.noname-e con 0x7f65680a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient(hunting): picked mon.noname-a con 0x7f65680a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient(hunting): picked mon.noname-d con 0x7f656813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65680a3b40 con 0x7f65680a6ba0 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65680a8130 con 0x7f656813f5c0 2024-08-19T09:50:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656806ede0 con 0x7f65680a67c0 2024-08-19T09:50:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f65707bc640 10 monclient(hunting): authenticate will time out at 678.203715s 2024-08-19T09:50:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f656e558640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f65680a67c0 0x7f65680a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44724/0 (socket says 172.21.15.1:44724) 2024-08-19T09:50:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f656e558640 1 -- 172.21.15.1:0/3619838313 learned_addr learned my addr 172.21.15.1:0/3619838313 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.127+0000 7f656ed59640 1 --1- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6792/0 conn(0x7f656813f5c0 0x7f65681419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52906/0 (socket says 172.21.15.1:52906) 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.129+0000 7f656dd57640 1 -- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6789/0 conn(0x7f65680a6ba0 legacy=0x7f65680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656e558640 1 -- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6793/0 conn(0x7f65680a67c0 legacy=0x7f65680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656ed59640 1 -- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6792/0 conn(0x7f656813f5c0 legacy=0x7f65681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145815750 0 0) 0x7f65680a3b40 con 0x7f65680a6ba0 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): my global_id is 4374 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6550003520 con 0x7f65680a6ba0 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964868128 0 0) 0x7f656806ede0 con 0x7f65680a67c0 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): my global_id is 4373 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65680a3b40 con 0x7f65680a67c0 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896790390 0 0) 0x7f65680a8130 con 0x7f656813f5c0 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient(hunting): my global_id is 4317 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f656806ede0 con 0x7f656813f5c0 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2492275123 0 0) 0x7f6550003520 con 0x7f65680a6ba0 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6793/0 conn(0x7f65680a67c0 legacy=0x7f65680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6792/0 conn(0x7f656813f5c0 legacy=0x7f65681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: _finish_hunting 0 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 monclient: found mon.noname-a 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65680a92a0 con 0x7f65680a6ba0 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: _finish_auth 0 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: _check_auth_tickets 2024-08-19T09:50:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f65707bc640 5 monclient: authenticate success, global_id 4374 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f655c001b30 con 0x7f65680a6ba0 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f65707bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: dump: 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.142 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f65707bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f655c003610 con 0x7f65680a6ba0 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:22.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 1 -- 172.21.15.1:0/3619838313 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f655c003b80 con 0x7f65680a6ba0 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f65707bc640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f65707bc640 4 set_mon_vals no callback set 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.130+0000 7f656d556640 10 monclient: dump: 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.144 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 10 monclient: shutdown 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 1 -- 172.21.15.1:0/3619838313 >> v1:172.21.15.1:6789/0 conn(0x7f65680a6ba0 legacy=0x7f65680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 1 -- 172.21.15.1:0/3619838313 shutdown_connections 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 1 -- 172.21.15.1:0/3619838313 >> 172.21.15.1:0/3619838313 conn(0x7f65680ac4b0 msgr2=0x7f65680a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 1 -- 172.21.15.1:0/3619838313 shutdown_connections 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 1 -- 172.21.15.1:0/3619838313 wait complete. 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 10 monclient: build_initial_monmap 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 10 monclient: monmap: 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:22.146 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.131+0000 7f65707bc640 1 librados: starting msgr at 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 librados: starting objecter 2024-08-19T09:50:22.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 Processor -- start 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 -- start start 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 librados: setting wanted keys 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 librados: calling monclient init 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient: init 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient: _add_conns ranks=[4,5,8,3,1,6,2,0,7] 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient(hunting): picked mon.noname-c con 0x7f65680a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient(hunting): picked mon.noname-h con 0x7f65680a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient(hunting): picked mon.noname-e con 0x7f656813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6568139ad0 con 0x7f65680a67c0 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656813a990 con 0x7f65680a6ba0 2024-08-19T09:50:22.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6568142e70 con 0x7f656813f5c0 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f65707bc640 10 monclient(hunting): authenticate will time out at 678.208632s 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f656e558640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f65680a67c0 0x7f65680a0010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57460/0 (socket says 172.21.15.1:57460) 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f656e558640 1 -- 172.21.15.1:0/3267341760 learned_addr learned my addr 172.21.15.1:0/3267341760 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.132+0000 7f656ed59640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f656813f5c0 0x7f65681d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44734/0 (socket says 172.21.15.1:44734) 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f656e558640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.1:6791/0 conn(0x7f65680a67c0 legacy=0x7f65680a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f6566ffd640 10 client.?.objecter ms_handle_connect 0x7f65680a67c0 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f6566ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f6566ffd640 10 client.?.objecter ms_handle_connect 0x7f656813f5c0 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f6566ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f656ed59640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.1:6793/0 conn(0x7f656813f5c0 legacy=0x7f65681d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.134+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 386157412 0 0) 0x7f6568139ad0 con 0x7f65680a67c0 2024-08-19T09:50:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): my global_id is 4351 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6554004eb0 con 0x7f65680a67c0 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934944191 0 0) 0x7f6568142e70 con 0x7f656813f5c0 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): my global_id is 4382 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6568139ad0 con 0x7f656813f5c0 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f656dd57640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.82:6791/0 conn(0x7f65680a6ba0 legacy=0x7f65680a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 client.?.objecter ms_handle_connect 0x7f65680a6ba0 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4151359934 0 0) 0x7f6554004eb0 con 0x7f65680a67c0 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6568142e70 con 0x7f65680a67c0 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6560001960 con 0x7f65680a67c0 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): dump: 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.151 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2983397329 0 0) 0x7f6568139ad0 con 0x7f656813f5c0 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6554004eb0 con 0x7f656813f5c0 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367186474 0 0) 0x7f656813a990 con 0x7f65680a6ba0 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): my global_id is 4406 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6568139ad0 con 0x7f65680a6ba0 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6558001b30 con 0x7f656813f5c0 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient(hunting): dump: 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1133957655 0 0) 0x7f6568142e70 con 0x7f65680a67c0 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.1:6793/0 conn(0x7f656813f5c0 legacy=0x7f65681d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.82:6791/0 conn(0x7f65680a6ba0 legacy=0x7f65680a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 monclient: found mon.c 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65680a17d0 con 0x7f65680a67c0 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 5 monclient: authenticate success, global_id 4351 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 10 monclient: _renew_subs 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f656813a6b0 con 0x7f65680a67c0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 10 monclient: _renew_subs 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f65680a0e30 con 0x7f65680a67c0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f65707bc640 1 librados: init done 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6560004dd0 con 0x7f65680a67c0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f65600052c0 con 0x7f65680a67c0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6567fff640 4 set_mon_vals no callback set 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.135+0000 7f6566ffd640 10 monclient: dump: 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6560011720 con 0x7f65680a67c0 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f65707bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f65707bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f65707bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f656004dc10 con 0x7f65680a67c0 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f6566ffd640 10 client.4351.objecter ms_dispatch 0x7f6568000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f6566ffd640 3 client.4351.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f6566ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6548005080 con 0x7f65680a67c0 2024-08-19T09:50:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f6566ffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-08-19T09:50:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.136+0000 7f656dd57640 10 monclient: get_auth_request con 0x7f6554041790 auth_method 0 2024-08-19T09:50:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.139+0000 7f6566ffd640 10 client.4351.objecter ms_handle_connect 0x7f6554041790 2024-08-19T09:50:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.141+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6560017710 con 0x7f65680a67c0 2024-08-19T09:50:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.141+0000 7f6566ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.141+0000 7f6566ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.320+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6548000b90 con 0x7f6554041790 2024-08-19T09:50:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.322+0000 7f6566ffd640 1 -- 172.21.15.1:0/3267341760 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34116 (unknown 2965378022 0 475199419) 0x7f6548000b90 con 0x7f6554041790 2024-08-19T09:50:22.322 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 10 librados: watch_flush enter 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 10 librados: watch_flush exit 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 20 client.4351.objecter shutdown clearing up homeless session... 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 10 client.4351.objecter successfully canceled tick 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6554041790 legacy=0x7f6554043c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 10 monclient: shutdown 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 >> v1:172.21.15.1:6791/0 conn(0x7f65680a67c0 legacy=0x7f65680a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 shutdown_connections 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 >> 172.21.15.1:0/3267341760 conn(0x7f65680ac4b0 msgr2=0x7f65680aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 shutdown_connections 2024-08-19T09:50:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 -- 172.21.15.1:0/3267341760 wait complete. 2024-08-19T09:50:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.323+0000 7f65707bc640 1 librados: shutdown 2024-08-19T09:50:22.334 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-08-19T09:50:20.758856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172536,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058504,"statfs":{"total":579820584960,"available":579643908096,"internally_reserved":0,"allocated":2056192,"data_stored":1366882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000723"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768240+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T09:50:17.768240+0000","last_peered":"2024-08-19T09:50:17.768240+0000","last_clean":"2024-08-19T09:50:17.768240+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T09:50:17.768240+0000","last_undegraded":"2024-08-19T09:50:17.768240+0000","last_fullsized":"2024-08-19T09:50:17.768240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769663+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T09:50:17.769663+0000","last_peered":"2024-08-19T09:50:17.769663+0000","last_clean":"2024-08-19T09:50:17.769663+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T09:50:17.769663+0000","last_undegraded":"2024-08-19T09:50:17.769663+0000","last_fullsized":"2024-08-19T09:50:17.769663+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768483+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T09:50:17.768483+0000","last_peered":"2024-08-19T09:50:17.768483+0000","last_clean":"2024-08-19T09:50:17.768483+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T09:50:17.768483+0000","last_undegraded":"2024-08-19T09:50:17.768483+0000","last_fullsized":"2024-08-19T09:50:17.768483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769608+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T09:50:17.769608+0000","last_peered":"2024-08-19T09:50:17.769608+0000","last_clean":"2024-08-19T09:50:17.769608+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T09:50:17.769608+0000","last_undegraded":"2024-08-19T09:50:17.769608+0000","last_fullsized":"2024-08-19T09:50:17.769608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768001+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T09:50:17.768001+0000","last_peered":"2024-08-19T09:50:17.768001+0000","last_clean":"2024-08-19T09:50:17.768001+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T09:50:17.768001+0000","last_undegraded":"2024-08-19T09:50:17.768001+0000","last_fullsized":"2024-08-19T09:50:17.768001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767693+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T09:50:17.767693+0000","last_peered":"2024-08-19T09:50:17.767693+0000","last_clean":"2024-08-19T09:50:17.767693+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T09:50:17.767693+0000","last_undegraded":"2024-08-19T09:50:17.767693+0000","last_fullsized":"2024-08-19T09:50:17.767693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768191+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T09:50:17.768191+0000","last_peered":"2024-08-19T09:50:17.768191+0000","last_clean":"2024-08-19T09:50:17.768191+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T09:50:17.768191+0000","last_undegraded":"2024-08-19T09:50:17.768191+0000","last_fullsized":"2024-08-19T09:50:17.768191+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768119+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T09:50:17.768119+0000","last_peered":"2024-08-19T09:50:17.768119+0000","last_clean":"2024-08-19T09:50:17.768119+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T09:50:17.768119+0000","last_undegraded":"2024-08-19T09:50:17.768119+0000","last_fullsized":"2024-08-19T09:50:17.768119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767720+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T09:50:17.767720+0000","last_peered":"2024-08-19T09:50:17.767720+0000","last_clean":"2024-08-19T09:50:17.767720+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T09:50:17.767720+0000","last_undegraded":"2024-08-19T09:50:17.767720+0000","last_fullsized":"2024-08-19T09:50:17.767720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T09:50:22.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T09:50:22.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.433+0000 7ffbdc928640 10 monclient: get_monmap_and_config 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.433+0000 7ffbdc928640 10 monclient: build_initial_monmap 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.433+0000 7ffbdc928640 10 monclient: monmap: 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:22.434637+0000 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:22.434637+0000 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:22.487 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:22.487 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:22.487 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:22.488 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 1 Processor -- start 2024-08-19T09:50:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 1 -- start start 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient: init 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient: _add_conns ranks=[6,3,5,2,7,4,1,8,0] 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient(hunting): picked mon.noname-d con 0x7ffbd80a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient(hunting): picked mon.noname-g con 0x7ffbd80a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient(hunting): picked mon.noname-h con 0x7ffbd813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd80a3b40 con 0x7ffbd80a6ba0 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd80a8130 con 0x7ffbd813f5c0 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd806ede0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.434+0000 7ffbdc928640 10 monclient(hunting): authenticate will time out at 678.510980s 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.435+0000 7ffbd659c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ffbd80a67c0 0x7ffbd80a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52912/0 (socket says 172.21.15.1:52912) 2024-08-19T09:50:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.435+0000 7ffbd659c640 1 -- 172.21.15.1:0/1985780411 learned_addr learned my addr 172.21.15.1:0/1985780411 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd659c640 1 -- 172.21.15.1:0/1985780411 >> v1:172.21.15.1:6792/0 conn(0x7ffbd80a67c0 legacy=0x7ffbd80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd6d9d640 1 -- 172.21.15.1:0/1985780411 >> v1:172.21.15.82:6791/0 conn(0x7ffbd813f5c0 legacy=0x7ffbd81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303564197 0 0) 0x7ffbd806ede0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient(hunting): my global_id is 4326 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbcffff640 1 -- 172.21.15.1:0/1985780411 >> v1:172.21.15.82:6790/0 conn(0x7ffbd80a6ba0 legacy=0x7ffbd80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbc4003520 con 0x7ffbd80a67c0 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2142119413 0 0) 0x7ffbc4003520 con 0x7ffbd80a67c0 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 >> v1:172.21.15.82:6791/0 conn(0x7ffbd813f5c0 legacy=0x7ffbd81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 >> v1:172.21.15.82:6790/0 conn(0x7ffbd80a6ba0 legacy=0x7ffbd80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 monclient: found mon.noname-d 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd80a92a0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: _finish_auth 0 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc8001ad0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: dump: 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbdc928640 5 monclient: authenticate success, global_id 4326 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbdc928640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffbc80048a0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 1 -- 172.21.15.1:0/1985780411 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc8004dd0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbdc928640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbdc928640 4 set_mon_vals no callback set 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.437+0000 7ffbd5d9b640 10 monclient: dump: 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 10 monclient: shutdown 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 1 -- 172.21.15.1:0/1985780411 >> v1:172.21.15.1:6792/0 conn(0x7ffbd80a67c0 legacy=0x7ffbd80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 1 -- 172.21.15.1:0/1985780411 shutdown_connections 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 1 -- 172.21.15.1:0/1985780411 >> 172.21.15.1:0/1985780411 conn(0x7ffbd80ac4b0 msgr2=0x7ffbd80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 1 -- 172.21.15.1:0/1985780411 shutdown_connections 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.438+0000 7ffbdc928640 1 -- 172.21.15.1:0/1985780411 wait complete. 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient: build_initial_monmap 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient: monmap: 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 librados: starting msgr at 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 librados: starting objecter 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 Processor -- start 2024-08-19T09:50:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 -- start start 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 librados: setting wanted keys 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 librados: calling monclient init 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient: init 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient: _add_conns ranks=[8,5,1,4,6,3,0,2,7] 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient(hunting): picked mon.noname-e con 0x7ffbd80a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient(hunting): picked mon.noname-h con 0x7ffbd80a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient(hunting): picked mon.noname-f con 0x7ffbd813f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd81cf020 con 0x7ffbd813f5c0 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd8139a70 con 0x7ffbd80a6ba0 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd80aa1d0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.439+0000 7ffbdc928640 10 monclient(hunting): authenticate will time out at 678.515970s 2024-08-19T09:50:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.440+0000 7ffbd659c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ffbd80a67c0 0x7ffbd81ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44740/0 (socket says 172.21.15.1:44740) 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.440+0000 7ffbd659c640 1 -- 172.21.15.1:0/3664009294 learned_addr learned my addr 172.21.15.1:0/3664009294 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbd659c640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.1:6793/0 conn(0x7ffbd80a67c0 legacy=0x7ffbd81ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 client.?.objecter ms_handle_connect 0x7ffbd80a67c0 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 client.?.objecter ms_handle_connect 0x7ffbd813f5c0 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 client.?.objecter ms_handle_connect 0x7ffbd80a6ba0 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbd6d9d640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.82:6789/0 conn(0x7ffbd813f5c0 legacy=0x7ffbd81d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135214737 0 0) 0x7ffbd80aa1d0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): my global_id is 4391 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbcffff640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.82:6791/0 conn(0x7ffbd80a6ba0 legacy=0x7ffbd81ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb8004eb0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166905221 0 0) 0x7ffbd81cf020 con 0x7ffbd813f5c0 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): my global_id is 4321 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbd80aa1d0 con 0x7ffbd813f5c0 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310275776 0 0) 0x7ffbd8139a70 con 0x7ffbd80a6ba0 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): my global_id is 4424 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbd81cf020 con 0x7ffbd80a6ba0 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 800764232 0 0) 0x7ffbb8004eb0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbd8139a70 con 0x7ffbd80a67c0 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc8004d90 con 0x7ffbd80a67c0 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): dump: 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1867821546 0 0) 0x7ffbd8139a70 con 0x7ffbd80a67c0 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.82:6791/0 conn(0x7ffbd80a6ba0 legacy=0x7ffbd81ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.82:6789/0 conn(0x7ffbd813f5c0 legacy=0x7ffbd81d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 monclient: found mon.e 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd813b510 con 0x7ffbd80a67c0 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 5 monclient: authenticate success, global_id 4391 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 10 monclient: _renew_subs 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd813e5b0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 10 monclient: _renew_subs 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7ffbd813d1e0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbdc928640 1 librados: init done 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffbc8001a70 con 0x7ffbd80a67c0 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc8005700 con 0x7ffbd80a67c0 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbd559a640 4 set_mon_vals no callback set 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.442+0000 7ffbceffd640 10 monclient: dump: 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ffbc8011b60 con 0x7ffbd80a67c0 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7ffbc804db40 con 0x7ffbd80a67c0 2024-08-19T09:50:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbceffd640 10 client.4391.objecter ms_dispatch 0x7ffbd8000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbceffd640 3 client.4391.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbceffd640 3 client.4391.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbdc928640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbdc928640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbdc928640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbceffd640 20 client.4391.objecter dump_active .. 0 homeless 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbbc005080 con 0x7ffbd80a67c0 2024-08-19T09:50:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.443+0000 7ffbcffff640 10 monclient: get_auth_request con 0x7ffbb8040fd0 auth_method 0 2024-08-19T09:50:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.446+0000 7ffbceffd640 10 client.4391.objecter ms_handle_connect 0x7ffbb8040fd0 2024-08-19T09:50:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.448+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ffbc80176a0 con 0x7ffbd80a67c0 2024-08-19T09:50:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.448+0000 7ffbceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.448+0000 7ffbceffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.629+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffbbc000b90 con 0x7ffbb8040fd0 2024-08-19T09:50:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.630+0000 7ffbceffd640 1 -- 172.21.15.1:0/3664009294 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34116 (unknown 2965378022 0 475199419) 0x7ffbbc000b90 con 0x7ffbb8040fd0 2024-08-19T09:50:22.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 10 librados: watch_flush enter 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 10 librados: watch_flush exit 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 20 client.4391.objecter shutdown clearing up homeless session... 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 10 client.4391.objecter successfully canceled tick 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.82:6812/2549104415 conn(0x7ffbb8040fd0 legacy=0x7ffbb8043490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 10 monclient: shutdown 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 >> v1:172.21.15.1:6793/0 conn(0x7ffbd80a67c0 legacy=0x7ffbd81ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 shutdown_connections 2024-08-19T09:50:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 >> 172.21.15.1:0/3664009294 conn(0x7ffbd80ac4b0 msgr2=0x7ffbd813ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 shutdown_connections 2024-08-19T09:50:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.632+0000 7ffbdc928640 1 -- 172.21.15.1:0/3664009294 wait complete. 2024-08-19T09:50:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.633+0000 7ffbdc928640 1 librados: shutdown 2024-08-19T09:50:22.643 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-08-19T09:50:20.758856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172536,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058504,"statfs":{"total":579820584960,"available":579643908096,"internally_reserved":0,"allocated":2056192,"data_stored":1366882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000723"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768240+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T09:50:17.768240+0000","last_peered":"2024-08-19T09:50:17.768240+0000","last_clean":"2024-08-19T09:50:17.768240+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T09:50:17.768240+0000","last_undegraded":"2024-08-19T09:50:17.768240+0000","last_fullsized":"2024-08-19T09:50:17.768240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769663+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T09:50:17.769663+0000","last_peered":"2024-08-19T09:50:17.769663+0000","last_clean":"2024-08-19T09:50:17.769663+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T09:50:17.769663+0000","last_undegraded":"2024-08-19T09:50:17.769663+0000","last_fullsized":"2024-08-19T09:50:17.769663+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768483+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T09:50:17.768483+0000","last_peered":"2024-08-19T09:50:17.768483+0000","last_clean":"2024-08-19T09:50:17.768483+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T09:50:17.768483+0000","last_undegraded":"2024-08-19T09:50:17.768483+0000","last_fullsized":"2024-08-19T09:50:17.768483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769608+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T09:50:17.769608+0000","last_peered":"2024-08-19T09:50:17.769608+0000","last_clean":"2024-08-19T09:50:17.769608+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T09:50:17.769608+0000","last_undegraded":"2024-08-19T09:50:17.769608+0000","last_fullsized":"2024-08-19T09:50:17.769608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768001+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T09:50:17.768001+0000","last_peered":"2024-08-19T09:50:17.768001+0000","last_clean":"2024-08-19T09:50:17.768001+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T09:50:17.768001+0000","last_undegraded":"2024-08-19T09:50:17.768001+0000","last_fullsized":"2024-08-19T09:50:17.768001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767693+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T09:50:17.767693+0000","last_peered":"2024-08-19T09:50:17.767693+0000","last_clean":"2024-08-19T09:50:17.767693+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T09:50:17.767693+0000","last_undegraded":"2024-08-19T09:50:17.767693+0000","last_fullsized":"2024-08-19T09:50:17.767693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768191+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T09:50:17.768191+0000","last_peered":"2024-08-19T09:50:17.768191+0000","last_clean":"2024-08-19T09:50:17.768191+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T09:50:17.768191+0000","last_undegraded":"2024-08-19T09:50:17.768191+0000","last_fullsized":"2024-08-19T09:50:17.768191+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768119+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T09:50:17.768119+0000","last_peered":"2024-08-19T09:50:17.768119+0000","last_clean":"2024-08-19T09:50:17.768119+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T09:50:17.768119+0000","last_undegraded":"2024-08-19T09:50:17.768119+0000","last_fullsized":"2024-08-19T09:50:17.768119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767720+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T09:50:17.767720+0000","last_peered":"2024-08-19T09:50:17.767720+0000","last_clean":"2024-08-19T09:50:17.767720+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T09:50:17.767720+0000","last_undegraded":"2024-08-19T09:50:17.767720+0000","last_fullsized":"2024-08-19T09:50:17.767720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T09:50:22.644 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-19T09:50:22.644 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-19T09:50:22.645 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-19T09:50:22.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-19T09:50:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.740+0000 7f784fd5a640 10 monclient: get_monmap_and_config 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.740+0000 7f784fd5a640 10 monclient: build_initial_monmap 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.740+0000 7f784fd5a640 10 monclient: monmap: 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:22.741654+0000 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:22.741654+0000 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:22.743 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 1 Processor -- start 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 1 -- start start 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 10 monclient: init 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 10 monclient: _add_conns ranks=[5,4,3,6,2,0,7,8,1] 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 10 monclient(hunting): picked mon.noname-h con 0x7f78480a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 10 monclient(hunting): picked mon.noname-c con 0x7f78480a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 10 monclient(hunting): picked mon.noname-g con 0x7f784813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78480a3b40 con 0x7f784813f5c0 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.741+0000 7f784fd5a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78480a8130 con 0x7f78480a6ba0 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.742+0000 7f784fd5a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f784806ede0 con 0x7f78480a67c0 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.742+0000 7f784fd5a640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.742+0000 7f784fd5a640 10 monclient(hunting): authenticate will time out at 678.818019s 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.742+0000 7f784d2f5640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f78480a6ba0 0x7f78480a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57464/0 (socket says 172.21.15.1:57464) 2024-08-19T09:50:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.742+0000 7f784d2f5640 1 -- 172.21.15.1:0/3458094480 learned_addr learned my addr 172.21.15.1:0/3458094480 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784d2f5640 1 -- 172.21.15.1:0/3458094480 >> v1:172.21.15.1:6791/0 conn(0x7f78480a6ba0 legacy=0x7f78480a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784e2f7640 1 -- 172.21.15.1:0/3458094480 >> v1:172.21.15.82:6790/0 conn(0x7f784813f5c0 legacy=0x7f78481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784daf6640 1 -- 172.21.15.1:0/3458094480 >> v1:172.21.15.82:6791/0 conn(0x7f78480a67c0 legacy=0x7f78480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1319998930 0 0) 0x7f78480a8130 con 0x7f78480a6ba0 2024-08-19T09:50:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): my global_id is 4360 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7834003520 con 0x7f78480a6ba0 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126246981 0 0) 0x7f78480a3b40 con 0x7f784813f5c0 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): my global_id is 4386 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78480a8130 con 0x7f784813f5c0 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602995048 0 0) 0x7f784806ede0 con 0x7f78480a67c0 2024-08-19T09:50:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 10 monclient(hunting): my global_id is 4433 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78480a3b40 con 0x7f78480a67c0 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 17935705 0 0) 0x7f7834003520 con 0x7f78480a6ba0 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.744+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 >> v1:172.21.15.82:6791/0 conn(0x7f78480a67c0 legacy=0x7f78480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 >> v1:172.21.15.82:6790/0 conn(0x7f784813f5c0 legacy=0x7f78481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: _finish_hunting 0 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 1 monclient: found mon.noname-c 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78480a92a0 con 0x7f78480a6ba0 2024-08-19T09:50:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: _finish_auth 0 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: _check_auth_tickets 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 5 monclient: authenticate success, global_id 4360 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7838001ae0 con 0x7f78480a6ba0 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: dump: 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.752 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78380048a0 con 0x7f78480a6ba0 2024-08-19T09:50:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 1 -- 172.21.15.1:0/3458094480 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7838004e10 con 0x7f78480a6ba0 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 4 set_mon_vals no callback set 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784caf4640 10 monclient: dump: 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 10 monclient: shutdown 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 1 -- 172.21.15.1:0/3458094480 >> v1:172.21.15.1:6791/0 conn(0x7f78480a6ba0 legacy=0x7f78480a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 1 -- 172.21.15.1:0/3458094480 shutdown_connections 2024-08-19T09:50:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 1 -- 172.21.15.1:0/3458094480 >> 172.21.15.1:0/3458094480 conn(0x7f78480ac4b0 msgr2=0x7f78480a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 1 -- 172.21.15.1:0/3458094480 shutdown_connections 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.745+0000 7f784fd5a640 1 -- 172.21.15.1:0/3458094480 wait complete. 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 10 monclient: build_initial_monmap 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 10 monclient: monmap: 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 1 librados: starting msgr at 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 1 librados: starting objecter 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 1 Processor -- start 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 1 -- start start 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 1 librados: setting wanted keys 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 1 librados: calling monclient init 2024-08-19T09:50:22.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.746+0000 7f784fd5a640 10 monclient: init 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient: _add_conns ranks=[8,4,1,0,5,7,3,6,2] 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient(hunting): picked mon.noname-e con 0x7f78480a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient(hunting): picked mon.noname-c con 0x7f78480a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient(hunting): picked mon.noname-f con 0x7f784813f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f784813e620 con 0x7f784813f5c0 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78481cf020 con 0x7f78480a6ba0 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78480a9cc0 con 0x7f78480a67c0 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784fd5a640 10 monclient(hunting): authenticate will time out at 678.823140s 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784daf6640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f78480a67c0 0x7f78481ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44750/0 (socket says 172.21.15.1:44750) 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784d2f5640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f78480a6ba0 0x7f78481ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57472/0 (socket says 172.21.15.1:57472) 2024-08-19T09:50:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.747+0000 7f784daf6640 1 -- 172.21.15.1:0/3926380306 learned_addr learned my addr 172.21.15.1:0/3926380306 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:22.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f784d2f5640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.1:6791/0 conn(0x7f78480a6ba0 legacy=0x7f78481ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 client.?.objecter ms_handle_connect 0x7f78480a6ba0 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 client.?.objecter ms_handle_connect 0x7f78480a67c0 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f784daf6640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.1:6793/0 conn(0x7f78480a67c0 legacy=0x7f78481ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f784e2f7640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.82:6789/0 conn(0x7f784813f5c0 legacy=0x7f78481d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 client.?.objecter ms_handle_connect 0x7f784813f5c0 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994996537 0 0) 0x7f78481cf020 con 0x7f78480a6ba0 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): my global_id is 4369 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f782c004eb0 con 0x7f78480a6ba0 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317781706 0 0) 0x7f78480a9cc0 con 0x7f78480a67c0 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): my global_id is 4400 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78481cf020 con 0x7f78480a67c0 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3191547970 0 0) 0x7f78480a8130 con 0x7f784813f5c0 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): my global_id is 4330 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78480a9cc0 con 0x7f784813f5c0 2024-08-19T09:50:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 853836340 0 0) 0x7f782c004eb0 con 0x7f78480a6ba0 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78480a8130 con 0x7f78480a6ba0 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7838004d90 con 0x7f78480a6ba0 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.765 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4016597318 0 0) 0x7f78481cf020 con 0x7f78480a67c0 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f782c004eb0 con 0x7f78480a67c0 2024-08-19T09:50:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7844001d40 con 0x7f78480a67c0 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.767 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1469506552 0 0) 0x7f78480a9cc0 con 0x7f784813f5c0 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78481cf020 con 0x7f784813f5c0 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f78400020b0 con 0x7f784813f5c0 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.768 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.769 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3474195292 0 0) 0x7f78480a8130 con 0x7f78480a6ba0 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.1:6793/0 conn(0x7f78480a67c0 legacy=0x7f78481ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.82:6789/0 conn(0x7f784813f5c0 legacy=0x7f78481d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 monclient: found mon.c 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.749+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784813af40 con 0x7f78480a6ba0 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 5 monclient: authenticate success, global_id 4369 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 10 monclient: _renew_subs 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f784813bdb0 con 0x7f78480a6ba0 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7838001d30 con 0x7f78480a6ba0 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7838005740 con 0x7f78480a6ba0 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783e7fc640 10 monclient: dump: 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f783ffff640 4 set_mon_vals no callback set 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 10 monclient: _renew_subs 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f78480aa020 con 0x7f78480a6ba0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.750+0000 7f784fd5a640 1 librados: init done 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f784fd5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7838011ba0 con 0x7f78480a6ba0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f784fd5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f784fd5a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f784809f160 con 0x7f78480a6ba0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f783804da00 con 0x7f78480a6ba0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f783e7fc640 10 client.4369.objecter ms_dispatch 0x7f7848000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f783e7fc640 3 client.4369.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f783e7fc640 3 client.4369.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f783e7fc640 20 client.4369.objecter dump_active .. 0 homeless 2024-08-19T09:50:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.751+0000 7f784daf6640 10 monclient: get_auth_request con 0x7f782c041b00 auth_method 0 2024-08-19T09:50:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.754+0000 7f783e7fc640 10 client.4369.objecter ms_handle_connect 0x7f782c041b00 2024-08-19T09:50:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.756+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f78380175c0 con 0x7f78480a6ba0 2024-08-19T09:50:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.756+0000 7f783e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.756+0000 7f783e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.976+0000 7f784fd5a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.976+0000 7f784fd5a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.976+0000 7f784fd5a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.976+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7848068f60 con 0x7f78480a6ba0 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.977+0000 7f783e7fc640 1 -- 172.21.15.1:0/3926380306 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f7838016ec0 con 0x7f78480a6ba0 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.977+0000 7f783e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.977+0000 7f783e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:22.978 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:22.979 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-19T09:50:22.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 10 librados: watch_flush enter 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 10 librados: watch_flush exit 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 20 client.4369.objecter shutdown clearing up homeless session... 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 10 client.4369.objecter successfully canceled tick 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.82:6812/2549104415 conn(0x7f782c041b00 legacy=0x7f782c043fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 10 monclient: shutdown 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 >> v1:172.21.15.1:6791/0 conn(0x7f78480a6ba0 legacy=0x7f78481ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 shutdown_connections 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 >> 172.21.15.1:0/3926380306 conn(0x7f78480ac4b0 msgr2=0x7f7848142070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 shutdown_connections 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 -- 172.21.15.1:0/3926380306 wait complete. 2024-08-19T09:50:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:22.979+0000 7f784fd5a640 1 librados: shutdown 2024-08-19T09:50:22.991 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-19T09:50:22.991 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-08-19T09:50:23.004 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-08-19T09:50:23.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T09:50:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.101+0000 7f2d26a3f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.101+0000 7f2d26a3f640 10 monclient: build_initial_monmap 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.101+0000 7f2d26a3f640 10 monclient: monmap: 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:23.102640+0000 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:23.102640+0000 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:23.104 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.102+0000 7f2d26a3f640 1 Processor -- start 2024-08-19T09:50:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.102+0000 7f2d26a3f640 1 -- start start 2024-08-19T09:50:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.102+0000 7f2d26a3f640 10 monclient: init 2024-08-19T09:50:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.102+0000 7f2d26a3f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.102+0000 7f2d26a3f640 10 monclient: _add_conns ranks=[3,2,5,0,8,6,4,1,7] 2024-08-19T09:50:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.102+0000 7f2d26a3f640 10 monclient(hunting): picked mon.noname-g con 0x7f2d200a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f2d200a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 10 monclient(hunting): picked mon.noname-h con 0x7f2d2013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d200a3b40 con 0x7f2d200a6ba0 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d200a8130 con 0x7f2d200a67c0 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d2006ede0 con 0x7f2d2013f5c0 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d26a3f640 10 monclient(hunting): authenticate will time out at 679.179069s 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d1f7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f2d200a6ba0 0x7f2d200a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52816/0 (socket says 172.21.15.1:52816) 2024-08-19T09:50:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.103+0000 7f2d1f7fe640 1 -- 172.21.15.1:0/2294402230 learned_addr learned my addr 172.21.15.1:0/2294402230 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1f7fe640 1 -- 172.21.15.1:0/2294402230 >> v1:172.21.15.1:6790/0 conn(0x7f2d200a6ba0 legacy=0x7f2d200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3049305829 0 0) 0x7f2d200a3b40 con 0x7f2d200a6ba0 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient(hunting): my global_id is 4367 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d08003520 con 0x7f2d200a6ba0 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1ffff640 1 -- 172.21.15.1:0/2294402230 >> v1:172.21.15.82:6790/0 conn(0x7f2d200a67c0 legacy=0x7f2d200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d24fdc640 1 -- 172.21.15.1:0/2294402230 >> v1:172.21.15.82:6791/0 conn(0x7f2d2013f5c0 legacy=0x7f2d201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3749070345 0 0) 0x7f2d08003520 con 0x7f2d200a6ba0 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 >> v1:172.21.15.82:6791/0 conn(0x7f2d2013f5c0 legacy=0x7f2d201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 >> v1:172.21.15.82:6790/0 conn(0x7f2d200a67c0 legacy=0x7f2d200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 monclient: found mon.noname-b 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d200a92a0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: _finish_auth 0 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d14001ae0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d26a3f640 5 monclient: authenticate success, global_id 4367 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d26a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: dump: 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d26a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d140048a0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 1 -- 172.21.15.1:0/2294402230 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d14004e10 con 0x7f2d200a6ba0 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d26a3f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d26a3f640 4 set_mon_vals no callback set 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.105+0000 7f2d1effd640 10 monclient: dump: 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 10 monclient: shutdown 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 1 -- 172.21.15.1:0/2294402230 >> v1:172.21.15.1:6790/0 conn(0x7f2d200a6ba0 legacy=0x7f2d200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 1 -- 172.21.15.1:0/2294402230 shutdown_connections 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 1 -- 172.21.15.1:0/2294402230 >> 172.21.15.1:0/2294402230 conn(0x7f2d200ac4b0 msgr2=0x7f2d200a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 1 -- 172.21.15.1:0/2294402230 shutdown_connections 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.106+0000 7f2d26a3f640 1 -- 172.21.15.1:0/2294402230 wait complete. 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 10 monclient: build_initial_monmap 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 10 monclient: monmap: 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:23.116 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 1 librados: starting msgr at 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 1 librados: starting objecter 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 1 Processor -- start 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 1 -- start start 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 1 librados: setting wanted keys 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 1 librados: calling monclient init 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 10 monclient: init 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.107+0000 7f2d26a3f640 10 monclient: _add_conns ranks=[0,4,3,8,7,2,1,5,6] 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 10 monclient(hunting): picked mon.noname-a con 0x7f2d200a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f2d200a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 10 monclient(hunting): picked mon.noname-g con 0x7f2d2013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d200aa060 con 0x7f2d200a67c0 2024-08-19T09:50:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d2013bc00 con 0x7f2d2013f5c0 2024-08-19T09:50:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d2013bee0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d26a3f640 10 monclient(hunting): authenticate will time out at 679.184094s 2024-08-19T09:50:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d1ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2d200a67c0 0x7f2d201ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37220/0 (socket says 172.21.15.1:37220) 2024-08-19T09:50:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.108+0000 7f2d1ffff640 1 -- 172.21.15.1:0/3673315457 learned_addr learned my addr 172.21.15.1:0/3673315457 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1f7fe640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.1:6791/0 conn(0x7f2d200a6ba0 legacy=0x7f2d201d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 client.?.objecter ms_handle_connect 0x7f2d200a6ba0 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 client.?.objecter ms_handle_connect 0x7f2d200a67c0 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1ffff640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.1:6789/0 conn(0x7f2d200a67c0 legacy=0x7f2d201ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 client.?.objecter ms_handle_connect 0x7f2d2013f5c0 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d24fdc640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.82:6790/0 conn(0x7f2d2013f5c0 legacy=0x7f2d201d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585588747 0 0) 0x7f2d2013bee0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): my global_id is 4378 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d0c004ee0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220433987 0 0) 0x7f2d200a8130 con 0x7f2d200a67c0 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): my global_id is 4383 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d2013bee0 con 0x7f2d200a67c0 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917072331 0 0) 0x7f2d2013bc00 con 0x7f2d2013f5c0 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): my global_id is 4404 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d200a8130 con 0x7f2d2013f5c0 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2846984588 0 0) 0x7f2d0c004ee0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d2013bc00 con 0x7f2d200a6ba0 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d14004d90 con 0x7f2d200a6ba0 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): dump: 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1314564020 0 0) 0x7f2d2013bee0 con 0x7f2d200a67c0 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d0c004ee0 con 0x7f2d200a67c0 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d10003610 con 0x7f2d200a67c0 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): dump: 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1032464734 0 0) 0x7f2d200a8130 con 0x7f2d2013f5c0 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d2013bee0 con 0x7f2d2013f5c0 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d18001f40 con 0x7f2d2013f5c0 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient(hunting): dump: 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1541662195 0 0) 0x7f2d2013bc00 con 0x7f2d200a6ba0 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.82:6790/0 conn(0x7f2d2013f5c0 legacy=0x7f2d201d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.1:6789/0 conn(0x7f2d200a67c0 legacy=0x7f2d201ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 monclient: found mon.c 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2013a6e0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.110+0000 7f2d1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 5 monclient: authenticate success, global_id 4378 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 monclient: _renew_subs 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d20122410 con 0x7f2d200a6ba0 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 monclient: _renew_subs 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d2013b640 con 0x7f2d200a6ba0 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 1 librados: init done 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d14001d30 con 0x7f2d200a6ba0 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d14005740 con 0x7f2d200a6ba0 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1e7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d1cff9640 10 monclient: dump: 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.131 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.111+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d00005080 con 0x7f2d200a6ba0 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2d14011ba0 con 0x7f2d200a6ba0 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7f2d1404dc60 con 0x7f2d200a6ba0 2024-08-19T09:50:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1cff9640 10 client.4378.objecter ms_dispatch 0x7f2d20000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1cff9640 3 client.4378.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1cff9640 3 client.4378.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1cff9640 20 client.4378.objecter dump_active .. 0 homeless 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.112+0000 7f2d1ffff640 10 monclient: get_auth_request con 0x7f2d0c0418f0 auth_method 0 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.114+0000 7f2d1cff9640 10 client.4378.objecter ms_handle_connect 0x7f2d0c0418f0 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.116+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f2d14017880 con 0x7f2d200a6ba0 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.116+0000 7f2d1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.116+0000 7f2d1cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:23.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d26a3f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:23.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d26a3f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:23.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d26a3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:23.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d00005370 con 0x7f2d200a6ba0 2024-08-19T09:50:23.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d1cff9640 1 -- 172.21.15.1:0/3673315457 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10897 (unknown 941590788 0 3619314660) 0x7f2d14017180 con 0x7f2d200a6ba0 2024-08-19T09:50:23.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T09:50:23.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.301+0000 7f2d1cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:23.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:23.303 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T09:50:17.763343+0000","last_up_change":"2024-08-19T09:50:07.419178+0000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T09:50:07.171793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T09:50:13.629390+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":3862255507}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":3862255507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":3862255507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":3862255507}]},"public_addr":"172.21.15.1:6804/3862255507","cluster_addr":"172.21.15.1:6805/3862255507","heartbeat_back_addr":"172.21.15.1:6807/3862255507","heartbeat_front_addr":"172.21.15.1:6806/3862255507","state":["exists","up"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1668590231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1668590231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1668590231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1668590231}]},"public_addr":"172.21.15.1:6800/1668590231","cluster_addr":"172.21.15.1:6801/1668590231","heartbeat_back_addr":"172.21.15.1:6803/1668590231","heartbeat_front_addr":"172.21.15.1:6802/1668590231","state":["exists","up"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":207870648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":207870648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":207870648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":207870648}]},"public_addr":"172.21.15.1:6808/207870648","cluster_addr":"172.21.15.1:6809/207870648","heartbeat_back_addr":"172.21.15.1:6811/207870648","heartbeat_front_addr":"172.21.15.1:6810/207870648","state":["exists","up"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6804","nonce":394826682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6805","nonce":394826682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6807","nonce":394826682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6806","nonce":394826682}]},"public_addr":"172.21.15.82:6804/394826682","cluster_addr":"172.21.15.82:6805/394826682","heartbeat_back_addr":"172.21.15.82:6807/394826682","heartbeat_front_addr":"172.21.15.82:6806/394826682","state":["exists","up"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6800","nonce":2865329823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6801","nonce":2865329823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6803","nonce":2865329823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6802","nonce":2865329823}]},"public_addr":"172.21.15.82:6800/2865329823","cluster_addr":"172.21.15.82:6801/2865329823","heartbeat_back_addr":"172.21.15.82:6803/2865329823","heartbeat_front_addr":"172.21.15.82:6802/2865329823","state":["exists","up"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6808","nonce":1813220032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6809","nonce":1813220032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6811","nonce":1813220032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6810","nonce":1813220032}]},"public_addr":"172.21.15.82:6808/1813220032","cluster_addr":"172.21.15.82:6809/1813220032","heartbeat_back_addr":"172.21.15.82:6811/1813220032","heartbeat_front_addr":"172.21.15.82:6810/1813220032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.564842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.315230+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.809229+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:05.004322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.803207+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.616501+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 10 librados: watch_flush enter 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 10 librados: watch_flush exit 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 20 client.4378.objecter shutdown clearing up homeless session... 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 10 client.4378.objecter successfully canceled tick 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2d0c0418f0 legacy=0x7f2d0c043db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 10 monclient: shutdown 2024-08-19T09:50:23.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.303+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 >> v1:172.21.15.1:6791/0 conn(0x7f2d200a6ba0 legacy=0x7f2d201d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.304+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 shutdown_connections 2024-08-19T09:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.304+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 >> 172.21.15.1:0/3673315457 conn(0x7f2d200ac4b0 msgr2=0x7f2d2013ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.304+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 shutdown_connections 2024-08-19T09:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.304+0000 7f2d26a3f640 1 -- 172.21.15.1:0/3673315457 wait complete. 2024-08-19T09:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.304+0000 7f2d26a3f640 1 librados: shutdown 2024-08-19T09:50:23.315 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T09:50:07.171793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-19T09:50:13.629390+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-19T09:50:23.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-19T09:50:23.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.416+0000 7fef731c9640 10 monclient: get_monmap_and_config 2024-08-19T09:50:23.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.416+0000 7fef731c9640 10 monclient: build_initial_monmap 2024-08-19T09:50:23.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.416+0000 7fef731c9640 10 monclient: monmap: 2024-08-19T09:50:23.417 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:23.417 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:23.417324+0000 2024-08-19T09:50:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:23.417324+0000 2024-08-19T09:50:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:23.419 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 1 Processor -- start 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 1 -- start start 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient: init 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient: _add_conns ranks=[6,4,5,2,7,1,8,0,3] 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient(hunting): picked mon.noname-d con 0x7fef6c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient(hunting): picked mon.noname-c con 0x7fef6c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient(hunting): picked mon.noname-h con 0x7fef6c13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6c0a3b40 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6c0a8130 con 0x7fef6c13f5c0 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6c06ede0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef731c9640 10 monclient(hunting): authenticate will time out at 679.493679s 2024-08-19T09:50:23.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef70f65640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fef6c0a67c0 0x7fef6c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52926/0 (socket says 172.21.15.1:52926) 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef6bfff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fef6c0a6ba0 0x7fef6c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57488/0 (socket says 172.21.15.1:57488) 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.417+0000 7fef70f65640 1 -- 172.21.15.1:0/2817979252 learned_addr learned my addr 172.21.15.1:0/2817979252 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.419+0000 7fef6bfff640 1 -- 172.21.15.1:0/2817979252 >> v1:172.21.15.1:6791/0 conn(0x7fef6c0a6ba0 legacy=0x7fef6c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.419+0000 7fef70f65640 1 -- 172.21.15.1:0/2817979252 >> v1:172.21.15.1:6792/0 conn(0x7fef6c0a67c0 legacy=0x7fef6c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170629522 0 0) 0x7fef6c0a3b40 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient(hunting): my global_id is 4387 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef5c003520 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef71766640 1 -- 172.21.15.1:0/2817979252 >> v1:172.21.15.82:6791/0 conn(0x7fef6c13f5c0 legacy=0x7fef6c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3205766143 0 0) 0x7fef5c003520 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 >> v1:172.21.15.1:6792/0 conn(0x7fef6c0a67c0 legacy=0x7fef6c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 >> v1:172.21.15.82:6791/0 conn(0x7fef6c13f5c0 legacy=0x7fef6c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 monclient: found mon.noname-c 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef6c0a92a0 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef731c9640 5 monclient: authenticate success, global_id 4387 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef58001ae0 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef731c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: dump: 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.427 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef731c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef580047e0 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef731c9640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 1 -- 172.21.15.1:0/2817979252 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef58004d50 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef731c9640 4 set_mon_vals no callback set 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef6b7fe640 10 monclient: dump: 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.420+0000 7fef731c9640 10 monclient: shutdown 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 -- 172.21.15.1:0/2817979252 >> v1:172.21.15.1:6791/0 conn(0x7fef6c0a6ba0 legacy=0x7fef6c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 -- 172.21.15.1:0/2817979252 shutdown_connections 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 -- 172.21.15.1:0/2817979252 >> 172.21.15.1:0/2817979252 conn(0x7fef6c0ac4b0 msgr2=0x7fef6c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 -- 172.21.15.1:0/2817979252 shutdown_connections 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 -- 172.21.15.1:0/2817979252 wait complete. 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 10 monclient: build_initial_monmap 2024-08-19T09:50:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 10 monclient: monmap: 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:23.431 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 librados: starting msgr at 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 librados: starting objecter 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.421+0000 7fef731c9640 1 Processor -- start 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 1 -- start start 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 1 librados: setting wanted keys 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 1 librados: calling monclient init 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient: init 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient: _add_conns ranks=[2,4,0,7,1,8,5,6,3] 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient(hunting): picked mon.noname-b con 0x7fef6c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient(hunting): picked mon.noname-c con 0x7fef6c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient(hunting): picked mon.noname-a con 0x7fef6c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6c13e5c0 con 0x7fef6c13f5c0 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6c13bd50 con 0x7fef6c0a67c0 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6c1cf020 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef731c9640 10 monclient(hunting): authenticate will time out at 679.498627s 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef6bfff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fef6c0a6ba0 0x7fef6c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57502/0 (socket says 172.21.15.1:57502) 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef70f65640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fef6c0a67c0 0x7fef6c1ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52818/0 (socket says 172.21.15.1:52818) 2024-08-19T09:50:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.422+0000 7fef6bfff640 1 -- 172.21.15.1:0/2341456216 learned_addr learned my addr 172.21.15.1:0/2341456216 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:23.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef71766640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.1:6789/0 conn(0x7fef6c13f5c0 legacy=0x7fef6c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef697fa640 10 client.?.objecter ms_handle_connect 0x7fef6c13f5c0 2024-08-19T09:50:23.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef697fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef6bfff640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.1:6791/0 conn(0x7fef6c0a6ba0 legacy=0x7fef6c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef697fa640 10 client.?.objecter ms_handle_connect 0x7fef6c0a6ba0 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef697fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef697fa640 10 client.?.objecter ms_handle_connect 0x7fef6c0a67c0 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef697fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.424+0000 7fef70f65640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.1:6790/0 conn(0x7fef6c0a67c0 legacy=0x7fef6c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679714853 0 0) 0x7fef6c1cf020 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): my global_id is 4396 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef50004ee0 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 433960076 0 0) 0x7fef6c13e5c0 con 0x7fef6c13f5c0 2024-08-19T09:50:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): my global_id is 4392 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef6c1cf020 con 0x7fef6c13f5c0 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973799067 0 0) 0x7fef6c13bd50 con 0x7fef6c0a67c0 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): my global_id is 4376 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef6c13e5c0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2228098561 0 0) 0x7fef50004ee0 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef6c13bd50 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 396808698 0 0) 0x7fef6c13e5c0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef50004ee0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef58004cd0 con 0x7fef6c0a6ba0 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): dump: 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.444 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1751401938 0 0) 0x7fef6c1cf020 con 0x7fef6c13f5c0 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef6c13e5c0 con 0x7fef6c13f5c0 2024-08-19T09:50:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef60001c10 con 0x7fef6c0a67c0 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): dump: 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef64002030 con 0x7fef6c13f5c0 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient(hunting): dump: 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.447 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 935297089 0 0) 0x7fef50004ee0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.1:6791/0 conn(0x7fef6c0a6ba0 legacy=0x7fef6c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.1:6789/0 conn(0x7fef6c13f5c0 legacy=0x7fef6c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 monclient: found mon.b 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef6c0a9cc0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 5 monclient: authenticate success, global_id 4376 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 10 monclient: _renew_subs 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef6c13aee0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 10 monclient: _renew_subs 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fef6c13ba70 con 0x7fef6c0a67c0 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef731c9640 1 librados: init done 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef60004b00 con 0x7fef6c0a67c0 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef60005030 con 0x7fef6c0a67c0 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef6affd640 4 set_mon_vals no callback set 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.425+0000 7fef697fa640 10 monclient: dump: 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fef600051d0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef731c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef731c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef731c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4c005080 con 0x7fef6c0a67c0 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2738130309 0 0) 0x7fef6004d870 con 0x7fef6c0a67c0 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef697fa640 10 client.4376.objecter ms_dispatch 0x7fef6c000b90 osd_map(19..19 src has 1..19) 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef697fa640 3 client.4376.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.426+0000 7fef697fa640 3 client.4376.objecter handle_osd_map decoding full epoch 19 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.427+0000 7fef697fa640 20 client.4376.objecter dump_active .. 0 homeless 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.427+0000 7fef6bfff640 10 monclient: get_auth_request con 0x7fef50041970 auth_method 0 2024-08-19T09:50:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.429+0000 7fef697fa640 10 client.4376.objecter ms_handle_connect 0x7fef50041970 2024-08-19T09:50:23.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.431+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fef600173e0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.431+0000 7fef697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.431+0000 7fef697fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:23.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.613+0000 7fef731c9640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-08-19T09:50:23.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.613+0000 7fef731c9640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-08-19T09:50:23.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.613+0000 7fef731c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:23.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.613+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fef4c005c40 con 0x7fef6c0a67c0 2024-08-19T09:50:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.616+0000 7fef697fa640 1 -- 172.21.15.1:0/2341456216 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (unknown 4170632370 0 2170607528) 0x7fef60016ce0 con 0x7fef6c0a67c0 2024-08-19T09:50:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.616+0000 7fef697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-08-19T09:50:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.616+0000 7fef697fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:23.619 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-08-19T09:50:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 10 librados: watch_flush enter 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 10 librados: watch_flush exit 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 20 client.4376.objecter shutdown clearing up homeless session... 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 10 client.4376.objecter successfully canceled tick 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.82:6812/2549104415 conn(0x7fef50041970 legacy=0x7fef50043e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 10 monclient: shutdown 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 >> v1:172.21.15.1:6790/0 conn(0x7fef6c0a67c0 legacy=0x7fef6c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 shutdown_connections 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 >> 172.21.15.1:0/2341456216 conn(0x7fef6c0ac4b0 msgr2=0x7fef6c13ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 shutdown_connections 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.619+0000 7fef731c9640 1 -- 172.21.15.1:0/2341456216 wait complete. 2024-08-19T09:50:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.620+0000 7fef731c9640 1 librados: shutdown 2024-08-19T09:50:23.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-08-19T09:50:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.735+0000 7fe13206e640 10 monclient: get_monmap_and_config 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.735+0000 7fe13206e640 10 monclient: build_initial_monmap 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.735+0000 7fe13206e640 10 monclient: monmap: 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:23.736671+0000 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:23.736671+0000 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:23.737 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:23.738 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.736+0000 7fe13206e640 1 Processor -- start 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.736+0000 7fe13206e640 1 -- start start 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.736+0000 7fe13206e640 10 monclient: init 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient: _add_conns ranks=[8,3,1,5,4,2,6,7,0] 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient(hunting): picked mon.noname-e con 0x7fe12c0a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient(hunting): picked mon.noname-g con 0x7fe12c0694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient(hunting): picked mon.noname-f con 0x7fe12c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c0a3c70 con 0x7fe12c0a46a0 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c0a33d0 con 0x7fe12c0694b0 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c0a36b0 con 0x7fe12c0a6830 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe13206e640 10 monclient(hunting): authenticate will time out at 679.813286s 2024-08-19T09:50:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe12b7fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe12c0a6830 0x7fe12c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44758/0 (socket says 172.21.15.1:44758) 2024-08-19T09:50:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.737+0000 7fe12b7fe640 1 -- 172.21.15.1:0/3030907301 learned_addr learned my addr 172.21.15.1:0/3030907301 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.739+0000 7fe12b7fe640 1 -- 172.21.15.1:0/3030907301 >> v1:172.21.15.1:6793/0 conn(0x7fe12c0a6830 legacy=0x7fe12c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.739+0000 7fe12bfff640 1 -- 172.21.15.1:0/3030907301 >> v1:172.21.15.82:6789/0 conn(0x7fe12c0a46a0 legacy=0x7fe12c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.739+0000 7fe123fff640 1 -- 172.21.15.1:0/3030907301 >> v1:172.21.15.82:6790/0 conn(0x7fe12c0694b0 legacy=0x7fe12c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.758+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911650792 0 0) 0x7fe12c0a3c70 con 0x7fe12c0a46a0 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.758+0000 7fe12affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.758+0000 7fe12affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.758+0000 7fe12affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.758+0000 7fe12affd640 10 monclient(hunting): my global_id is 4339 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.758+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe118003520 con 0x7fe12c0a46a0 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.759+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945306735 0 0) 0x7fe12c0a33d0 con 0x7fe12c0694b0 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.759+0000 7fe12affd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.759+0000 7fe12affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.759+0000 7fe12affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.759+0000 7fe12affd640 10 monclient(hunting): my global_id is 4413 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.759+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe12c0a3c70 con 0x7fe12c0694b0 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 685887990 0 0) 0x7fe118003520 con 0x7fe12c0a46a0 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 >> v1:172.21.15.1:6793/0 conn(0x7fe12c0a6830 legacy=0x7fe12c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 >> v1:172.21.15.82:6790/0 conn(0x7fe12c0694b0 legacy=0x7fe12c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 monclient: found mon.noname-f 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c06dbb0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: _finish_auth 0 2024-08-19T09:50:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe124001eb0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: dump: 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe13206e640 5 monclient: authenticate success, global_id 4339 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe13206e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe124003990 con 0x7fe12c0a46a0 2024-08-19T09:50:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 1 -- 172.21.15.1:0/3030907301 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe124003f00 con 0x7fe12c0a46a0 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe13206e640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe13206e640 4 set_mon_vals no callback set 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.760+0000 7fe12affd640 10 monclient: dump: 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.767 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 10 monclient: shutdown 2024-08-19T09:50:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 1 -- 172.21.15.1:0/3030907301 >> v1:172.21.15.82:6789/0 conn(0x7fe12c0a46a0 legacy=0x7fe12c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 1 -- 172.21.15.1:0/3030907301 shutdown_connections 2024-08-19T09:50:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 1 -- 172.21.15.1:0/3030907301 >> 172.21.15.1:0/3030907301 conn(0x7fe12c0a9020 msgr2=0x7fe12c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 1 -- 172.21.15.1:0/3030907301 shutdown_connections 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.761+0000 7fe13206e640 1 -- 172.21.15.1:0/3030907301 wait complete. 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 10 monclient: build_initial_monmap 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 10 monclient: monmap: 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 1 librados: starting msgr at 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 1 librados: starting objecter 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 1 Processor -- start 2024-08-19T09:50:23.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 1 -- start start 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 1 librados: setting wanted keys 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 1 librados: calling monclient init 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.762+0000 7fe13206e640 10 monclient: init 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient: _add_conns ranks=[1,6,4,7,3,0,2,8,5] 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient(hunting): picked mon.noname-f con 0x7fe12c0694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient(hunting): picked mon.noname-d con 0x7fe12c0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient(hunting): picked mon.noname-c con 0x7fe12c0a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c0a9d30 con 0x7fe12c0694b0 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c138a00 con 0x7fe12c0a6830 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c138ce0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe13206e640 10 monclient(hunting): authenticate will time out at 679.839361s 2024-08-19T09:50:23.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe123fff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe12c0a46a0 0x7fe12c1359f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52930/0 (socket says 172.21.15.1:52930) 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe123fff640 1 -- 172.21.15.1:0/3014476514 learned_addr learned my addr 172.21.15.1:0/3014476514 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.763+0000 7fe12bfff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fe12c0a6830 0x7fe12c1d6610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57518/0 (socket says 172.21.15.1:57518) 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe12bfff640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.1:6791/0 conn(0x7fe12c0a6830 legacy=0x7fe12c1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe123fff640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.1:6792/0 conn(0x7fe12c0a46a0 legacy=0x7fe12c1359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 client.?.objecter ms_handle_connect 0x7fe12c0a6830 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 client.?.objecter ms_handle_connect 0x7fe12c0a46a0 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 client.?.objecter ms_handle_connect 0x7fe12c0694b0 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe12b7fe640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.82:6789/0 conn(0x7fe12c0694b0 legacy=0x7fe12c1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:23.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011828812 0 0) 0x7fe12c138ce0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): my global_id is 4335 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe10c004eb0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775443444 0 0) 0x7fe12c138a00 con 0x7fe12c0a6830 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): my global_id is 4405 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe12c138ce0 con 0x7fe12c0a6830 2024-08-19T09:50:23.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646214726 0 0) 0x7fe11c001500 con 0x7fe12c0694b0 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 10 monclient(hunting): my global_id is 4348 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.765+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe12c138a00 con 0x7fe12c0694b0 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3674027385 0 0) 0x7fe10c004eb0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe11c001500 con 0x7fe12c0a46a0 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe114003950 con 0x7fe12c0a46a0 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.781 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2179728534 0 0) 0x7fe12c138ce0 con 0x7fe12c0a6830 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe10c004eb0 con 0x7fe12c0a6830 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe124003e80 con 0x7fe12c0a6830 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2228691870 0 0) 0x7fe12c138a00 con 0x7fe12c0694b0 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe12c138ce0 con 0x7fe12c0694b0 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe11c0047e0 con 0x7fe12c0694b0 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3827523999 0 0) 0x7fe11c001500 con 0x7fe12c0a46a0 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.1:6791/0 conn(0x7fe12c0a6830 legacy=0x7fe12c1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.82:6789/0 conn(0x7fe12c0694b0 legacy=0x7fe12c1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 monclient: found mon.d 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c136100 con 0x7fe12c0a46a0 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 5 monclient: authenticate success, global_id 4335 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 10 monclient: _renew_subs 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe12c0aa0a0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 10 monclient: _renew_subs 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe12c137a30 con 0x7fe12c0a46a0 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe13206e640 1 librados: init done 2024-08-19T09:50:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe114003b70 con 0x7fe12c0a46a0 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe1140040e0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe12a7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.766+0000 7fe128ff9640 10 monclient: dump: 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe114004280 con 0x7fe12c0a46a0 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe13206e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe13206e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe13206e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe110005080 con 0x7fe12c0a46a0 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fe11404dac0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe128ff9640 10 client.4335.objecter ms_dispatch 0x7fe12c000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe128ff9640 3 client.4335.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe128ff9640 3 client.4335.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe128ff9640 20 client.4335.objecter dump_active .. 0 homeless 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.767+0000 7fe12b7fe640 10 monclient: get_auth_request con 0x7fe10c0419a0 auth_method 0 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.769+0000 7fe128ff9640 10 client.4335.objecter ms_handle_connect 0x7fe10c0419a0 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.772+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe114017640 con 0x7fe12c0a46a0 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.772+0000 7fe128ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.772+0000 7fe128ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:23.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe13206e640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-08-19T09:50:23.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe13206e640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-08-19T09:50:23.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe13206e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:23.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fe110005b80 con 0x7fe12c0a46a0 2024-08-19T09:50:23.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe128ff9640 1 -- 172.21.15.1:0/3014476514 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (unknown 3412311620 0 265718887) 0x7fe11401fed0 con 0x7fe12c0a46a0 2024-08-19T09:50:23.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe128ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-08-19T09:50:23.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.955+0000 7fe128ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:23.957 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 8 2024-08-19T09:50:23.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 10 librados: watch_flush enter 2024-08-19T09:50:23.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 10 librados: watch_flush exit 2024-08-19T09:50:23.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 20 client.4335.objecter shutdown clearing up homeless session... 2024-08-19T09:50:23.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 10 client.4335.objecter successfully canceled tick 2024-08-19T09:50:23.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe10c0419a0 legacy=0x7fe10c043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 10 monclient: shutdown 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.957+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 >> v1:172.21.15.1:6792/0 conn(0x7fe12c0a46a0 legacy=0x7fe12c1359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.958+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 shutdown_connections 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.958+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 >> 172.21.15.1:0/3014476514 conn(0x7fe12c0a9020 msgr2=0x7fe12c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.958+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 shutdown_connections 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.958+0000 7fe13206e640 1 -- 172.21.15.1:0/3014476514 wait complete. 2024-08-19T09:50:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:23.958+0000 7fe13206e640 1 librados: shutdown 2024-08-19T09:50:23.969 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-08-19T09:50:23.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.066+0000 7ff0169d9640 10 monclient: get_monmap_and_config 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.066+0000 7ff0169d9640 10 monclient: build_initial_monmap 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.066+0000 7ff0169d9640 10 monclient: monmap: 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:24.067886+0000 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:24.067886+0000 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:24.069 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.067+0000 7ff0169d9640 1 Processor -- start 2024-08-19T09:50:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.067+0000 7ff0169d9640 1 -- start start 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.067+0000 7ff0169d9640 10 monclient: init 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient: _add_conns ranks=[5,1,8,4,3,0,6,7,2] 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient(hunting): picked mon.noname-h con 0x7ff0100a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient(hunting): picked mon.noname-f con 0x7ff0100a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient(hunting): picked mon.noname-e con 0x7ff01013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0100a3b40 con 0x7ff0100a6ba0 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0100a8130 con 0x7ff0100a67c0 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff01006ede0 con 0x7ff01013f5c0 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff0169d9640 10 monclient(hunting): authenticate will time out at 680.144297s 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff014f76640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff01013f5c0 0x7ff0101419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44772/0 (socket says 172.21.15.1:44772) 2024-08-19T09:50:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.068+0000 7ff014f76640 1 -- 172.21.15.1:0/3976366530 learned_addr learned my addr 172.21.15.1:0/3976366530 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff014f76640 1 -- 172.21.15.1:0/3976366530 >> v1:172.21.15.1:6793/0 conn(0x7ff01013f5c0 legacy=0x7ff0101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00f7fe640 1 -- 172.21.15.1:0/3976366530 >> v1:172.21.15.82:6789/0 conn(0x7ff0100a6ba0 legacy=0x7ff0100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 103024489 0 0) 0x7ff01006ede0 con 0x7ff01013f5c0 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): my global_id is 4418 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00ffff640 1 -- 172.21.15.1:0/3976366530 >> v1:172.21.15.82:6791/0 conn(0x7ff0100a67c0 legacy=0x7ff0100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feff8003520 con 0x7ff01013f5c0 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875795216 0 0) 0x7ff0100a3b40 con 0x7ff0100a6ba0 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): my global_id is 4357 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff01006ede0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569232377 0 0) 0x7ff0100a8130 con 0x7ff0100a67c0 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 10 monclient(hunting): my global_id is 4460 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0100a3b40 con 0x7ff0100a67c0 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.070+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3473732675 0 0) 0x7feff8003520 con 0x7ff01013f5c0 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 >> v1:172.21.15.82:6791/0 conn(0x7ff0100a67c0 legacy=0x7ff0100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 >> v1:172.21.15.82:6789/0 conn(0x7ff0100a6ba0 legacy=0x7ff0100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 monclient: found mon.noname-e 2024-08-19T09:50:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0100a92a0 con 0x7ff01013f5c0 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: _finish_auth 0 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff008001e50 con 0x7ff01013f5c0 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 5 monclient: authenticate success, global_id 4418 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: dump: 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.079 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff008004c20 con 0x7ff01013f5c0 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 1 -- 172.21.15.1:0/3976366530 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff008005190 con 0x7ff01013f5c0 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 4 set_mon_vals no callback set 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff00effd640 10 monclient: dump: 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 10 monclient: shutdown 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 1 -- 172.21.15.1:0/3976366530 >> v1:172.21.15.1:6793/0 conn(0x7ff01013f5c0 legacy=0x7ff0101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 1 -- 172.21.15.1:0/3976366530 shutdown_connections 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 1 -- 172.21.15.1:0/3976366530 >> 172.21.15.1:0/3976366530 conn(0x7ff0100ac4b0 msgr2=0x7ff0100a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 1 -- 172.21.15.1:0/3976366530 shutdown_connections 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.071+0000 7ff0169d9640 1 -- 172.21.15.1:0/3976366530 wait complete. 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 10 monclient: build_initial_monmap 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 10 monclient: monmap: 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 1 librados: starting msgr at 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 1 librados: starting objecter 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 1 Processor -- start 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 1 -- start start 2024-08-19T09:50:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 1 librados: setting wanted keys 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 1 librados: calling monclient init 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.072+0000 7ff0169d9640 10 monclient: init 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient: _add_conns ranks=[4,0,2,6,5,7,8,3,1] 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff0100a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff0100a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff01013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff01013e3a0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0101395a0 con 0x7ff01013f5c0 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff010139b50 con 0x7ff0100a67c0 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff0169d9640 10 monclient(hunting): authenticate will time out at 680.149245s 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff00f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff0100a6ba0 0x7ff010138e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37248/0 (socket says 172.21.15.1:37248) 2024-08-19T09:50:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff00ffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff0100a67c0 0x7ff010138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57528/0 (socket says 172.21.15.1:57528) 2024-08-19T09:50:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.073+0000 7ff00f7fe640 1 -- 172.21.15.1:0/3291156829 learned_addr learned my addr 172.21.15.1:0/3291156829 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00ffff640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.1:6791/0 conn(0x7ff0100a67c0 legacy=0x7ff010138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 client.?.objecter ms_handle_connect 0x7ff0100a67c0 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00f7fe640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.1:6789/0 conn(0x7ff0100a6ba0 legacy=0x7ff010138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 client.?.objecter ms_handle_connect 0x7ff0100a6ba0 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff014f76640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.1:6790/0 conn(0x7ff01013f5c0 legacy=0x7ff0101d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 client.?.objecter ms_handle_connect 0x7ff01013f5c0 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330258599 0 0) 0x7ff01013e3a0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): my global_id is 4401 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feffc004eb0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 173579019 0 0) 0x7ff010139b50 con 0x7ff0100a67c0 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): my global_id is 4414 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff01013e3a0 con 0x7ff0100a67c0 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161218798 0 0) 0x7ff0101395a0 con 0x7ff01013f5c0 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): my global_id is 4385 2024-08-19T09:50:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff010139b50 con 0x7ff01013f5c0 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877697040 0 0) 0x7feffc004eb0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0101395a0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff004003610 con 0x7ff0100a6ba0 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): dump: 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.075+0000 7ff00cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2138464912 0 0) 0x7ff010139b50 con 0x7ff01013f5c0 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feffc004eb0 con 0x7ff01013f5c0 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 382449369 0 0) 0x7ff01013e3a0 con 0x7ff0100a67c0 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff010139b50 con 0x7ff0100a67c0 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0080020a0 con 0x7ff01013f5c0 2024-08-19T09:50:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient(hunting): dump: 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff000003610 con 0x7ff0100a67c0 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient(hunting): dump: 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1073945569 0 0) 0x7ff0101395a0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.1:6791/0 conn(0x7ff0100a67c0 legacy=0x7ff010138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.1:6790/0 conn(0x7ff01013f5c0 legacy=0x7ff0101d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 monclient: found mon.a 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01013beb0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 5 monclient: authenticate success, global_id 4401 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 10 monclient: _renew_subs 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff01013b610 con 0x7ff0100a6ba0 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 10 monclient: _renew_subs 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0100a28d0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff0169d9640 1 librados: init done 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff004003ba0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff004004090 con 0x7ff0100a6ba0 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00e7fc640 4 set_mon_vals no callback set 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.076+0000 7ff00cff9640 10 monclient: dump: 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff0040042e0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff0169d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff0169d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff0169d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff0005080 con 0x7ff0100a6ba0 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7ff00404dba0 con 0x7ff0100a6ba0 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00cff9640 10 client.4401.objecter ms_dispatch 0x7ff010000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00cff9640 3 client.4401.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00cff9640 3 client.4401.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00cff9640 20 client.4401.objecter dump_active .. 0 homeless 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.077+0000 7ff00ffff640 10 monclient: get_auth_request con 0x7feffc0419e0 auth_method 0 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.080+0000 7ff00cff9640 10 client.4401.objecter ms_handle_connect 0x7feffc0419e0 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.081+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff004017600 con 0x7ff0100a6ba0 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.081+0000 7ff00cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.081+0000 7ff00cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:24.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.260+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feff0000b90 con 0x7feffc0419e0 2024-08-19T09:50:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.261+0000 7ff00cff9640 1 -- 172.21.15.1:0/3291156829 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34092 (unknown 2965378022 0 2476224230) 0x7feff0000b90 con 0x7feffc0419e0 2024-08-19T09:50:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:24.263 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T09:50:24.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.263+0000 7ff0169d9640 10 librados: watch_flush enter 2024-08-19T09:50:24.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.263+0000 7ff0169d9640 10 librados: watch_flush exit 2024-08-19T09:50:24.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.263+0000 7ff0169d9640 20 client.4401.objecter shutdown clearing up homeless session... 2024-08-19T09:50:24.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.263+0000 7ff0169d9640 10 client.4401.objecter successfully canceled tick 2024-08-19T09:50:24.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.82:6812/2549104415 conn(0x7feffc0419e0 legacy=0x7feffc043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 10 monclient: shutdown 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 >> v1:172.21.15.1:6789/0 conn(0x7ff0100a6ba0 legacy=0x7ff010138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 shutdown_connections 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 >> 172.21.15.1:0/3291156829 conn(0x7ff0100ac4b0 msgr2=0x7ff010141da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 shutdown_connections 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 -- 172.21.15.1:0/3291156829 wait complete. 2024-08-19T09:50:24.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.264+0000 7ff0169d9640 1 librados: shutdown 2024-08-19T09:50:24.275 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-08-19T09:50:23.730313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172536,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058504,"statfs":{"total":579820584960,"available":579643908096,"internally_reserved":0,"allocated":2056192,"data_stored":1366882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.972108"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768240+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T09:50:17.768240+0000","last_peered":"2024-08-19T09:50:17.768240+0000","last_clean":"2024-08-19T09:50:17.768240+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T09:50:17.768240+0000","last_undegraded":"2024-08-19T09:50:17.768240+0000","last_fullsized":"2024-08-19T09:50:17.768240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769663+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T09:50:17.769663+0000","last_peered":"2024-08-19T09:50:17.769663+0000","last_clean":"2024-08-19T09:50:17.769663+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T09:50:17.769663+0000","last_undegraded":"2024-08-19T09:50:17.769663+0000","last_fullsized":"2024-08-19T09:50:17.769663+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768483+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T09:50:17.768483+0000","last_peered":"2024-08-19T09:50:17.768483+0000","last_clean":"2024-08-19T09:50:17.768483+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T09:50:17.768483+0000","last_undegraded":"2024-08-19T09:50:17.768483+0000","last_fullsized":"2024-08-19T09:50:17.768483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769608+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T09:50:17.769608+0000","last_peered":"2024-08-19T09:50:17.769608+0000","last_clean":"2024-08-19T09:50:17.769608+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T09:50:17.769608+0000","last_undegraded":"2024-08-19T09:50:17.769608+0000","last_fullsized":"2024-08-19T09:50:17.769608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768001+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T09:50:17.768001+0000","last_peered":"2024-08-19T09:50:17.768001+0000","last_clean":"2024-08-19T09:50:17.768001+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T09:50:17.768001+0000","last_undegraded":"2024-08-19T09:50:17.768001+0000","last_fullsized":"2024-08-19T09:50:17.768001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767693+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T09:50:17.767693+0000","last_peered":"2024-08-19T09:50:17.767693+0000","last_clean":"2024-08-19T09:50:17.767693+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T09:50:17.767693+0000","last_undegraded":"2024-08-19T09:50:17.767693+0000","last_fullsized":"2024-08-19T09:50:17.767693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768191+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T09:50:17.768191+0000","last_peered":"2024-08-19T09:50:17.768191+0000","last_clean":"2024-08-19T09:50:17.768191+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T09:50:17.768191+0000","last_undegraded":"2024-08-19T09:50:17.768191+0000","last_fullsized":"2024-08-19T09:50:17.768191+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768119+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T09:50:17.768119+0000","last_peered":"2024-08-19T09:50:17.768119+0000","last_clean":"2024-08-19T09:50:17.768119+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T09:50:17.768119+0000","last_undegraded":"2024-08-19T09:50:17.768119+0000","last_fullsized":"2024-08-19T09:50:17.768119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767720+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T09:50:17.767720+0000","last_peered":"2024-08-19T09:50:17.767720+0000","last_clean":"2024-08-19T09:50:17.767720+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T09:50:17.767720+0000","last_undegraded":"2024-08-19T09:50:17.767720+0000","last_fullsized":"2024-08-19T09:50:17.767720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T09:50:24.276 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T09:50:24.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.373+0000 7fbbd539f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.373+0000 7fbbd539f640 10 monclient: build_initial_monmap 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.373+0000 7fbbd539f640 10 monclient: monmap: 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:24.374217+0000 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:24.374217+0000 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:24.375 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.373+0000 7fbbd539f640 1 Processor -- start 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.373+0000 7fbbd539f640 1 -- start start 2024-08-19T09:50:24.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.373+0000 7fbbd539f640 10 monclient: init 2024-08-19T09:50:24.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:24.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient: _add_conns ranks=[8,7,6,5,2,3,1,0,4] 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient(hunting): picked mon.noname-e con 0x7fbbd00a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient(hunting): picked mon.noname-i con 0x7fbbd00694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient(hunting): picked mon.noname-d con 0x7fbbd00a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd00a3c70 con 0x7fbbd00a46a0 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd00a33d0 con 0x7fbbd00694b0 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd00a36b0 con 0x7fbbd00a6830 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbd539f640 10 monclient(hunting): authenticate will time out at 680.450409s 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbcf7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fbbd00a46a0 0x7fbbd0141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52942/0 (socket says 172.21.15.1:52942) 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.374+0000 7fbbcf7fe640 1 -- 172.21.15.1:0/1229262939 learned_addr learned my addr 172.21.15.1:0/1229262939 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcf7fe640 1 -- 172.21.15.1:0/1229262939 >> v1:172.21.15.1:6792/0 conn(0x7fbbd00a46a0 legacy=0x7fbbd0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbceffd640 1 -- 172.21.15.1:0/1229262939 >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6830 legacy=0x7fbbd00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3078720270 0 0) 0x7fbbd00a3c70 con 0x7fbbd00a46a0 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): my global_id is 4344 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbce7fc640 1 -- 172.21.15.1:0/1229262939 >> v1:172.21.15.82:6792/0 conn(0x7fbbd00694b0 legacy=0x7fbbd00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbbc003520 con 0x7fbbd00a46a0 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594595109 0 0) 0x7fbbd00a36b0 con 0x7fbbd00a6830 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 10 monclient(hunting): my global_id is 4427 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.376+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd00a3c70 con 0x7fbbd00a6830 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3768933130 0 0) 0x7fbbbc003520 con 0x7fbbd00a46a0 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6830 legacy=0x7fbbd00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 >> v1:172.21.15.82:6792/0 conn(0x7fbbd00694b0 legacy=0x7fbbd00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 monclient: found mon.noname-d 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd006dbb0 con 0x7fbbd00a46a0 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc0001ae0 con 0x7fbbd00a46a0 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 5 monclient: authenticate success, global_id 4344 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: dump: 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbc00048a0 con 0x7fbbd00a46a0 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 1 -- 172.21.15.1:0/1229262939 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc0004e10 con 0x7fbbd00a46a0 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 4 set_mon_vals no callback set 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbcdffb640 10 monclient: dump: 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.385 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 10 monclient: shutdown 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 1 -- 172.21.15.1:0/1229262939 >> v1:172.21.15.1:6792/0 conn(0x7fbbd00a46a0 legacy=0x7fbbd0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 1 -- 172.21.15.1:0/1229262939 shutdown_connections 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 1 -- 172.21.15.1:0/1229262939 >> 172.21.15.1:0/1229262939 conn(0x7fbbd00a9020 msgr2=0x7fbbd0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 1 -- 172.21.15.1:0/1229262939 shutdown_connections 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.377+0000 7fbbd539f640 1 -- 172.21.15.1:0/1229262939 wait complete. 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 10 monclient: build_initial_monmap 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 10 monclient: monmap: 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:24.386 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 1 librados: starting msgr at 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 1 librados: starting objecter 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 1 Processor -- start 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 1 -- start start 2024-08-19T09:50:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 1 librados: setting wanted keys 2024-08-19T09:50:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 1 librados: calling monclient init 2024-08-19T09:50:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.378+0000 7fbbd539f640 10 monclient: init 2024-08-19T09:50:24.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:24.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient: _add_conns ranks=[5,7,0,2,8,3,4,6,1] 2024-08-19T09:50:24.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient(hunting): picked mon.noname-h con 0x7fbbd00694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:24.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient(hunting): picked mon.noname-i con 0x7fbbd00a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:24.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient(hunting): picked mon.noname-a con 0x7fbbd00a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:24.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd0138a10 con 0x7fbbd00a6830 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd01429b0 con 0x7fbbd00694b0 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd00a9ec0 con 0x7fbbd00a46a0 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbd539f640 10 monclient(hunting): authenticate will time out at 680.455449s 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbcf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbbd00a6830 0x7fbbd01d6400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37262/0 (socket says 172.21.15.1:37262) 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.379+0000 7fbbcf7fe640 1 -- 172.21.15.1:0/3007245272 learned_addr learned my addr 172.21.15.1:0/3007245272 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbcf7fe640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.1:6789/0 conn(0x7fbbd00a6830 legacy=0x7fbbd01d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 client.?.objecter ms_handle_connect 0x7fbbd00a6830 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 client.?.objecter ms_handle_connect 0x7fbbd00a46a0 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbce7fc640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.82:6792/0 conn(0x7fbbd00a46a0 legacy=0x7fbbd01353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 client.?.objecter ms_handle_connect 0x7fbbd00694b0 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbceffd640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.82:6791/0 conn(0x7fbbd00694b0 legacy=0x7fbbd0134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257400803 0 0) 0x7fbbd0138a10 con 0x7fbbd00a6830 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 10 monclient(hunting): my global_id is 4410 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.381+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb0004eb0 con 0x7fbbd00a6830 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2213790046 0 0) 0x7fbbd00a9ec0 con 0x7fbbd00a46a0 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): my global_id is 4390 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd0138a10 con 0x7fbbd00a46a0 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2938260206 0 0) 0x7fbbb0004eb0 con 0x7fbbd00a6830 2024-08-19T09:50:24.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbd00a9ec0 con 0x7fbbd00a6830 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170827825 0 0) 0x7fbbd01429b0 con 0x7fbbd00694b0 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): my global_id is 4469 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb0004eb0 con 0x7fbbd00694b0 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc0004d90 con 0x7fbbd00a6830 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): dump: 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.395 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2253768728 0 0) 0x7fbbd00a9ec0 con 0x7fbbd00a6830 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.82:6792/0 conn(0x7fbbd00a46a0 legacy=0x7fbbd01353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.82:6791/0 conn(0x7fbbd00694b0 legacy=0x7fbbd0134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 monclient: found mon.a 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd00689e0 con 0x7fbbd00a6830 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 5 monclient: authenticate success, global_id 4410 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 10 monclient: _renew_subs 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd0135af0 con 0x7fbbd00a6830 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 10 monclient: _renew_subs 2024-08-19T09:50:24.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbd0137400 con 0x7fbbd00a6830 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbd539f640 1 librados: init done 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbc00048a0 con 0x7fbbd00a6830 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc0001b00 con 0x7fbbd00a6830 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbcd7fa640 4 set_mon_vals no callback set 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.382+0000 7fbbc77fe640 10 monclient: dump: 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.398 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbbc0011a00 con 0x7fbbd00a6830 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbd539f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbd539f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbd539f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb4005080 con 0x7fbbd00a6830 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fbbc004da30 con 0x7fbbd00a6830 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbc77fe640 10 client.4410.objecter ms_dispatch 0x7fbbd0000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbc77fe640 3 client.4410.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbc77fe640 3 client.4410.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbc77fe640 20 client.4410.objecter dump_active .. 0 homeless 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.383+0000 7fbbceffd640 10 monclient: get_auth_request con 0x7fbbb0041090 auth_method 0 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.386+0000 7fbbc77fe640 10 client.4410.objecter ms_handle_connect 0x7fbbb0041090 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.387+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fbbc0017620 con 0x7fbbd00a6830 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.387+0000 7fbbc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.387+0000 7fbbc77fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:24.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.566+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbb4000b90 con 0x7fbbb0041090 2024-08-19T09:50:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.568+0000 7fbbc77fe640 1 -- 172.21.15.1:0/3007245272 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34092 (unknown 2965378022 0 2476224230) 0x7fbbb4000b90 con 0x7fbbb0041090 2024-08-19T09:50:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T09:50:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T09:50:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 10 librados: watch_flush enter 2024-08-19T09:50:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 10 librados: watch_flush exit 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 20 client.4410.objecter shutdown clearing up homeless session... 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 10 client.4410.objecter successfully canceled tick 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbbb0041090 legacy=0x7fbbb0043550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 10 monclient: shutdown 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 >> v1:172.21.15.1:6789/0 conn(0x7fbbd00a6830 legacy=0x7fbbd01d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 shutdown_connections 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 >> 172.21.15.1:0/3007245272 conn(0x7fbbd00a9020 msgr2=0x7fbbd0133790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 shutdown_connections 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 -- 172.21.15.1:0/3007245272 wait complete. 2024-08-19T09:50:24.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.569+0000 7fbbd539f640 1 librados: shutdown 2024-08-19T09:50:24.581 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-08-19T09:50:23.730313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172536,"kb_used_data":2008,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058504,"statfs":{"total":579820584960,"available":579643908096,"internally_reserved":0,"allocated":2056192,"data_stored":1366882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.972108"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768240+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T09:50:17.768240+0000","last_peered":"2024-08-19T09:50:17.768240+0000","last_clean":"2024-08-19T09:50:17.768240+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T09:50:17.768240+0000","last_undegraded":"2024-08-19T09:50:17.768240+0000","last_fullsized":"2024-08-19T09:50:17.768240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769663+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T09:50:17.769663+0000","last_peered":"2024-08-19T09:50:17.769663+0000","last_clean":"2024-08-19T09:50:17.769663+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T09:50:17.769663+0000","last_undegraded":"2024-08-19T09:50:17.769663+0000","last_fullsized":"2024-08-19T09:50:17.769663+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768483+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T09:50:17.768483+0000","last_peered":"2024-08-19T09:50:17.768483+0000","last_clean":"2024-08-19T09:50:17.768483+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T09:50:17.768483+0000","last_undegraded":"2024-08-19T09:50:17.768483+0000","last_fullsized":"2024-08-19T09:50:17.768483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.769608+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T09:50:17.769608+0000","last_peered":"2024-08-19T09:50:17.769608+0000","last_clean":"2024-08-19T09:50:17.769608+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T09:50:17.769608+0000","last_undegraded":"2024-08-19T09:50:17.769608+0000","last_fullsized":"2024-08-19T09:50:17.769608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768001+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T09:50:17.768001+0000","last_peered":"2024-08-19T09:50:17.768001+0000","last_clean":"2024-08-19T09:50:17.768001+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T09:50:17.768001+0000","last_undegraded":"2024-08-19T09:50:17.768001+0000","last_fullsized":"2024-08-19T09:50:17.768001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767693+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T09:50:17.767693+0000","last_peered":"2024-08-19T09:50:17.767693+0000","last_clean":"2024-08-19T09:50:17.767693+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T09:50:17.767693+0000","last_undegraded":"2024-08-19T09:50:17.767693+0000","last_fullsized":"2024-08-19T09:50:17.767693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768191+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T09:50:17.768191+0000","last_peered":"2024-08-19T09:50:17.768191+0000","last_clean":"2024-08-19T09:50:17.768191+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T09:50:17.768191+0000","last_undegraded":"2024-08-19T09:50:17.768191+0000","last_fullsized":"2024-08-19T09:50:17.768191+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.768119+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T09:50:17.768119+0000","last_peered":"2024-08-19T09:50:17.768119+0000","last_clean":"2024-08-19T09:50:17.768119+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T09:50:17.768119+0000","last_undegraded":"2024-08-19T09:50:17.768119+0000","last_fullsized":"2024-08-19T09:50:17.768119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T09:50:17.767720+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T09:50:17.767720+0000","last_peered":"2024-08-19T09:50:17.767720+0000","last_clean":"2024-08-19T09:50:17.767720+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T09:50:17.767720+0000","last_undegraded":"2024-08-19T09:50:17.767720+0000","last_fullsized":"2024-08-19T09:50:17.767720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":580426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T09:50:24.581 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-08-19T09:50:24.582 DEBUG:tasks.mon_thrash:Yielding 2024-08-19T09:50:24.582 INFO:teuthology.run_tasks:Running task workunit... 2024-08-19T09:50:24.594 INFO:tasks.workunit:Pulling workunits from ref 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-19T09:50:24.594 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-19T09:50:24.595 DEBUG:teuthology.orchestra.run.smithi082:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-19T09:50:24.595 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-08-19T09:50:24.595 INFO:tasks.mon_thrash.mon_thrasher:seed: 1724061024, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-08-19T09:50:24.595 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:50:24.595 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:50:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T09:50:24.615 INFO:teuthology.orchestra.run.smithi082.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-19T09:50:24.615 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-19T09:50:24.673 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-19T09:50:24.673 DEBUG:teuthology.orchestra.run.smithi082:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-19T09:50:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.693+0000 7f6895398640 10 monclient: get_monmap_and_config 2024-08-19T09:50:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.693+0000 7f6895398640 10 monclient: build_initial_monmap 2024-08-19T09:50:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.693+0000 7f6895398640 10 monclient: monmap: 2024-08-19T09:50:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:24.694630+0000 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:24.694630+0000 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:24.695 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 1 Processor -- start 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 1 -- start start 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient: init 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient: _add_conns ranks=[5,3,4,0,2,1,6,7,8] 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient(hunting): picked mon.noname-h con 0x7f68900a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient(hunting): picked mon.noname-g con 0x7f68900a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient(hunting): picked mon.noname-c con 0x7f689013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68900a3b40 con 0x7f68900a6ba0 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68900a8130 con 0x7f689013f5c0 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689006ede0 con 0x7f68900a67c0 2024-08-19T09:50:24.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.694+0000 7f6895398640 10 monclient(hunting): authenticate will time out at 680.770923s 2024-08-19T09:50:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.695+0000 7f688f7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f689013f5c0 0x7f68901419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57532/0 (socket says 172.21.15.1:57532) 2024-08-19T09:50:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.695+0000 7f688f7fe640 1 -- 172.21.15.1:0/3159270825 learned_addr learned my addr 172.21.15.1:0/3159270825 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688f7fe640 1 -- 172.21.15.1:0/3159270825 >> v1:172.21.15.1:6791/0 conn(0x7f689013f5c0 legacy=0x7f68901419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3120479166 0 0) 0x7f68900a8130 con 0x7f689013f5c0 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient(hunting): my global_id is 4423 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6878003520 con 0x7f689013f5c0 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688effd640 1 -- 172.21.15.1:0/3159270825 >> v1:172.21.15.82:6791/0 conn(0x7f68900a67c0 legacy=0x7f68900a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688e7fc640 1 -- 172.21.15.1:0/3159270825 >> v1:172.21.15.82:6790/0 conn(0x7f68900a6ba0 legacy=0x7f68900a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3327953587 0 0) 0x7f6878003520 con 0x7f689013f5c0 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 >> v1:172.21.15.82:6791/0 conn(0x7f68900a67c0 legacy=0x7f68900a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 >> v1:172.21.15.82:6790/0 conn(0x7f68900a6ba0 legacy=0x7f68900a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 monclient: found mon.noname-c 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68900a92a0 con 0x7f689013f5c0 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6888001e60 con 0x7f689013f5c0 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f6895398640 5 monclient: authenticate success, global_id 4423 2024-08-19T09:50:24.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f6895398640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: dump: 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.703 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f6895398640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6888004c20 con 0x7f689013f5c0 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f6895398640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 1 -- 172.21.15.1:0/3159270825 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6888005150 con 0x7f689013f5c0 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f6895398640 4 set_mon_vals no callback set 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.697+0000 7f688dffb640 10 monclient: dump: 2024-08-19T09:50:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.705 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 10 monclient: shutdown 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 1 -- 172.21.15.1:0/3159270825 >> v1:172.21.15.1:6791/0 conn(0x7f689013f5c0 legacy=0x7f68901419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 1 -- 172.21.15.1:0/3159270825 shutdown_connections 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 1 -- 172.21.15.1:0/3159270825 >> 172.21.15.1:0/3159270825 conn(0x7f68900ac4b0 msgr2=0x7f68900a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 1 -- 172.21.15.1:0/3159270825 shutdown_connections 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 1 -- 172.21.15.1:0/3159270825 wait complete. 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 10 monclient: build_initial_monmap 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.698+0000 7f6895398640 10 monclient: monmap: 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:24.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 librados: starting msgr at 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 librados: starting objecter 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 Processor -- start 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 -- start start 2024-08-19T09:50:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 librados: setting wanted keys 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 librados: calling monclient init 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient: init 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient: _add_conns ranks=[5,2,4,6,7,8,1,0,3] 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient(hunting): picked mon.noname-h con 0x7f68900a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient(hunting): picked mon.noname-b con 0x7f68900a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient(hunting): picked mon.noname-c con 0x7f689013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689013b8a0 con 0x7f68900a6ba0 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689013d180 con 0x7f689013f5c0 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6890139b20 con 0x7f68900a67c0 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f6895398640 10 monclient(hunting): authenticate will time out at 680.775898s 2024-08-19T09:50:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f688f7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f689013f5c0 0x7f68901d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57544/0 (socket says 172.21.15.1:57544) 2024-08-19T09:50:24.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f688f7fe640 1 -- 172.21.15.1:0/2917737646 learned_addr learned my addr 172.21.15.1:0/2917737646 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:24.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.699+0000 7f688e7fc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f68900a6ba0 0x7f689013ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52834/0 (socket says 172.21.15.1:52834) 2024-08-19T09:50:24.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f688f7fe640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.1:6791/0 conn(0x7f689013f5c0 legacy=0x7f68901d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 client.?.objecter ms_handle_connect 0x7f689013f5c0 2024-08-19T09:50:24.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 client.?.objecter ms_handle_connect 0x7f68900a6ba0 2024-08-19T09:50:24.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f688e7fc640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.1:6790/0 conn(0x7f68900a6ba0 legacy=0x7f689013ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 client.?.objecter ms_handle_connect 0x7f68900a67c0 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f688effd640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.82:6791/0 conn(0x7f68900a67c0 legacy=0x7f689013c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877752507 0 0) 0x7f689013d180 con 0x7f689013f5c0 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): my global_id is 4432 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f687c004eb0 con 0x7f689013f5c0 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97015529 0 0) 0x7f689013b8a0 con 0x7f68900a6ba0 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): my global_id is 4394 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f689013d180 con 0x7f68900a6ba0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510917306 0 0) 0x7f6890139b20 con 0x7f68900a67c0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): my global_id is 4487 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f689013b8a0 con 0x7f68900a67c0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3260040745 0 0) 0x7f687c004eb0 con 0x7f689013f5c0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6890139b20 con 0x7f689013f5c0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f68880020b0 con 0x7f689013f5c0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): dump: 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.714 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1071858881 0 0) 0x7f689013d180 con 0x7f68900a6ba0 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f687c004eb0 con 0x7f68900a6ba0 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6884003610 con 0x7f68900a6ba0 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): dump: 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 843321492 0 0) 0x7f689013b8a0 con 0x7f68900a67c0 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f689013d180 con 0x7f68900a67c0 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6880001c60 con 0x7f68900a67c0 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient(hunting): dump: 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.718 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1250989912 0 0) 0x7f6890139b20 con 0x7f689013f5c0 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.82:6791/0 conn(0x7f68900a67c0 legacy=0x7f689013c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.1:6790/0 conn(0x7f68900a6ba0 legacy=0x7f689013ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 monclient: found mon.c 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689013be60 con 0x7f689013f5c0 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 5 monclient: authenticate success, global_id 4432 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 10 monclient: _renew_subs 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f689013b5c0 con 0x7f689013f5c0 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 10 monclient: _renew_subs 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f689013a250 con 0x7f689013f5c0 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f6895398640 1 librados: init done 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6888004c20 con 0x7f689013f5c0 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6888005700 con 0x7f689013f5c0 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f688d7fa640 4 set_mon_vals no callback set 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.702+0000 7f68777fe640 10 monclient: dump: 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.703+0000 7f6895398640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.703+0000 7f6895398640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.703+0000 7f6895398640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.703+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6888011b60 con 0x7f689013f5c0 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.703+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6870005080 con 0x7f689013f5c0 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.704+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f688804de00 con 0x7f689013f5c0 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.704+0000 7f68777fe640 10 client.4432.objecter ms_dispatch 0x7f6890000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.704+0000 7f68777fe640 3 client.4432.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:24.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.704+0000 7f68777fe640 3 client.4432.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.704+0000 7f68777fe640 20 client.4432.objecter dump_active .. 0 homeless 2024-08-19T09:50:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.704+0000 7f688effd640 10 monclient: get_auth_request con 0x7f687c041b50 auth_method 0 2024-08-19T09:50:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.706+0000 7f68777fe640 10 client.4432.objecter ms_handle_connect 0x7f687c041b50 2024-08-19T09:50:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.708+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6888017980 con 0x7f689013f5c0 2024-08-19T09:50:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.708+0000 7f68777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.708+0000 7f68777fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:24.740 INFO:tasks.workunit:timeout=3h 2024-08-19T09:50:24.740 INFO:tasks.workunit:cleanup=True 2024-08-19T09:50:24.740 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-19T09:50:24.798 INFO:tasks.workunit.client.0.smithi082.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-19T09:50:24.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.966+0000 7f6895398640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:50:24.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.966+0000 7f6895398640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:50:24.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.966+0000 7f6895398640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:24.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.966+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6870005370 con 0x7f689013f5c0 2024-08-19T09:50:24.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.966+0000 7f68777fe640 1 -- 172.21.15.1:0/2917737646 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 571132268) 0x7f6888017280 con 0x7f689013f5c0 2024-08-19T09:50:24.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.967+0000 7f68777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:50:24.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.967+0000 7f68777fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:24.968 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":4,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.968+0000 7f6895398640 10 librados: watch_flush enter 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.968+0000 7f6895398640 10 librados: watch_flush exit 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.968+0000 7f6895398640 20 client.4432.objecter shutdown clearing up homeless session... 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.968+0000 7f6895398640 10 client.4432.objecter successfully canceled tick 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.82:6812/2549104415 conn(0x7f687c041b50 legacy=0x7f687c044010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 10 monclient: shutdown 2024-08-19T09:50:24.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:24.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 >> v1:172.21.15.1:6791/0 conn(0x7f689013f5c0 legacy=0x7f68901d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:24.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 shutdown_connections 2024-08-19T09:50:24.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 >> 172.21.15.1:0/2917737646 conn(0x7f68900ac4b0 msgr2=0x7f689013eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:24.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 shutdown_connections 2024-08-19T09:50:24.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 -- 172.21.15.1:0/2917737646 wait complete. 2024-08-19T09:50:24.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:24.969+0000 7f6895398640 1 librados: shutdown 2024-08-19T09:50:24.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:50:24.980 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T09:50:24.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:50:24.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:50:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.078+0000 7f149be20640 10 monclient: get_monmap_and_config 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.078+0000 7f149be20640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.078+0000 7f149be20640 10 monclient: monmap: 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:25.079725+0000 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:25.079725+0000 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.079+0000 7f149be20640 1 Processor -- start 2024-08-19T09:50:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.079+0000 7f149be20640 1 -- start start 2024-08-19T09:50:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.079+0000 7f149be20640 10 monclient: init 2024-08-19T09:50:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.079+0000 7f149be20640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.079+0000 7f149be20640 10 monclient: _add_conns ranks=[4,6,3,2,5,0,1,7,8] 2024-08-19T09:50:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 10 monclient(hunting): picked mon.noname-c con 0x7f14940a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 10 monclient(hunting): picked mon.noname-d con 0x7f14940a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 10 monclient(hunting): picked mon.noname-g con 0x7f149413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14940a3b40 con 0x7f149413f5c0 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14940a8130 con 0x7f14940a67c0 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149406ede0 con 0x7f14940a6ba0 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f149be20640 10 monclient(hunting): authenticate will time out at 681.156073s 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f1499bbc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f14940a67c0 0x7f14940a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57560/0 (socket says 172.21.15.1:57560) 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f14993bb640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f14940a6ba0 0x7f14940a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52948/0 (socket says 172.21.15.1:52948) 2024-08-19T09:50:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.080+0000 7f1499bbc640 1 -- 172.21.15.1:0/162678772 learned_addr learned my addr 172.21.15.1:0/162678772 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f14993bb640 1 -- 172.21.15.1:0/162678772 >> v1:172.21.15.1:6792/0 conn(0x7f14940a6ba0 legacy=0x7f14940a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1499bbc640 1 -- 172.21.15.1:0/162678772 >> v1:172.21.15.1:6791/0 conn(0x7f14940a67c0 legacy=0x7f14940a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f149a3bd640 1 -- 172.21.15.1:0/162678772 >> v1:172.21.15.82:6790/0 conn(0x7f149413f5c0 legacy=0x7f14941419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2419987093 0 0) 0x7f149406ede0 con 0x7f14940a6ba0 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): my global_id is 4353 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f147c003520 con 0x7f14940a6ba0 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 62574521 0 0) 0x7f14940a8130 con 0x7f14940a67c0 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): my global_id is 4441 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f149406ede0 con 0x7f14940a67c0 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278339051 0 0) 0x7f14940a3b40 con 0x7f149413f5c0 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 10 monclient(hunting): my global_id is 4431 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.082+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14940a8130 con 0x7f149413f5c0 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2429771798 0 0) 0x7f147c003520 con 0x7f14940a6ba0 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 >> v1:172.21.15.1:6791/0 conn(0x7f14940a67c0 legacy=0x7f14940a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 >> v1:172.21.15.82:6790/0 conn(0x7f149413f5c0 legacy=0x7f14941419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 monclient: found mon.noname-d 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14940a92a0 con 0x7f14940a6ba0 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 5 monclient: authenticate success, global_id 4353 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1488001ad0 con 0x7f14940a6ba0 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: dump: 2024-08-19T09:50:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14880047e0 con 0x7f14940a6ba0 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 1 -- 172.21.15.1:0/162678772 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1488004d50 con 0x7f14940a6ba0 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 4 set_mon_vals no callback set 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f1498bba640 10 monclient: dump: 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 10 monclient: shutdown 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 1 -- 172.21.15.1:0/162678772 >> v1:172.21.15.1:6792/0 conn(0x7f14940a6ba0 legacy=0x7f14940a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 1 -- 172.21.15.1:0/162678772 shutdown_connections 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 1 -- 172.21.15.1:0/162678772 >> 172.21.15.1:0/162678772 conn(0x7f14940ac4b0 msgr2=0x7f14940a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 1 -- 172.21.15.1:0/162678772 shutdown_connections 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.083+0000 7f149be20640 1 -- 172.21.15.1:0/162678772 wait complete. 2024-08-19T09:50:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 10 monclient: monmap: 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 1 librados: starting msgr at 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 1 librados: starting objecter 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 1 Processor -- start 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 1 -- start start 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 1 librados: setting wanted keys 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 1 librados: calling monclient init 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.084+0000 7f149be20640 10 monclient: init 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient: _add_conns ranks=[8,5,0,4,7,6,1,2,3] 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient(hunting): picked mon.noname-e con 0x7f14940a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient(hunting): picked mon.noname-h con 0x7f14940a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient(hunting): picked mon.noname-a con 0x7f149413f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149413a560 con 0x7f149413f5c0 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149413a840 con 0x7f14940a6ba0 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1494139b50 con 0x7f14940a67c0 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149be20640 10 monclient(hunting): authenticate will time out at 681.161365s 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f1499bbc640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f14940a67c0 0x7f149409fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44802/0 (socket says 172.21.15.1:44802) 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f149a3bd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f149413f5c0 0x7f14940a0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37266/0 (socket says 172.21.15.1:37266) 2024-08-19T09:50:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.085+0000 7f1499bbc640 1 -- 172.21.15.1:0/232965816 learned_addr learned my addr 172.21.15.1:0/232965816 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f1499bbc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6793/0 conn(0x7f14940a67c0 legacy=0x7f149409fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 client.?.objecter ms_handle_connect 0x7f14940a67c0 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 client.?.objecter ms_handle_connect 0x7f149413f5c0 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f149a3bd640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f149413f5c0 legacy=0x7f14940a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 client.?.objecter ms_handle_connect 0x7f14940a6ba0 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14993bb640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.82:6791/0 conn(0x7f14940a6ba0 legacy=0x7f14940a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3117056238 0 0) 0x7f1494139b50 con 0x7f14940a67c0 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): my global_id is 4436 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1480004eb0 con 0x7f14940a67c0 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015972842 0 0) 0x7f149413a560 con 0x7f149413f5c0 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): my global_id is 4419 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1494139b50 con 0x7f149413f5c0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324942682 0 0) 0x7f149413a840 con 0x7f14940a6ba0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 10 monclient(hunting): my global_id is 4496 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.087+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f149413a560 con 0x7f14940a6ba0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2422114270 0 0) 0x7f1494139b50 con 0x7f149413f5c0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f149413a840 con 0x7f149413f5c0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f148c001cf0 con 0x7f149413f5c0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): dump: 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.107 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1013428785 0 0) 0x7f1480004eb0 con 0x7f14940a67c0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1494139b50 con 0x7f14940a67c0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1940314555 0 0) 0x7f149413a560 con 0x7f14940a6ba0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1480004eb0 con 0x7f14940a6ba0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1484004b60 con 0x7f14940a67c0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): dump: 2024-08-19T09:50:25.108 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.109 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1488004cd0 con 0x7f14940a6ba0 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient(hunting): dump: 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.110 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 205378689 0 0) 0x7f149413a840 con 0x7f149413f5c0 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6793/0 conn(0x7f14940a67c0 legacy=0x7f149409fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.82:6791/0 conn(0x7f14940a6ba0 legacy=0x7f14940a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 monclient: found mon.a 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1494142c60 con 0x7f149413f5c0 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 5 monclient: authenticate success, global_id 4419 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 10 monclient: _renew_subs 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:25.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14940a2850 con 0x7f149413f5c0 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 10 monclient: _renew_subs 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1494139fa0 con 0x7f149413f5c0 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f149be20640 1 librados: init done 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f148c003a30 con 0x7f149413f5c0 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f148c003f20 con 0x7f149413f5c0 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f1493fff640 4 set_mon_vals no callback set 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:25.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.088+0000 7f14927fc640 10 monclient: dump: 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.114 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f149be20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f14740065c0 con 0x7f1474002860 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f148c0041c0 con 0x7f149413f5c0 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f148c04da10 con 0x7f149413f5c0 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14927fc640 10 client.4419.objecter ms_dispatch 0x7f1494000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14927fc640 3 client.4419.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14927fc640 3 client.4419.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14927fc640 20 client.4419.objecter dump_active .. 0 homeless 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.089+0000 7f14993bb640 10 monclient: get_auth_request con 0x7f14800419c0 auth_method 0 2024-08-19T09:50:25.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f1499bbc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f1474002860 legacy=0x7f1474004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 client.4419.objecter ms_handle_connect 0x7f1474002860 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 client.4419.objecter resend_mon_ops 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082222817 0 0) 0x7f14740065c0 con 0x7f1474002860 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 monclient(hunting): my global_id is 4428 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f149413a840 con 0x7f1474002860 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1177435642 0 0) 0x7f149413a840 con 0x7f1474002860 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14740065c0 con 0x7f1474002860 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1484005230 con 0x7f1474002860 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.091+0000 7f14927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:25.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 10 monclient: dump: 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.117 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.118 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.118 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.118 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 305900442 0 0) 0x7f14740065c0 con 0x7f1474002860 2024-08-19T09:50:25.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f147400a160 con 0x7f1474002860 2024-08-19T09:50:25.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 10 client.4419.objecter ms_handle_connect 0x7f14800419c0 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f14840074f0 con 0x7f1474002860 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f1474002860 legacy=0x7f1474004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.092+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f1474002860 legacy=0x7f1474004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.098+0000 7f149be20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.098+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1474009e70 con 0x7f1474009130 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.100+0000 7f14927fc640 10 client.4419.objecter ms_handle_connect 0x7f1474009130 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.100+0000 7f14927fc640 10 client.4419.objecter resend_mon_ops 2024-08-19T09:50:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.100+0000 7f1499bbc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f1474009130 legacy=0x7f14740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666256180 0 0) 0x7f1474009e70 con 0x7f1474009130 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient(hunting): my global_id is 4437 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14740065c0 con 0x7f1474009130 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 382071948 0 0) 0x7f14740065c0 con 0x7f1474009130 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1474009e70 con 0x7f1474009130 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1484006d60 con 0x7f1474009130 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient: dump: 2024-08-19T09:50:25.120 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.121 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3947849350 0 0) 0x7f1474009e70 con 0x7f1474009130 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f147400c410 con 0x7f1474009130 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2491870232) 0x7f1484006f10 con 0x7f1474009130 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f1474009130 legacy=0x7f14740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f14927fc640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f1474009130 legacy=0x7f14740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.122 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 31381, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 30, 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:25.144 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:25.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 10 librados: watch_flush enter 2024-08-19T09:50:25.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 10 librados: watch_flush exit 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 20 client.4419.objecter shutdown clearing up homeless session... 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 10 client.4419.objecter successfully canceled tick 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.82:6812/2549104415 conn(0x7f14800419c0 legacy=0x7f1480043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 10 monclient: shutdown 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 >> v1:172.21.15.1:6789/0 conn(0x7f149413f5c0 legacy=0x7f14940a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149a3bd640 1 -- 172.21.15.1:0/232965816 reap_dead start 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.101+0000 7f149a3bd640 1 -- 172.21.15.1:0/232965816 reap_dead start 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.102+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 shutdown_connections 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.102+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 >> 172.21.15.1:0/232965816 conn(0x7f14940ac4b0 msgr2=0x7f14940aa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.102+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 shutdown_connections 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.102+0000 7f149be20640 1 -- 172.21.15.1:0/232965816 wait complete. 2024-08-19T09:50:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.102+0000 7f149be20640 1 librados: shutdown 2024-08-19T09:50:25.148 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:25.148 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:50:25.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:50:25.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.245+0000 7f020e9a1640 10 monclient: get_monmap_and_config 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.245+0000 7f020e9a1640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.245+0000 7f020e9a1640 10 monclient: monmap: 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:25.247819+0000 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:25.247819+0000 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.249 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.246+0000 7f020e9a1640 1 Processor -- start 2024-08-19T09:50:25.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.246+0000 7f020e9a1640 1 -- start start 2024-08-19T09:50:25.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.246+0000 7f020e9a1640 10 monclient: init 2024-08-19T09:50:25.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient: _add_conns ranks=[4,0,6,5,3,1,7,8,2] 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient(hunting): picked mon.noname-c con 0x7f02080a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient(hunting): picked mon.noname-a con 0x7f02080a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient(hunting): picked mon.noname-d con 0x7f020813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02080a3b40 con 0x7f02080a6ba0 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02080a8130 con 0x7f02080a67c0 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020806ede0 con 0x7f020813f5c0 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f020e9a1640 10 monclient(hunting): authenticate will time out at 681.324195s 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f0207fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f02080a67c0 0x7f02080a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57572/0 (socket says 172.21.15.1:57572) 2024-08-19T09:50:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.247+0000 7f0207fff640 1 -- 172.21.15.1:0/2242670104 learned_addr learned my addr 172.21.15.1:0/2242670104 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f020cf3e640 1 -- 172.21.15.1:0/2242670104 >> v1:172.21.15.1:6792/0 conn(0x7f020813f5c0 legacy=0x7f02081419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0207fff640 1 -- 172.21.15.1:0/2242670104 >> v1:172.21.15.1:6791/0 conn(0x7f02080a67c0 legacy=0x7f02080a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f02077fe640 1 -- 172.21.15.1:0/2242670104 >> v1:172.21.15.1:6789/0 conn(0x7f02080a6ba0 legacy=0x7f02080a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392628705 0 0) 0x7f020806ede0 con 0x7f020813f5c0 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): my global_id is 4362 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01f0003520 con 0x7f020813f5c0 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155886125 0 0) 0x7f02080a8130 con 0x7f02080a67c0 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): my global_id is 4450 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f020806ede0 con 0x7f02080a67c0 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490107004 0 0) 0x7f02080a3b40 con 0x7f02080a6ba0 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient(hunting): my global_id is 4446 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02080a8130 con 0x7f02080a6ba0 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2402640458 0 0) 0x7f01f0003520 con 0x7f020813f5c0 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 >> v1:172.21.15.1:6791/0 conn(0x7f02080a67c0 legacy=0x7f02080a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 >> v1:172.21.15.1:6789/0 conn(0x7f02080a6ba0 legacy=0x7f02080a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 monclient: found mon.noname-d 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02080a92a0 con 0x7f020813f5c0 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0200001ea0 con 0x7f020813f5c0 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f0206ffd640 10 monclient: dump: 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f020e9a1640 5 monclient: authenticate success, global_id 4362 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.249+0000 7f020e9a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0200003990 con 0x7f020813f5c0 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f0206ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f0206ffd640 1 -- 172.21.15.1:0/2242670104 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0200003ec0 con 0x7f020813f5c0 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f0206ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 4 set_mon_vals no callback set 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f0206ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f0206ffd640 10 monclient: dump: 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 10 monclient: shutdown 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 1 -- 172.21.15.1:0/2242670104 >> v1:172.21.15.1:6792/0 conn(0x7f020813f5c0 legacy=0x7f02081419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 1 -- 172.21.15.1:0/2242670104 shutdown_connections 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 1 -- 172.21.15.1:0/2242670104 >> 172.21.15.1:0/2242670104 conn(0x7f02080ac4b0 msgr2=0x7f02080a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 1 -- 172.21.15.1:0/2242670104 shutdown_connections 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.250+0000 7f020e9a1640 1 -- 172.21.15.1:0/2242670104 wait complete. 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 10 monclient: monmap: 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.262 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 1 librados: starting msgr at 2024-08-19T09:50:25.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 1 librados: starting objecter 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 1 Processor -- start 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 1 -- start start 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 1 librados: setting wanted keys 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 1 librados: calling monclient init 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.251+0000 7f020e9a1640 10 monclient: init 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient: _add_conns ranks=[5,7,6,2,4,0,3,1,8] 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient(hunting): picked mon.noname-h con 0x7f02080a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient(hunting): picked mon.noname-i con 0x7f02080a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient(hunting): picked mon.noname-d con 0x7f020813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02080a2770 con 0x7f02080a67c0 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020813a7d0 con 0x7f020813f5c0 2024-08-19T09:50:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02080aa0f0 con 0x7f02080a6ba0 2024-08-19T09:50:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020e9a1640 10 monclient(hunting): authenticate will time out at 681.329192s 2024-08-19T09:50:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020cf3e640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f020813f5c0 0x7f02081d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52970/0 (socket says 172.21.15.1:52970) 2024-08-19T09:50:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.252+0000 7f020cf3e640 1 -- 172.21.15.1:0/2145074252 learned_addr learned my addr 172.21.15.1:0/2145074252 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f020cf3e640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6792/0 conn(0x7f020813f5c0 legacy=0x7f02081d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 client.?.objecter ms_handle_connect 0x7f020813f5c0 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 client.?.objecter ms_handle_connect 0x7f02080a6ba0 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f02077fe640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.82:6792/0 conn(0x7f02080a6ba0 legacy=0x7f020813a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 client.?.objecter ms_handle_connect 0x7f02080a67c0 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0207fff640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.82:6791/0 conn(0x7f02080a67c0 legacy=0x7f02081399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 237478627 0 0) 0x7f020813a7d0 con 0x7f020813f5c0 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): my global_id is 4371 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01f4004eb0 con 0x7f020813f5c0 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932445966 0 0) 0x7f02080a2770 con 0x7f02080a67c0 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): my global_id is 4505 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f020813a7d0 con 0x7f02080a67c0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1926335940 0 0) 0x7f02080aa0f0 con 0x7f02080a6ba0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): my global_id is 4399 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02080a2770 con 0x7f02080a6ba0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3442239159 0 0) 0x7f01f4004eb0 con 0x7f020813f5c0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02080aa0f0 con 0x7f020813f5c0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0200003e80 con 0x7f020813f5c0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): dump: 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.273 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1185223065 0 0) 0x7f02080aa0f0 con 0x7f020813f5c0 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.254+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.82:6792/0 conn(0x7f02080a6ba0 legacy=0x7f020813a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.82:6791/0 conn(0x7f02080a67c0 legacy=0x7f02081399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 1 monclient: found mon.d 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02080a9cc0 con 0x7f020813f5c0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 5 monclient: authenticate success, global_id 4371 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 10 monclient: _renew_subs 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f020813c840 con 0x7f020813f5c0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 10 monclient: _renew_subs 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f02080a1fa0 con 0x7f020813f5c0 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f020e9a1640 1 librados: init done 2024-08-19T09:50:25.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0200003990 con 0x7f020813f5c0 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0200004690 con 0x7f020813f5c0 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f02067fc640 4 set_mon_vals no callback set 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 10 monclient: dump: 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.255+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0200011e00 con 0x7f020813f5c0 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f020e9a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01e80065c0 con 0x7f01e8002860 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f020004de90 con 0x7f020813f5c0 2024-08-19T09:50:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f0204ff9640 10 client.4371.objecter ms_dispatch 0x7f0208000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f0204ff9640 3 client.4371.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f0204ff9640 3 client.4371.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f0204ff9640 20 client.4371.objecter dump_active .. 0 homeless 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.256+0000 7f02077fe640 10 monclient: get_auth_request con 0x7f01f4040c20 auth_method 0 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0207fff640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6790/0 conn(0x7f01e8002860 legacy=0x7f01e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 client.4371.objecter ms_handle_connect 0x7f01e8002860 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 client.4371.objecter resend_mon_ops 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030188971 0 0) 0x7f01e80065c0 con 0x7f01e8002860 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient(hunting): my global_id is 4403 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02080aa0f0 con 0x7f01e8002860 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 client.4371.objecter ms_handle_connect 0x7f01f4040c20 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2108021281 0 0) 0x7f02080aa0f0 con 0x7f01e8002860 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01e80065c0 con 0x7f01e8002860 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01f8004eb0 con 0x7f01e8002860 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 10 monclient: dump: 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2181853118 0 0) 0x7f01e80065c0 con 0x7f01e8002860 2024-08-19T09:50:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.258+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01e800a160 con 0x7f01e8002860 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.259+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f01f8007090 con 0x7f01e8002860 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.259+0000 7f0204ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.259+0000 7f0204ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.259+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6790/0 conn(0x7f01e8002860 legacy=0x7f01e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.259+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6790/0 conn(0x7f01e8002860 legacy=0x7f01e8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.265+0000 7f020e9a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.265+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01e8009e50 con 0x7f01e8009110 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0207fff640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6790/0 conn(0x7f01e8009110 legacy=0x7f01e80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 10 client.4371.objecter ms_handle_connect 0x7f01e8009110 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 10 client.4371.objecter resend_mon_ops 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955523569 0 0) 0x7f01e8009e50 con 0x7f01e8009110 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 10 monclient(hunting): my global_id is 4412 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.267+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e80065c0 con 0x7f01e8009110 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2807918991 0 0) 0x7f01e80065c0 con 0x7f01e8009110 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01e8009e50 con 0x7f01e8009110 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01f8005810 con 0x7f01e8009110 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 10 monclient: dump: 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3673295727 0 0) 0x7f01e8009e50 con 0x7f01e8009110 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01e800c3a0 con 0x7f01e8009110 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 660237543) 0x7f01f8006cc0 con 0x7f01e8009110 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6790/0 conn(0x7f01e8009110 legacy=0x7f01e80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f0204ff9640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6790/0 conn(0x7f01e8009110 legacy=0x7f01e80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 31559, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 30, 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:25.304 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 10 librados: watch_flush enter 2024-08-19T09:50:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 10 librados: watch_flush exit 2024-08-19T09:50:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 20 client.4371.objecter shutdown clearing up homeless session... 2024-08-19T09:50:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 10 client.4371.objecter successfully canceled tick 2024-08-19T09:50:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.82:6812/2549104415 conn(0x7f01f4040c20 legacy=0x7f01f40430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 10 monclient: shutdown 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 >> v1:172.21.15.1:6792/0 conn(0x7f020813f5c0 legacy=0x7f02081d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020cf3e640 1 -- 172.21.15.1:0/2145074252 reap_dead start 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.268+0000 7f020cf3e640 1 -- 172.21.15.1:0/2145074252 reap_dead start 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.269+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 shutdown_connections 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.269+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 >> 172.21.15.1:0/2145074252 conn(0x7f02080ac4b0 msgr2=0x7f02081419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.269+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 shutdown_connections 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.269+0000 7f020e9a1640 1 -- 172.21.15.1:0/2145074252 wait complete. 2024-08-19T09:50:25.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.269+0000 7f020e9a1640 1 librados: shutdown 2024-08-19T09:50:25.308 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:25.308 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:50:25.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.405+0000 7f9c4ee4f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.405+0000 7f9c4ee4f640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.405+0000 7f9c4ee4f640 10 monclient: monmap: 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:25.407027+0000 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:25.407027+0000 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.408 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.405+0000 7f9c4ee4f640 1 Processor -- start 2024-08-19T09:50:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.405+0000 7f9c4ee4f640 1 -- start start 2024-08-19T09:50:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.405+0000 7f9c4ee4f640 10 monclient: init 2024-08-19T09:50:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient: _add_conns ranks=[2,5,4,3,0,6,7,8,1] 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c480a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient(hunting): picked mon.noname-h con 0x7f9c480694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c480a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c480a3c70 con 0x7f9c480a6830 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c480a33d0 con 0x7f9c480a46a0 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c480a36b0 con 0x7f9c480694b0 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4ee4f640 10 monclient(hunting): authenticate will time out at 681.483413s 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4cbeb640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f9c480a6830 0x7f9c480a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52856/0 (socket says 172.21.15.1:52856) 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4cbeb640 1 -- 172.21.15.1:0/1257864557 learned_addr learned my addr 172.21.15.1:0/1257864557 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.406+0000 7f9c4d3ec640 1 --1- 172.21.15.1:0/1257864557 >> v1:172.21.15.1:6791/0 conn(0x7f9c480a46a0 0x7f9c48141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57586/0 (socket says 172.21.15.1:57586) 2024-08-19T09:50:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c4cbeb640 1 -- 172.21.15.1:0/1257864557 >> v1:172.21.15.1:6790/0 conn(0x7f9c480a6830 legacy=0x7f9c480a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c4d3ec640 1 -- 172.21.15.1:0/1257864557 >> v1:172.21.15.1:6791/0 conn(0x7f9c480a46a0 legacy=0x7f9c48141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c47fff640 1 -- 172.21.15.1:0/1257864557 >> v1:172.21.15.82:6791/0 conn(0x7f9c480694b0 legacy=0x7f9c480a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325762383 0 0) 0x7f9c480a33d0 con 0x7f9c480a46a0 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): my global_id is 4459 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c30003520 con 0x7f9c480a46a0 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2636989752 0 0) 0x7f9c480a3c70 con 0x7f9c480a6830 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 10 monclient(hunting): my global_id is 4421 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.408+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c480a33d0 con 0x7f9c480a6830 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393028657 0 0) 0x7f9c480a36b0 con 0x7f9c480694b0 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient(hunting): my global_id is 4514 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c480a3c70 con 0x7f9c480694b0 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3194045039 0 0) 0x7f9c30003520 con 0x7f9c480a46a0 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 >> v1:172.21.15.82:6791/0 conn(0x7f9c480694b0 legacy=0x7f9c480a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 >> v1:172.21.15.1:6790/0 conn(0x7f9c480a6830 legacy=0x7f9c480a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 monclient: found mon.noname-c 2024-08-19T09:50:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4806dbb0 con 0x7f9c480a46a0 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c38001a10 con 0x7f9c480a46a0 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 5 monclient: authenticate success, global_id 4459 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: dump: 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c38001ce0 con 0x7f9c480a46a0 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 1 -- 172.21.15.1:0/1257864557 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c38004b20 con 0x7f9c480a46a0 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 4 set_mon_vals no callback set 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c477fe640 10 monclient: dump: 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.419 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.420 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 10 monclient: shutdown 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/1257864557 >> v1:172.21.15.1:6791/0 conn(0x7f9c480a46a0 legacy=0x7f9c48141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/1257864557 shutdown_connections 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/1257864557 >> 172.21.15.1:0/1257864557 conn(0x7f9c480a9020 msgr2=0x7f9c48068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/1257864557 shutdown_connections 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.409+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/1257864557 wait complete. 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 10 monclient: monmap: 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:25.421 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 1 librados: starting msgr at 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 1 librados: starting objecter 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 1 Processor -- start 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 1 -- start start 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 1 librados: setting wanted keys 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 1 librados: calling monclient init 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.410+0000 7f9c4ee4f640 10 monclient: init 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient: _add_conns ranks=[0,4,6,7,8,3,5,2,1] 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c480694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c480a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient(hunting): picked mon.noname-d con 0x7f9c480a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c48136e50 con 0x7f9c480694b0 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c48138780 con 0x7f9c480a46a0 2024-08-19T09:50:25.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c48136720 con 0x7f9c480a6830 2024-08-19T09:50:25.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4ee4f640 10 monclient(hunting): authenticate will time out at 681.488430s 2024-08-19T09:50:25.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4cbeb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c480694b0 0x7f9c48142290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37300/0 (socket says 172.21.15.1:37300) 2024-08-19T09:50:25.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c4cbeb640 1 -- 172.21.15.1:0/673546483 learned_addr learned my addr 172.21.15.1:0/673546483 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.411+0000 7f9c47fff640 1 --1- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c480a46a0 0x7f9c481d2460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57602/0 (socket says 172.21.15.1:57602) 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.413+0000 7f9c47fff640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c480a46a0 legacy=0x7f9c481d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.413+0000 7f9c4d3ec640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6792/0 conn(0x7f9c480a6830 legacy=0x7f9c481d5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.413+0000 7f9c457fa640 10 client.?.objecter ms_handle_connect 0x7f9c480a46a0 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.413+0000 7f9c457fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.413+0000 7f9c4cbeb640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6789/0 conn(0x7f9c480694b0 legacy=0x7f9c48142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.413+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 client.?.objecter ms_handle_connect 0x7f9c480a6830 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 client.?.objecter ms_handle_connect 0x7f9c480694b0 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067781601 0 0) 0x7f9c48138780 con 0x7f9c480a46a0 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): my global_id is 4468 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c34004ee0 con 0x7f9c480a46a0 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257167122 0 0) 0x7f9c48136720 con 0x7f9c480a6830 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): my global_id is 4380 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c48138780 con 0x7f9c480a6830 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462170653 0 0) 0x7f9c48136e50 con 0x7f9c480694b0 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 10 monclient(hunting): my global_id is 4455 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.415+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c48136720 con 0x7f9c480694b0 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1335960026 0 0) 0x7f9c34004ee0 con 0x7f9c480a46a0 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c48136e50 con 0x7f9c480a46a0 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c3c001cf0 con 0x7f9c480a46a0 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): dump: 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:25.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4033242018 0 0) 0x7f9c48138780 con 0x7f9c480a6830 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c34004ee0 con 0x7f9c480a6830 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 164237583 0 0) 0x7f9c48136720 con 0x7f9c480694b0 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c48138780 con 0x7f9c480694b0 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c38004b20 con 0x7f9c480a6830 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): dump: 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c40004b60 con 0x7f9c480694b0 2024-08-19T09:50:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient(hunting): dump: 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 796223821 0 0) 0x7f9c48136e50 con 0x7f9c480a46a0 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6792/0 conn(0x7f9c480a6830 legacy=0x7f9c481d5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6789/0 conn(0x7f9c480694b0 legacy=0x7f9c48142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 monclient: found mon.c 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c480aa080 con 0x7f9c480a46a0 2024-08-19T09:50:25.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 5 monclient: authenticate success, global_id 4468 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 10 monclient: _renew_subs 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c48142d00 con 0x7f9c480a46a0 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 10 monclient: _renew_subs 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c481d72b0 con 0x7f9c480a46a0 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c4ee4f640 1 librados: init done 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c3c003ab0 con 0x7f9c480a46a0 2024-08-19T09:50:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c3c003fa0 con 0x7f9c480a46a0 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c46ffd640 4 set_mon_vals no callback set 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.416+0000 7f9c457fa640 10 monclient: dump: 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.438 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c4ee4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9c280065c0 con 0x7f9c28002860 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9c3c004240 con 0x7f9c480a46a0 2024-08-19T09:50:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f9c3c04da90 con 0x7f9c480a46a0 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c457fa640 10 client.4468.objecter ms_dispatch 0x7f9c48000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c457fa640 3 client.4468.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.417+0000 7f9c457fa640 3 client.4468.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.418+0000 7f9c457fa640 20 client.4468.objecter dump_active .. 0 homeless 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.418+0000 7f9c4d3ec640 10 monclient: get_auth_request con 0x7f9c34041ab0 auth_method 0 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c4cbeb640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c28002860 legacy=0x7f9c28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 10 client.4468.objecter ms_handle_connect 0x7f9c28002860 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 10 client.4468.objecter resend_mon_ops 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888833354 0 0) 0x7f9c280065c0 con 0x7f9c28002860 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 10 monclient(hunting): my global_id is 4477 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.419+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c48136e50 con 0x7f9c28002860 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2755492198 0 0) 0x7f9c48136e50 con 0x7f9c28002860 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c280065c0 con 0x7f9c28002860 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c40005230 con 0x7f9c28002860 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 10 monclient: dump: 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4185915297 0 0) 0x7f9c280065c0 con 0x7f9c28002860 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c2800a160 con 0x7f9c28002860 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9c40007470 con 0x7f9c28002860 2024-08-19T09:50:25.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:25.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c28002860 legacy=0x7f9c28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c28002860 legacy=0x7f9c28004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.420+0000 7f9c457fa640 10 client.4468.objecter ms_handle_connect 0x7f9c34041ab0 2024-08-19T09:50:25.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.426+0000 7f9c4ee4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:25.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.426+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9c28009e70 con 0x7f9c28009130 2024-08-19T09:50:25.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c4cbeb640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c28009130 legacy=0x7f9c280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 client.4468.objecter ms_handle_connect 0x7f9c28009130 2024-08-19T09:50:25.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 client.4468.objecter resend_mon_ops 2024-08-19T09:50:25.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306614765 0 0) 0x7f9c28009e70 con 0x7f9c28009130 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient(hunting): my global_id is 4486 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c280065c0 con 0x7f9c28009130 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2333164365 0 0) 0x7f9c280065c0 con 0x7f9c28009130 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c28009e70 con 0x7f9c28009130 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9c40006c60 con 0x7f9c28009130 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient: dump: 2024-08-19T09:50:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2398209092 0 0) 0x7f9c28009e70 con 0x7f9c28009130 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c2800c3f0 con 0x7f9c28009130 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7683 (unknown 0 0 1143794240) 0x7f9c40006f40 con 0x7f9c28009130 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c28009130 legacy=0x7f9c280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c457fa640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c28009130 legacy=0x7f9c280095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:25.447 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 31701, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 30, 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:25.468 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c4ee4f640 10 librados: watch_flush enter 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.429+0000 7f9c4ee4f640 10 librados: watch_flush exit 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 20 client.4468.objecter shutdown clearing up homeless session... 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 10 client.4468.objecter successfully canceled tick 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9c34041ab0 legacy=0x7f9c34043f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 10 monclient: shutdown 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 >> v1:172.21.15.1:6791/0 conn(0x7f9c480a46a0 legacy=0x7f9c481d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4d3ec640 1 -- 172.21.15.1:0/673546483 reap_dead start 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4d3ec640 1 -- 172.21.15.1:0/673546483 reap_dead start 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 shutdown_connections 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 >> 172.21.15.1:0/673546483 conn(0x7f9c480a9020 msgr2=0x7f9c480a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 shutdown_connections 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 -- 172.21.15.1:0/673546483 wait complete. 2024-08-19T09:50:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.430+0000 7f9c4ee4f640 1 librados: shutdown 2024-08-19T09:50:25.471 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:25.471 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:50:25.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:50:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.569+0000 7f9faba36640 10 monclient: get_monmap_and_config 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.569+0000 7f9faba36640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.569+0000 7f9faba36640 10 monclient: monmap: 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:25.571249+0000 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:25.571249+0000 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.569+0000 7f9faba36640 1 Processor -- start 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 1 -- start start 2024-08-19T09:50:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient: init 2024-08-19T09:50:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient: _add_conns ranks=[4,2,3,6,1,8,7,5,0] 2024-08-19T09:50:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa40a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient(hunting): picked mon.noname-g con 0x7f9fa413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa40a3b40 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa40a8130 con 0x7f9fa413f5c0 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa406ede0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9faba36640 10 monclient(hunting): authenticate will time out at 681.647665s 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9fa8fd1640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f9fa40a6ba0 0x7f9fa40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52868/0 (socket says 172.21.15.1:52868) 2024-08-19T09:50:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.570+0000 7f9fa8fd1640 1 -- 172.21.15.1:0/3771967552 learned_addr learned my addr 172.21.15.1:0/3771967552 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa8fd1640 1 -- 172.21.15.1:0/3771967552 >> v1:172.21.15.1:6790/0 conn(0x7f9fa40a6ba0 legacy=0x7f9fa40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa97d2640 1 -- 172.21.15.1:0/3771967552 >> v1:172.21.15.1:6791/0 conn(0x7f9fa40a67c0 legacy=0x7f9fa40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa9fd3640 1 -- 172.21.15.1:0/3771967552 >> v1:172.21.15.82:6790/0 conn(0x7f9fa413f5c0 legacy=0x7f9fa41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468597435 0 0) 0x7f9fa40a3b40 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): my global_id is 4430 2024-08-19T09:50:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f8c003520 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968895411 0 0) 0x7f9fa406ede0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): my global_id is 4495 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa40a3b40 con 0x7f9fa40a67c0 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063632954 0 0) 0x7f9fa40a8130 con 0x7f9fa413f5c0 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient(hunting): my global_id is 4440 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa406ede0 con 0x7f9fa413f5c0 2024-08-19T09:50:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3521543505 0 0) 0x7f9f8c003520 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 >> v1:172.21.15.1:6791/0 conn(0x7f9fa40a67c0 legacy=0x7f9fa40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 >> v1:172.21.15.82:6790/0 conn(0x7f9fa413f5c0 legacy=0x7f9fa41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 monclient: found mon.noname-b 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa40a92a0 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f98001ae0 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9faba36640 5 monclient: authenticate success, global_id 4430 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9faba36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: dump: 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.585 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9faba36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f980048a0 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9faba36640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 1 -- 172.21.15.1:0/3771967552 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f98004d90 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9faba36640 4 set_mon_vals no callback set 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.573+0000 7f9fa3fff640 10 monclient: dump: 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.587 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 10 monclient: shutdown 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 -- 172.21.15.1:0/3771967552 >> v1:172.21.15.1:6790/0 conn(0x7f9fa40a6ba0 legacy=0x7f9fa40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 -- 172.21.15.1:0/3771967552 shutdown_connections 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 -- 172.21.15.1:0/3771967552 >> 172.21.15.1:0/3771967552 conn(0x7f9fa40ac4b0 msgr2=0x7f9fa40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 -- 172.21.15.1:0/3771967552 shutdown_connections 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 -- 172.21.15.1:0/3771967552 wait complete. 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 10 monclient: monmap: 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 librados: starting msgr at 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.574+0000 7f9faba36640 1 librados: starting objecter 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 Processor -- start 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 -- start start 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 librados: setting wanted keys 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 librados: calling monclient init 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient: init 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient: _add_conns ranks=[0,3,6,2,4,7,5,8,1] 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient(hunting): picked mon.noname-g con 0x7f9fa40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient(hunting): picked mon.noname-d con 0x7f9fa413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa413a6b0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa413a990 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa4139c70 con 0x7f9fa413f5c0 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9faba36640 10 monclient(hunting): authenticate will time out at 681.652589s 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9fa97d2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9fa40a67c0 0x7f9fa409fe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37304/0 (socket says 172.21.15.1:37304) 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.575+0000 7f9fa97d2640 1 -- 172.21.15.1:0/2419166539 learned_addr learned my addr 172.21.15.1:0/2419166539 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa97d2640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6789/0 conn(0x7f9fa40a67c0 legacy=0x7f9fa409fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa9fd3640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa413f5c0 legacy=0x7f9fa41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9fa40a67c0 2024-08-19T09:50:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9fa413f5c0 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2558309335 0 0) 0x7f9fa4139c70 con 0x7f9fa413f5c0 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa1ffb640 10 monclient(hunting): my global_id is 4389 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.577+0000 7f9fa8fd1640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.82:6790/0 conn(0x7f9fa40a6ba0 legacy=0x7f9fa40a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f90004eb0 con 0x7f9fa413f5c0 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9fa40a6ba0 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164410336 0 0) 0x7f9fa413a6b0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): my global_id is 4464 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa4139c70 con 0x7f9fa40a67c0 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3323677509 0 0) 0x7f9fa4139c70 con 0x7f9fa40a67c0 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fa413a6b0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f94003990 con 0x7f9fa40a67c0 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.593 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375756109 0 0) 0x7f9fa413a990 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): my global_id is 4449 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa4139c70 con 0x7f9fa40a6ba0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1335863209 0 0) 0x7f9f90004eb0 con 0x7f9fa413f5c0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fa413a990 con 0x7f9fa413f5c0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f9c001cf0 con 0x7f9fa413f5c0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1826948089 0 0) 0x7f9fa413a6b0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa413f5c0 legacy=0x7f9fa41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.82:6790/0 conn(0x7f9fa40a6ba0 legacy=0x7f9fa40a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 monclient: found mon.a 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa40a0c70 con 0x7f9fa40a67c0 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 5 monclient: authenticate success, global_id 4464 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 10 monclient: _renew_subs 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa40a0f00 con 0x7f9fa40a67c0 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f94003e00 con 0x7f9fa40a67c0 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f94004330 con 0x7f9fa40a67c0 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa1ffb640 10 monclient: dump: 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9fa37fe640 4 set_mon_vals no callback set 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 10 monclient: _renew_subs 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9fa41d7b50 con 0x7f9fa40a67c0 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.578+0000 7f9faba36640 1 librados: init done 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9f940045d0 con 0x7f9fa40a67c0 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9faba36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9fa4142c40 con 0x7f9fa413fce0 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f9f9404dd40 con 0x7f9fa40a67c0 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa1ffb640 10 client.4464.objecter ms_dispatch 0x7f9fa4000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa1ffb640 3 client.4464.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa1ffb640 3 client.4464.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa1ffb640 20 client.4464.objecter dump_active .. 0 homeless 2024-08-19T09:50:25.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.579+0000 7f9fa9fd3640 10 monclient: get_auth_request con 0x7f9f900416c0 auth_method 0 2024-08-19T09:50:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa8fd1640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa413fce0 legacy=0x7f9fa41400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 10 client.4464.objecter ms_handle_connect 0x7f9fa413fce0 2024-08-19T09:50:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 10 client.4464.objecter resend_mon_ops 2024-08-19T09:50:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3202272846 0 0) 0x7f9fa4142c40 con 0x7f9fa413fce0 2024-08-19T09:50:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 10 monclient(hunting): my global_id is 4398 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.581+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa413a6b0 con 0x7f9fa413fce0 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2110771551 0 0) 0x7f9fa413a6b0 con 0x7f9fa413fce0 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fa4142c40 con 0x7f9fa413fce0 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f98001d30 con 0x7f9fa413fce0 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 10 monclient: dump: 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.604 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.605 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 10 client.4464.objecter ms_handle_connect 0x7f9f900416c0 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1663093714 0 0) 0x7f9fa4142c40 con 0x7f9fa413fce0 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fa40a9de0 con 0x7f9fa413fce0 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9f98004ac0 con 0x7f9fa413fce0 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa413fce0 legacy=0x7f9fa41400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.582+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa413fce0 legacy=0x7f9fa41400c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.588+0000 7f9faba36640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.588+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9fa4068f60 con 0x7f9fa41407d0 2024-08-19T09:50:25.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa8fd1640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa41407d0 legacy=0x7f9fa41dbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 client.4464.objecter ms_handle_connect 0x7f9fa41407d0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 client.4464.objecter resend_mon_ops 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473961193 0 0) 0x7f9fa4068f60 con 0x7f9fa41407d0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient(hunting): my global_id is 4407 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa4142c40 con 0x7f9fa41407d0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3622102167 0 0) 0x7f9fa4142c40 con 0x7f9fa41407d0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fa4068f60 con 0x7f9fa41407d0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f98005390 con 0x7f9fa41407d0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:25.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient: dump: 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3818648707 0 0) 0x7f9fa4068f60 con 0x7f9fa41407d0 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9fa4140bb0 con 0x7f9fa41407d0 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7683 (unknown 0 0 2271954761) 0x7f9f98008500 con 0x7f9fa41407d0 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa41407d0 legacy=0x7f9fa41dbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.591+0000 7f9fa1ffb640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6792/0 conn(0x7f9fa41407d0 legacy=0x7f9fa41dbd40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 31848, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 30, 2024-08-19T09:50:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:25.633 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 10 librados: watch_flush enter 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 10 librados: watch_flush exit 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 20 client.4464.objecter shutdown clearing up homeless session... 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 10 client.4464.objecter successfully canceled tick 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9f900416c0 legacy=0x7f9f90043b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 10 monclient: shutdown 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 >> v1:172.21.15.1:6789/0 conn(0x7f9fa40a67c0 legacy=0x7f9fa409fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9fa9fd3640 1 -- 172.21.15.1:0/2419166539 reap_dead start 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9fa9fd3640 1 -- 172.21.15.1:0/2419166539 reap_dead start 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 shutdown_connections 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 >> 172.21.15.1:0/2419166539 conn(0x7f9fa40ac4b0 msgr2=0x7f9fa40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 shutdown_connections 2024-08-19T09:50:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 -- 172.21.15.1:0/2419166539 wait complete. 2024-08-19T09:50:25.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.592+0000 7f9faba36640 1 librados: shutdown 2024-08-19T09:50:25.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:25.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:50:25.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.737+0000 7f4df8827640 10 monclient: get_monmap_and_config 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.737+0000 7f4df8827640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.737+0000 7f4df8827640 10 monclient: monmap: 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:25.739035+0000 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:25.739035+0000 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.740 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.737+0000 7f4df8827640 1 Processor -- start 2024-08-19T09:50:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.737+0000 7f4df8827640 1 -- start start 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.737+0000 7f4df8827640 10 monclient: init 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient: _add_conns ranks=[3,1,7,0,8,2,4,5,6] 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient(hunting): picked mon.noname-g con 0x7f4df40a5610 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient(hunting): picked mon.noname-f con 0x7f4df4073940 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient(hunting): picked mon.noname-i con 0x7f4df40a6100 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df4072f60 con 0x7f4df4073940 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df4074340 con 0x7f4df40a5610 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df4070d20 con 0x7f4df40a6100 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df8827640 10 monclient(hunting): authenticate will time out at 681.815401s 2024-08-19T09:50:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df259c640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 0x7f4df40a59f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48594/0 (socket says 172.21.15.1:48594) 2024-08-19T09:50:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df2d9d640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f4df40a6100 0x7f4df413ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:58598/0 (socket says 172.21.15.1:58598) 2024-08-19T09:50:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.738+0000 7f4df259c640 1 -- 172.21.15.1:0/311645301 learned_addr learned my addr 172.21.15.1:0/311645301 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.740+0000 7f4df259c640 1 -- 172.21.15.1:0/311645301 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 legacy=0x7f4df40a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.740+0000 7f4df2d9d640 1 -- 172.21.15.1:0/311645301 >> v1:172.21.15.82:6792/0 conn(0x7f4df40a6100 legacy=0x7f4df413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.740+0000 7f4df1d9b640 1 -- 172.21.15.1:0/311645301 >> v1:172.21.15.82:6789/0 conn(0x7f4df4073940 legacy=0x7f4df413c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632706073 0 0) 0x7f4df4072f60 con 0x7f4df4073940 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): my global_id is 4366 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4de0003520 con 0x7f4df4073940 2024-08-19T09:50:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 253492821 0 0) 0x7f4df4070d20 con 0x7f4df40a6100 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): my global_id is 4408 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df4072f60 con 0x7f4df40a6100 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024058823 0 0) 0x7f4df4074340 con 0x7f4df40a5610 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 10 monclient(hunting): my global_id is 4458 2024-08-19T09:50:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.756+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df4070d20 con 0x7f4df40a5610 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2875926969 0 0) 0x7f4df4070d20 con 0x7f4df40a5610 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 >> v1:172.21.15.82:6792/0 conn(0x7f4df40a6100 legacy=0x7f4df413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 >> v1:172.21.15.82:6789/0 conn(0x7f4df4073940 legacy=0x7f4df413c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 monclient: found mon.noname-g 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df4071620 con 0x7f4df40a5610 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4dec001a00 con 0x7f4df40a5610 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df8827640 5 monclient: authenticate success, global_id 4458 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df8827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: dump: 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df8827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4dec001c50 con 0x7f4df40a5610 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df8827640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df8827640 4 set_mon_vals no callback set 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 1 -- 172.21.15.1:0/311645301 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4dec004be0 con 0x7f4df40a5610 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.771+0000 7f4df159a640 10 monclient: dump: 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 10 monclient: shutdown 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 1 -- 172.21.15.1:0/311645301 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 legacy=0x7f4df40a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 1 -- 172.21.15.1:0/311645301 shutdown_connections 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 1 -- 172.21.15.1:0/311645301 >> 172.21.15.1:0/311645301 conn(0x7f4df40aa2f0 msgr2=0x7f4df40a6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 1 -- 172.21.15.1:0/311645301 shutdown_connections 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.772+0000 7f4df8827640 1 -- 172.21.15.1:0/311645301 wait complete. 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 10 monclient: monmap: 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.780 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 1 librados: starting msgr at 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 1 librados: starting objecter 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 1 Processor -- start 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 1 -- start start 2024-08-19T09:50:25.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 1 librados: setting wanted keys 2024-08-19T09:50:25.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 1 librados: calling monclient init 2024-08-19T09:50:25.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 10 monclient: init 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 10 monclient: _add_conns ranks=[1,3,4,6,2,5,7,8,0] 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.773+0000 7f4df8827640 10 monclient(hunting): picked mon.noname-f con 0x7f4df4073940 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 10 monclient(hunting): picked mon.noname-g con 0x7f4df40a5610 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 10 monclient(hunting): picked mon.noname-c con 0x7f4df40a6100 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df40a9570 con 0x7f4df4073940 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df40a3080 con 0x7f4df40a5610 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df4125410 con 0x7f4df40a6100 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df8827640 10 monclient(hunting): authenticate will time out at 681.851095s 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df2d9d640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 0x7f4df41d4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57634/0 (socket says 172.21.15.1:57634) 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.774+0000 7f4df2d9d640 1 -- 172.21.15.1:0/4015043918 learned_addr learned my addr 172.21.15.1:0/4015043918 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df2d9d640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 legacy=0x7f4df41d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 client.?.objecter ms_handle_connect 0x7f4df40a6100 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df2d9d640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 legacy=0x7f4df41d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df2d9d640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 legacy=0x7f4df41d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df2d9d640 1 --1- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 0x7f4df41d4420 :-1 s=OPENED pgs=87 cs=1 l=1).handle_message read tag failed 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df2d9d640 1 --1- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 0x7f4df41d4420 :-1 s=OPENED pgs=87 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6791/0 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 client.?.objecter ms_handle_connect 0x7f4df40a5610 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df1d9b640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 legacy=0x7f4df41d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 client.?.objecter ms_handle_connect 0x7f4df4073940 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df259c640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6789/0 conn(0x7f4df4073940 legacy=0x7f4df413fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df1d9b640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 legacy=0x7f4df41d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df1d9b640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 legacy=0x7f4df41d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df1d9b640 1 --1- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 0x7f4df41d0cf0 :-1 s=OPENED pgs=83 cs=1 l=1).handle_message read tag failed 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4df1d9b640 1 --1- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 0x7f4df41d0cf0 :-1 s=OPENED pgs=83 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6790/0 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316931860 0 0) 0x7f4df40a9570 con 0x7f4df4073940 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): my global_id is 4375 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd4004eb0 con 0x7f4df4073940 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 663909060 0 0) 0x7f4dd4004eb0 con 0x7f4df4073940 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4df40a9570 con 0x7f4df4073940 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4dec004b20 con 0x7f4df4073940 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): dump: 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.776+0000 7f4deaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2023441011 0 0) 0x7f4df40a9570 con 0x7f4df4073940 2024-08-19T09:50:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6791/0 conn(0x7f4df40a6100 legacy=0x7f4df41d4420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6790/0 conn(0x7f4df40a5610 legacy=0x7f4df41d0cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 monclient: found mon.f 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df4127770 con 0x7f4df4073940 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 5 monclient: authenticate success, global_id 4375 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 10 monclient: _renew_subs 2024-08-19T09:50:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4df4140420 con 0x7f4df4073940 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 10 monclient: _renew_subs 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4df4126150 con 0x7f4df4073940 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 1 librados: init done 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4dec0019a0 con 0x7f4df4073940 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4dec005510 con 0x7f4df4073940 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df0d99640 4 set_mon_vals no callback set 2024-08-19T09:50:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 10 monclient: dump: 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4dd80065c0 con 0x7f4dd8002860 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.777+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4dec011970 con 0x7f4df4073940 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.778+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f4dec04d920 con 0x7f4df4073940 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.778+0000 7f4deaffd640 10 client.4375.objecter ms_dispatch 0x7f4df4000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.778+0000 7f4deaffd640 3 client.4375.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.778+0000 7f4deaffd640 3 client.4375.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.778+0000 7f4deaffd640 20 client.4375.objecter dump_active .. 0 homeless 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.778+0000 7f4df2d9d640 10 monclient: get_auth_request con 0x7f4dd4040e40 auth_method 0 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 client.4375.objecter ms_handle_connect 0x7f4dd8002860 2024-08-19T09:50:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 client.4375.objecter resend_mon_ops 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4df1d9b640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6793/0 conn(0x7f4dd8002860 legacy=0x7f4dd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395335466 0 0) 0x7f4dd80065c0 con 0x7f4dd8002860 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient(hunting): my global_id is 4445 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df40a9570 con 0x7f4dd8002860 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2866358059 0 0) 0x7f4df40a9570 con 0x7f4dd8002860 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dd80065c0 con 0x7f4dd8002860 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ddc003610 con 0x7f4dd8002860 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient: dump: 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 client.4375.objecter ms_handle_connect 0x7f4dd4040e40 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 202014914 0 0) 0x7f4dd80065c0 con 0x7f4dd8002860 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4dd800a160 con 0x7f4dd8002860 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4ddc0057c0 con 0x7f4dd8002860 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6793/0 conn(0x7f4dd8002860 legacy=0x7f4dd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.780+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6793/0 conn(0x7f4dd8002860 legacy=0x7f4dd8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.787+0000 7f4df8827640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.787+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4dd8009e50 con 0x7f4dd8009110 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 client.4375.objecter ms_handle_connect 0x7f4dd8009110 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4df1d9b640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6793/0 conn(0x7f4dd8009110 legacy=0x7f4dd80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 client.4375.objecter resend_mon_ops 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156222215 0 0) 0x7f4dd8009e50 con 0x7f4dd8009110 2024-08-19T09:50:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient(hunting): my global_id is 4454 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd80065c0 con 0x7f4dd8009110 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1405259171 0 0) 0x7f4dd80065c0 con 0x7f4dd8009110 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dd8009e50 con 0x7f4dd8009110 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ddc003e70 con 0x7f4dd8009110 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.789+0000 7f4deaffd640 10 monclient: dump: 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.802 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 418275672 0 0) 0x7f4dd8009e50 con 0x7f4dd8009110 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4dd800c3a0 con 0x7f4dd8009110 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3994181649) 0x7f4ddc004110 con 0x7f4dd8009110 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.803 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 32063, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:25.804 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 31, 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.805 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.806 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:25.807 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.808 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:25.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.812 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.813 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:25.816 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.817 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.819 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.820 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.821 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.822 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:25.823 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6793/0 conn(0x7f4dd8009110 legacy=0x7f4dd80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4deaffd640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.1:6793/0 conn(0x7f4dd8009110 legacy=0x7f4dd80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 10 librados: watch_flush enter 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 10 librados: watch_flush exit 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 20 client.4375.objecter shutdown clearing up homeless session... 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 10 client.4375.objecter successfully canceled tick 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4dd4040e40 legacy=0x7f4dd4043300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 10 monclient: shutdown 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 >> v1:172.21.15.82:6789/0 conn(0x7f4df4073940 legacy=0x7f4df413fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df2d9d640 1 -- 172.21.15.1:0/4015043918 reap_dead start 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df2d9d640 1 -- 172.21.15.1:0/4015043918 reap_dead start 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 shutdown_connections 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 >> 172.21.15.1:0/4015043918 conn(0x7f4df40aa2f0 msgr2=0x7f4df4140b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 shutdown_connections 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 -- 172.21.15.1:0/4015043918 wait complete. 2024-08-19T09:50:25.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.790+0000 7f4df8827640 1 librados: shutdown 2024-08-19T09:50:25.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:25.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:50:25.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.924+0000 7f3d93ac5640 10 monclient: get_monmap_and_config 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.924+0000 7f3d93ac5640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.924+0000 7f3d93ac5640 10 monclient: monmap: 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:25.926853+0000 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:25.926853+0000 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.925+0000 7f3d93ac5640 1 Processor -- start 2024-08-19T09:50:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.925+0000 7f3d93ac5640 1 -- start start 2024-08-19T09:50:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.925+0000 7f3d93ac5640 10 monclient: init 2024-08-19T09:50:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient: _add_conns ranks=[4,2,8,5,1,3,6,0,7] 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8c0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient(hunting): picked mon.noname-e con 0x7f3d8c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a3b40 con 0x7f3d8c0a6ba0 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a8130 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c06ede0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d93ac5640 10 monclient(hunting): authenticate will time out at 682.003201s 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d92062640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f3d8c13f5c0 0x7f3d8c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44842/0 (socket says 172.21.15.1:44842) 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d92062640 1 -- 172.21.15.1:0/2545215066 learned_addr learned my addr 172.21.15.1:0/2545215066 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.926+0000 7f3d91861640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f3d8c0a67c0 0x7f3d8c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57638/0 (socket says 172.21.15.1:57638) 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d92062640 1 -- 172.21.15.1:0/2545215066 >> v1:172.21.15.1:6793/0 conn(0x7f3d8c13f5c0 legacy=0x7f3d8c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d91060640 1 -- 172.21.15.1:0/2545215066 >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0a6ba0 legacy=0x7f3d8c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d91861640 1 -- 172.21.15.1:0/2545215066 >> v1:172.21.15.1:6791/0 conn(0x7f3d8c0a67c0 legacy=0x7f3d8c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069076610 0 0) 0x7f3d8c0a8130 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): my global_id is 4504 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d74003520 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676046984 0 0) 0x7f3d8c0a3b40 con 0x7f3d8c0a6ba0 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient(hunting): my global_id is 4439 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c0a8130 con 0x7f3d8c0a6ba0 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4245141882 0 0) 0x7f3d74003520 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 >> v1:172.21.15.1:6793/0 conn(0x7f3d8c13f5c0 legacy=0x7f3d8c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0a6ba0 legacy=0x7f3d8c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 monclient: found mon.noname-c 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c0a92a0 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d93ac5640 5 monclient: authenticate success, global_id 4504 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.928+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d7c001e60 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: dump: 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d7c004b60 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 1 -- 172.21.15.1:0/2545215066 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d7c0050d0 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d9085f640 10 monclient: dump: 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 4 set_mon_vals no callback set 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 10 monclient: shutdown 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 1 -- 172.21.15.1:0/2545215066 >> v1:172.21.15.1:6791/0 conn(0x7f3d8c0a67c0 legacy=0x7f3d8c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 1 -- 172.21.15.1:0/2545215066 shutdown_connections 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 1 -- 172.21.15.1:0/2545215066 >> 172.21.15.1:0/2545215066 conn(0x7f3d8c0ac4b0 msgr2=0x7f3d8c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 1 -- 172.21.15.1:0/2545215066 shutdown_connections 2024-08-19T09:50:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.929+0000 7f3d93ac5640 1 -- 172.21.15.1:0/2545215066 wait complete. 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 10 monclient: build_initial_monmap 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 10 monclient: monmap: 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 1 librados: starting msgr at 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 1 librados: starting objecter 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 1 Processor -- start 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 1 -- start start 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 1 librados: setting wanted keys 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 1 librados: calling monclient init 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.930+0000 7f3d93ac5640 10 monclient: init 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:25.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient: _add_conns ranks=[2,5,4,0,6,7,8,1,3] 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient(hunting): picked mon.noname-h con 0x7f3d8c0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a17f0 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c13a930 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a2810 con 0x7f3d8c0a6ba0 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d93ac5640 10 monclient(hunting): authenticate will time out at 682.008208s 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d91861640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0a67c0 0x7f3d8c09ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52880/0 (socket says 172.21.15.1:52880) 2024-08-19T09:50:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.931+0000 7f3d91861640 1 -- 172.21.15.1:0/1575752201 learned_addr learned my addr 172.21.15.1:0/1575752201 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d92062640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.1:6791/0 conn(0x7f3d8c13f5c0 legacy=0x7f3d8c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 client.?.objecter ms_handle_connect 0x7f3d8c13f5c0 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d91861640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0a67c0 legacy=0x7f3d8c09ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 client.?.objecter ms_handle_connect 0x7f3d8c0a67c0 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589687936 0 0) 0x7f3d8c13a930 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d91060640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6791/0 conn(0x7f3d8c0a6ba0 legacy=0x7f3d8c0a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): my global_id is 4513 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d78004eb0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 client.?.objecter ms_handle_connect 0x7f3d8c0a6ba0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769138544 0 0) 0x7f3d8c0a17f0 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): my global_id is 4448 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c13a930 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1690505026 0 0) 0x7f3d78004eb0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d8c0a17f0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d84001bf0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1608172729 0 0) 0x7f3d8c13a930 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d78004eb0 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d7c005050 con 0x7f3d8c0a67c0 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 761941130 0 0) 0x7f3d8c0a2810 con 0x7f3d8c0a6ba0 2024-08-19T09:50:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 10 monclient(hunting): my global_id is 4523 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.933+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c13a930 con 0x7f3d8c0a6ba0 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 137336957 0 0) 0x7f3d8c0a17f0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6791/0 conn(0x7f3d8c0a6ba0 legacy=0x7f3d8c0a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0a67c0 legacy=0x7f3d8c09ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 monclient: found mon.c 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c0a2350 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 5 monclient: authenticate success, global_id 4513 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 10 monclient: _renew_subs 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8c139a90 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 10 monclient: _renew_subs 2024-08-19T09:50:25.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d8c0a0fa0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 1 librados: init done 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d84004a90 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d84004f80 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8bfff640 4 set_mon_vals no callback set 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d8a7fc640 10 monclient: dump: 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.957 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.934+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d6c0065c0 con 0x7f3d6c002860 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3d84011460 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f3d8404d8b0 con 0x7f3d8c13f5c0 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d8a7fc640 10 client.4513.objecter ms_dispatch 0x7f3d8c000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d8a7fc640 3 client.4513.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d8a7fc640 3 client.4513.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d8a7fc640 20 client.4513.objecter dump_active .. 0 homeless 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.935+0000 7f3d91060640 10 monclient: get_auth_request con 0x7f3d78041680 auth_method 0 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d91861640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6789/0 conn(0x7f3d6c002860 legacy=0x7f3d6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 client.4513.objecter ms_handle_connect 0x7f3d6c002860 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 client.4513.objecter resend_mon_ops 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120272918 0 0) 0x7f3d6c0065c0 con 0x7f3d6c002860 2024-08-19T09:50:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient(hunting): my global_id is 4384 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c0a17f0 con 0x7f3d6c002860 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2630373713 0 0) 0x7f3d8c0a17f0 con 0x7f3d6c002860 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d6c0065c0 con 0x7f3d6c002860 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d7c004b60 con 0x7f3d6c002860 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 monclient: dump: 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3960177729 0 0) 0x7f3d6c0065c0 con 0x7f3d6c002860 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d6c00a160 con 0x7f3d6c002860 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.937+0000 7f3d8a7fc640 10 client.4513.objecter ms_handle_connect 0x7f3d78041680 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.938+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3d7c005b30 con 0x7f3d6c002860 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.938+0000 7f3d8a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.938+0000 7f3d8a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.938+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6789/0 conn(0x7f3d6c002860 legacy=0x7f3d6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.938+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6789/0 conn(0x7f3d6c002860 legacy=0x7f3d6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.944+0000 7f3d93ac5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.944+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d6c009e70 con 0x7f3d6c009130 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.946+0000 7f3d91861640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6789/0 conn(0x7f3d6c009130 legacy=0x7f3d6c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.946+0000 7f3d8a7fc640 10 client.4513.objecter ms_handle_connect 0x7f3d6c009130 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.946+0000 7f3d8a7fc640 10 client.4513.objecter resend_mon_ops 2024-08-19T09:50:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1716778101 0 0) 0x7f3d6c009e70 con 0x7f3d6c009130 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient(hunting): my global_id is 4393 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d6c0065c0 con 0x7f3d6c009130 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1473360399 0 0) 0x7f3d6c0065c0 con 0x7f3d6c009130 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d6c009e70 con 0x7f3d6c009130 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d7c006340 con 0x7f3d6c009130 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient: dump: 2024-08-19T09:50:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2114811360 0 0) 0x7f3d6c009e70 con 0x7f3d6c009130 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d6c00c410 con 0x7f3d6c009130 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7850 (unknown 0 0 498949132) 0x7f3d7c0085c0 con 0x7f3d6c009130 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6789/0 conn(0x7f3d6c009130 legacy=0x7f3d6c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.947+0000 7f3d8a7fc640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6789/0 conn(0x7f3d6c009130 legacy=0x7f3d6c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:25.965 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 32208, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 31, 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.981 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.982 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:25.983 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.984 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.985 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:25.986 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:25.987 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:25.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:25.987 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:25.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:25.987 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:25.987 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 10 librados: watch_flush enter 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 10 librados: watch_flush exit 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 20 client.4513.objecter shutdown clearing up homeless session... 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 10 client.4513.objecter successfully canceled tick 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3d78041680 legacy=0x7f3d78043b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 10 monclient: shutdown 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 >> v1:172.21.15.1:6791/0 conn(0x7f3d8c13f5c0 legacy=0x7f3d8c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d92062640 1 -- 172.21.15.1:0/1575752201 reap_dead start 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d92062640 1 -- 172.21.15.1:0/1575752201 reap_dead start 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 shutdown_connections 2024-08-19T09:50:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 >> 172.21.15.1:0/1575752201 conn(0x7f3d8c0ac4b0 msgr2=0x7f3d8c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 shutdown_connections 2024-08-19T09:50:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 -- 172.21.15.1:0/1575752201 wait complete. 2024-08-19T09:50:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:25.948+0000 7f3d93ac5640 1 librados: shutdown 2024-08-19T09:50:25.990 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:25.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:50:25.990 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:50:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.088+0000 7fcbab3aa640 10 monclient: get_monmap_and_config 2024-08-19T09:50:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.088+0000 7fcbab3aa640 10 monclient: build_initial_monmap 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.088+0000 7fcbab3aa640 10 monclient: monmap: 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:26.090387+0000 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:26.090387+0000 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:26.095 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 1 Processor -- start 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 1 -- start start 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient: init 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient: _add_conns ranks=[3,8,4,1,0,5,2,6,7] 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient(hunting): picked mon.noname-g con 0x7fcba40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient(hunting): picked mon.noname-e con 0x7fcba40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient(hunting): picked mon.noname-c con 0x7fcba413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:26.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba40a3b40 con 0x7fcba40a67c0 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba40a8130 con 0x7fcba413f5c0 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba406ede0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcbab3aa640 10 monclient(hunting): authenticate will time out at 682.166769s 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcba8945640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcba40a6ba0 0x7fcba40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44846/0 (socket says 172.21.15.1:44846) 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcba8945640 1 -- 172.21.15.1:0/2617303080 learned_addr learned my addr 172.21.15.1:0/2617303080 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.089+0000 7fcba9947640 1 --1- 172.21.15.1:0/2617303080 >> v1:172.21.15.1:6791/0 conn(0x7fcba413f5c0 0x7fcba41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57658/0 (socket says 172.21.15.1:57658) 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcba9947640 1 -- 172.21.15.1:0/2617303080 >> v1:172.21.15.1:6791/0 conn(0x7fcba413f5c0 legacy=0x7fcba41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcba8945640 1 -- 172.21.15.1:0/2617303080 >> v1:172.21.15.1:6793/0 conn(0x7fcba40a6ba0 legacy=0x7fcba40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcba9146640 1 -- 172.21.15.1:0/2617303080 >> v1:172.21.15.82:6790/0 conn(0x7fcba40a67c0 legacy=0x7fcba40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955139626 0 0) 0x7fcba406ede0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): my global_id is 4463 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb90003520 con 0x7fcba40a6ba0 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482514989 0 0) 0x7fcba40a8130 con 0x7fcba413f5c0 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): my global_id is 4522 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba406ede0 con 0x7fcba413f5c0 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783765675 0 0) 0x7fcba40a3b40 con 0x7fcba40a67c0 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient(hunting): my global_id is 4467 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba40a8130 con 0x7fcba40a67c0 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2518480526 0 0) 0x7fcb90003520 con 0x7fcba40a6ba0 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 >> v1:172.21.15.1:6791/0 conn(0x7fcba413f5c0 legacy=0x7fcba41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 >> v1:172.21.15.82:6790/0 conn(0x7fcba40a67c0 legacy=0x7fcba40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 monclient: found mon.noname-e 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba40a92a0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: _finish_auth 0 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb94001ad0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcbab3aa640 5 monclient: authenticate success, global_id 4463 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcbab3aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: dump: 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.109 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcbab3aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb940048a0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 1 -- 172.21.15.1:0/2617303080 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb94004d90 con 0x7fcba40a6ba0 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcbab3aa640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcbab3aa640 4 set_mon_vals no callback set 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.092+0000 7fcb9bfff640 10 monclient: dump: 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 10 monclient: shutdown 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:26.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 -- 172.21.15.1:0/2617303080 >> v1:172.21.15.1:6793/0 conn(0x7fcba40a6ba0 legacy=0x7fcba40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 -- 172.21.15.1:0/2617303080 shutdown_connections 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 -- 172.21.15.1:0/2617303080 >> 172.21.15.1:0/2617303080 conn(0x7fcba40ac4b0 msgr2=0x7fcba40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 -- 172.21.15.1:0/2617303080 shutdown_connections 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 -- 172.21.15.1:0/2617303080 wait complete. 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 10 monclient: build_initial_monmap 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 10 monclient: monmap: 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:26.113 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 librados: starting msgr at 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.093+0000 7fcbab3aa640 1 librados: starting objecter 2024-08-19T09:50:26.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 Processor -- start 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 -- start start 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 librados: setting wanted keys 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 librados: calling monclient init 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient: init 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient: _add_conns ranks=[5,4,6,0,1,8,7,2,3] 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient(hunting): picked mon.noname-h con 0x7fcba40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient(hunting): picked mon.noname-c con 0x7fcba40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient(hunting): picked mon.noname-d con 0x7fcba413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba40a2870 con 0x7fcba40a6ba0 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba40a1610 con 0x7fcba40a67c0 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba4139fa0 con 0x7fcba413f5c0 2024-08-19T09:50:26.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcbab3aa640 10 monclient(hunting): authenticate will time out at 682.171633s 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcba8945640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fcba40a6ba0 0x7fcba40a04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57664/0 (socket says 172.21.15.1:57664) 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.094+0000 7fcba8945640 1 -- 172.21.15.1:0/3198461301 learned_addr learned my addr 172.21.15.1:0/3198461301 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcba8945640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.1:6791/0 conn(0x7fcba40a6ba0 legacy=0x7fcba40a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcb99ffb640 10 client.?.objecter ms_handle_connect 0x7fcba40a6ba0 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcb99ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcb99ffb640 10 client.?.objecter ms_handle_connect 0x7fcba413f5c0 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcb99ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcba9947640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.1:6792/0 conn(0x7fcba413f5c0 legacy=0x7fcba41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.096+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556342082 0 0) 0x7fcba40a2870 con 0x7fcba40a6ba0 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): my global_id is 4531 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb88004eb0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262298148 0 0) 0x7fcba4139fa0 con 0x7fcba413f5c0 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): my global_id is 4416 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba40a2870 con 0x7fcba413f5c0 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcba9146640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6791/0 conn(0x7fcba40a67c0 legacy=0x7fcba409fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 client.?.objecter ms_handle_connect 0x7fcba40a67c0 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4119051621 0 0) 0x7fcba40a2870 con 0x7fcba413f5c0 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcba4139fa0 con 0x7fcba413f5c0 2024-08-19T09:50:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb9c002090 con 0x7fcba413f5c0 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.118 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050696827 0 0) 0x7fcba40a1610 con 0x7fcba40a67c0 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): my global_id is 4532 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba40a2870 con 0x7fcba40a67c0 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1170972276 0 0) 0x7fcb88004eb0 con 0x7fcba40a6ba0 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcba40a1610 con 0x7fcba40a6ba0 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb94004d90 con 0x7fcba40a6ba0 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3951587445 0 0) 0x7fcba4139fa0 con 0x7fcba413f5c0 2024-08-19T09:50:26.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6791/0 conn(0x7fcba40a67c0 legacy=0x7fcba409fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.1:6791/0 conn(0x7fcba40a6ba0 legacy=0x7fcba40a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 monclient: found mon.d 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba413a280 con 0x7fcba413f5c0 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 5 monclient: authenticate success, global_id 4416 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 10 monclient: _renew_subs 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:26.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcba4139b50 con 0x7fcba413f5c0 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb9c004e10 con 0x7fcba413f5c0 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb9c005300 con 0x7fcba413f5c0 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb99ffb640 10 monclient: dump: 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcb9b7fe640 4 set_mon_vals no callback set 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 10 monclient: _renew_subs 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fcba40a0be0 con 0x7fcba413f5c0 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.097+0000 7fcbab3aa640 1 librados: init done 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcb9c0054a0 con 0x7fcba413f5c0 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcbab3aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcba40a2a90 con 0x7fcba413fa30 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fcb9c04db60 con 0x7fcba413f5c0 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcb99ffb640 10 client.4416.objecter ms_dispatch 0x7fcba4000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcb99ffb640 3 client.4416.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcb99ffb640 3 client.4416.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcb99ffb640 20 client.4416.objecter dump_active .. 0 homeless 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.098+0000 7fcba8945640 10 monclient: get_auth_request con 0x7fcb880416c0 auth_method 0 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.100+0000 7fcb99ffb640 10 client.4416.objecter ms_handle_connect 0x7fcba413fa30 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.100+0000 7fcb99ffb640 10 client.4416.objecter resend_mon_ops 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.100+0000 7fcba9146640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6790/0 conn(0x7fcba413fa30 legacy=0x7fcba413fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028736025 0 0) 0x7fcba40a2a90 con 0x7fcba413fa30 2024-08-19T09:50:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient(hunting): my global_id is 4476 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba4139fa0 con 0x7fcba413fa30 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 client.4416.objecter ms_handle_connect 0x7fcb880416c0 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3489724999 0 0) 0x7fcba4139fa0 con 0x7fcba413fa30 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcba40a2a90 con 0x7fcba413fa30 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcba0003610 con 0x7fcba413fa30 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient: dump: 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.127 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3064310240 0 0) 0x7fcba40a2a90 con 0x7fcba413fa30 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcba40008d0 con 0x7fcba413fa30 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcba00057c0 con 0x7fcba413fa30 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6790/0 conn(0x7fcba413fa30 legacy=0x7fcba413fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.101+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6790/0 conn(0x7fcba413fa30 legacy=0x7fcba413fe10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.107+0000 7fcbab3aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.107+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcba4140900 con 0x7fcba4140520 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcba9146640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6790/0 conn(0x7fcba4140520 legacy=0x7fcba41db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 client.4416.objecter ms_handle_connect 0x7fcba4140520 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 client.4416.objecter resend_mon_ops 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3429115655 0 0) 0x7fcba4140900 con 0x7fcba4140520 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 monclient(hunting): my global_id is 4485 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba40a2a90 con 0x7fcba4140520 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3839561319 0 0) 0x7fcba40a2a90 con 0x7fcba4140520 2024-08-19T09:50:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcba4140900 con 0x7fcba4140520 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcba0003c50 con 0x7fcba4140520 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.110+0000 7fcb99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 10 monclient: dump: 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2066543949 0 0) 0x7fcba4140900 con 0x7fcba4140520 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcba41df240 con 0x7fcba4140520 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7683 (unknown 0 0 1406060439) 0x7fcba0005a50 con 0x7fcba4140520 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6790/0 conn(0x7fcba4140520 legacy=0x7fcba41db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcb99ffb640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6790/0 conn(0x7fcba4140520 legacy=0x7fcba41db990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 32357, 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 31, 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.136 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:26.138 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.139 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:26.140 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.141 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:26.145 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.147 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:26.148 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.149 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:26.150 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.151 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.152 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.153 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.154 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:26.155 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 10 librados: watch_flush enter 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 10 librados: watch_flush exit 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 20 client.4416.objecter shutdown clearing up homeless session... 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 10 client.4416.objecter successfully canceled tick 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcb880416c0 legacy=0x7fcb88043b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 10 monclient: shutdown 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 >> v1:172.21.15.1:6792/0 conn(0x7fcba413f5c0 legacy=0x7fcba41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcba9947640 1 -- 172.21.15.1:0/3198461301 reap_dead start 2024-08-19T09:50:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcba9947640 1 -- 172.21.15.1:0/3198461301 reap_dead start 2024-08-19T09:50:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 shutdown_connections 2024-08-19T09:50:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.111+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 >> 172.21.15.1:0/3198461301 conn(0x7fcba40ac4b0 msgr2=0x7fcba40a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.112+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 shutdown_connections 2024-08-19T09:50:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.112+0000 7fcbab3aa640 1 -- 172.21.15.1:0/3198461301 wait complete. 2024-08-19T09:50:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.112+0000 7fcbab3aa640 1 librados: shutdown 2024-08-19T09:50:26.158 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:26.158 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:50:26.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:50:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.255+0000 7f865f070640 10 monclient: get_monmap_and_config 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.255+0000 7f865f070640 10 monclient: build_initial_monmap 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.255+0000 7f865f070640 10 monclient: monmap: 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:26.257711+0000 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:26.257711+0000 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:26.259 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.256+0000 7f865f070640 1 Processor -- start 2024-08-19T09:50:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.256+0000 7f865f070640 1 -- start start 2024-08-19T09:50:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.256+0000 7f865f070640 10 monclient: init 2024-08-19T09:50:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.256+0000 7f865f070640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.256+0000 7f865f070640 10 monclient: _add_conns ranks=[6,1,7,4,5,0,8,2,3] 2024-08-19T09:50:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.256+0000 7f865f070640 10 monclient(hunting): picked mon.noname-d con 0x7f86580a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 10 monclient(hunting): picked mon.noname-f con 0x7f86580a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 10 monclient(hunting): picked mon.noname-i con 0x7f865813f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a3b40 con 0x7f86580a6ba0 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a8130 con 0x7f86580a67c0 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865806ede0 con 0x7f865813f5c0 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865f070640 10 monclient(hunting): authenticate will time out at 682.334083s 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865ce0c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f86580a67c0 0x7f86580a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:53030/0 (socket says 172.21.15.1:53030) 2024-08-19T09:50:26.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.257+0000 7f865ce0c640 1 -- 172.21.15.1:0/1125025355 learned_addr learned my addr 172.21.15.1:0/1125025355 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:26.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865ce0c640 1 -- 172.21.15.1:0/1125025355 >> v1:172.21.15.1:6792/0 conn(0x7f86580a67c0 legacy=0x7f86580a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865d60d640 1 -- 172.21.15.1:0/1125025355 >> v1:172.21.15.82:6792/0 conn(0x7f865813f5c0 legacy=0x7f86581419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141016944 0 0) 0x7f86580a8130 con 0x7f86580a67c0 2024-08-19T09:50:26.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): my global_id is 4425 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864ffff640 1 -- 172.21.15.1:0/1125025355 >> v1:172.21.15.82:6789/0 conn(0x7f86580a6ba0 legacy=0x7f86580a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8640003520 con 0x7f86580a67c0 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1034847972 0 0) 0x7f865806ede0 con 0x7f865813f5c0 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): my global_id is 4417 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86580a8130 con 0x7f865813f5c0 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4013638038 0 0) 0x7f86580a3b40 con 0x7f86580a6ba0 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient(hunting): my global_id is 4402 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f865806ede0 con 0x7f86580a6ba0 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1771265617 0 0) 0x7f8640003520 con 0x7f86580a67c0 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 >> v1:172.21.15.82:6792/0 conn(0x7f865813f5c0 legacy=0x7f86581419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 >> v1:172.21.15.82:6789/0 conn(0x7f86580a6ba0 legacy=0x7f86580a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 monclient: found mon.noname-d 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86580a92a0 con 0x7f86580a67c0 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: _finish_auth 0 2024-08-19T09:50:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8654001ae0 con 0x7f86580a67c0 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865f070640 5 monclient: authenticate success, global_id 4425 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865f070640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: dump: 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865f070640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86540048a0 con 0x7f86580a67c0 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 1 -- 172.21.15.1:0/1125025355 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8654004e10 con 0x7f86580a67c0 2024-08-19T09:50:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865f070640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f865f070640 4 set_mon_vals no callback set 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.259+0000 7f864f7fe640 10 monclient: dump: 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.274 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 10 monclient: shutdown 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 1 -- 172.21.15.1:0/1125025355 >> v1:172.21.15.1:6792/0 conn(0x7f86580a67c0 legacy=0x7f86580a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 1 -- 172.21.15.1:0/1125025355 shutdown_connections 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 1 -- 172.21.15.1:0/1125025355 >> 172.21.15.1:0/1125025355 conn(0x7f86580ac4b0 msgr2=0x7f86580a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 1 -- 172.21.15.1:0/1125025355 shutdown_connections 2024-08-19T09:50:26.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 1 -- 172.21.15.1:0/1125025355 wait complete. 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 10 monclient: build_initial_monmap 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.260+0000 7f865f070640 10 monclient: monmap: 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:26.276 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 librados: starting msgr at 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 librados: starting objecter 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 Processor -- start 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 -- start start 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 librados: setting wanted keys 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 librados: calling monclient init 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient: init 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:26.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient: _add_conns ranks=[6,2,1,8,4,3,0,5,7] 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient(hunting): picked mon.noname-d con 0x7f86580a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient(hunting): picked mon.noname-b con 0x7f86580a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient(hunting): picked mon.noname-f con 0x7f865813f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a1870 con 0x7f865813f5c0 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8658142bd0 con 0x7f86580a6ba0 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865813a9b0 con 0x7f86580a67c0 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.261+0000 7f865f070640 10 monclient(hunting): authenticate will time out at 682.338945s 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.262+0000 7f865ce0c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f86580a67c0 0x7f86580a01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:53038/0 (socket says 172.21.15.1:53038) 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.262+0000 7f865ce0c640 1 -- 172.21.15.1:0/662981251 learned_addr learned my addr 172.21.15.1:0/662981251 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865ce0c640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.1:6792/0 conn(0x7f86580a67c0 legacy=0x7f86580a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 client.?.objecter ms_handle_connect 0x7f86580a67c0 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 client.?.objecter ms_handle_connect 0x7f86580a6ba0 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864ffff640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.1:6790/0 conn(0x7f86580a6ba0 legacy=0x7f86580a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 client.?.objecter ms_handle_connect 0x7f865813f5c0 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865d60d640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6789/0 conn(0x7f865813f5c0 legacy=0x7f86581d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2205477120 0 0) 0x7f865813a9b0 con 0x7f86580a67c0 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): my global_id is 4434 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8644004eb0 con 0x7f86580a67c0 2024-08-19T09:50:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034413476 0 0) 0x7f8658142bd0 con 0x7f86580a6ba0 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): my global_id is 4457 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f865813a9b0 con 0x7f86580a6ba0 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095396753 0 0) 0x7f86580a1870 con 0x7f865813f5c0 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): my global_id is 4411 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8658142bd0 con 0x7f865813f5c0 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2122881943 0 0) 0x7f865813a9b0 con 0x7f86580a6ba0 2024-08-19T09:50:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86580a1870 con 0x7f86580a6ba0 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1563424168 0 0) 0x7f8644004eb0 con 0x7f86580a67c0 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f865813a9b0 con 0x7f86580a67c0 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8648003610 con 0x7f86580a6ba0 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): dump: 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 801675682 0 0) 0x7f8658142bd0 con 0x7f865813f5c0 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8644004eb0 con 0x7f865813f5c0 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8654004d90 con 0x7f86580a67c0 2024-08-19T09:50:26.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): dump: 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.283 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8650002070 con 0x7f865813f5c0 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient(hunting): dump: 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.284 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 799340333 0 0) 0x7f86580a1870 con 0x7f86580a6ba0 2024-08-19T09:50:26.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.1:6792/0 conn(0x7f86580a67c0 legacy=0x7f86580a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6789/0 conn(0x7f865813f5c0 legacy=0x7f86581d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 monclient: found mon.b 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86580a2890 con 0x7f86580a6ba0 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient: _finish_auth 0 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f864d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 5 monclient: authenticate success, global_id 4457 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 10 monclient: _renew_subs 2024-08-19T09:50:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f865813a110 con 0x7f86580a6ba0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 10 monclient: _renew_subs 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f865813f290 con 0x7f86580a6ba0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.264+0000 7f865f070640 1 librados: init done 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8648003b90 con 0x7f86580a6ba0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86480040c0 con 0x7f86580a6ba0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864effd640 4 set_mon_vals no callback set 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 10 monclient: dump: 2024-08-19T09:50:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f865f070640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86380065c0 con 0x7f8638002860 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.265+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8648004260 con 0x7f86580a6ba0 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.266+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f864804dad0 con 0x7f86580a6ba0 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.266+0000 7f864d7fa640 10 client.4457.objecter ms_dispatch 0x7f8658000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.266+0000 7f864d7fa640 3 client.4457.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.266+0000 7f864d7fa640 3 client.4457.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.266+0000 7f864d7fa640 20 client.4457.objecter dump_active .. 0 homeless 2024-08-19T09:50:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.266+0000 7f865d60d640 10 monclient: get_auth_request con 0x7f8644041910 auth_method 0 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 client.4457.objecter ms_handle_connect 0x7f8638002860 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 client.4457.objecter resend_mon_ops 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f865ce0c640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6791/0 conn(0x7f8638002860 legacy=0x7f8638004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1203851556 0 0) 0x7f86380065c0 con 0x7f8638002860 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient(hunting): my global_id is 4541 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86580a1870 con 0x7f8638002860 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 client.4457.objecter ms_handle_connect 0x7f8644041910 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4085702968 0 0) 0x7f86580a1870 con 0x7f8638002860 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86380065c0 con 0x7f8638002860 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86540048a0 con 0x7f8638002860 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 10 monclient: dump: 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1861728941 0 0) 0x7f86380065c0 con 0x7f8638002860 2024-08-19T09:50:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.268+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f863800a160 con 0x7f8638002860 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.269+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8654005740 con 0x7f8638002860 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.269+0000 7f864d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.269+0000 7f864d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.269+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6791/0 conn(0x7f8638002860 legacy=0x7f8638004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.269+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6791/0 conn(0x7f8638002860 legacy=0x7f8638004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.275+0000 7f865f070640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.275+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8638009e50 con 0x7f8638009110 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f865ce0c640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6791/0 conn(0x7f8638009110 legacy=0x7f86380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 10 client.4457.objecter ms_handle_connect 0x7f8638009110 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 10 client.4457.objecter resend_mon_ops 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980197392 0 0) 0x7f8638009e50 con 0x7f8638009110 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 10 monclient(hunting): my global_id is 4550 2024-08-19T09:50:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.277+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86380065c0 con 0x7f8638009110 2024-08-19T09:50:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3126810778 0 0) 0x7f86380065c0 con 0x7f8638009110 2024-08-19T09:50:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8638009e50 con 0x7f8638009110 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8654005f60 con 0x7f8638009110 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 10 monclient: dump: 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 374630390 0 0) 0x7f8638009e50 con 0x7f8638009110 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f863800c3a0 con 0x7f8638009110 2024-08-19T09:50:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7683 (unknown 0 0 3784131329) 0x7f8654007380 con 0x7f8638009110 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6791/0 conn(0x7f8638009110 legacy=0x7f86380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.278+0000 7f864d7fa640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6791/0 conn(0x7f8638009110 legacy=0x7f86380095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:26.301 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 32545, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 31, 2024-08-19T09:50:26.302 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.303 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.304 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:26.305 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.306 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:26.307 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.308 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.309 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:26.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.312 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:26.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:26.314 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:26.317 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:26.318 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:26.319 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:26.321 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 10 librados: watch_flush enter 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 10 librados: watch_flush exit 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 20 client.4457.objecter shutdown clearing up homeless session... 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 10 client.4457.objecter successfully canceled tick 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8644041910 legacy=0x7f8644043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 10 monclient: shutdown 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 >> v1:172.21.15.1:6790/0 conn(0x7f86580a6ba0 legacy=0x7f86580a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865d60d640 1 -- 172.21.15.1:0/662981251 reap_dead start 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 shutdown_connections 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 >> 172.21.15.1:0/662981251 conn(0x7f86580ac4b0 msgr2=0x7f8658138750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 shutdown_connections 2024-08-19T09:50:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 -- 172.21.15.1:0/662981251 wait complete. 2024-08-19T09:50:26.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.279+0000 7f865f070640 1 librados: shutdown 2024-08-19T09:50:26.324 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:26.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:50:26.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:50:26.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.422+0000 7fae4004f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:26.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.422+0000 7fae4004f640 10 monclient: build_initial_monmap 2024-08-19T09:50:26.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.422+0000 7fae4004f640 10 monclient: monmap: 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:26.424522+0000 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:26.424522+0000 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:26.425 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:26.426 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:26.426 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:26.426 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:26.426 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 1 Processor -- start 2024-08-19T09:50:26.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 1 -- start start 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient: init 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient: _add_conns ranks=[5,8,7,3,4,0,2,6,1] 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient(hunting): picked mon.noname-h con 0x7fae380a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient(hunting): picked mon.noname-e con 0x7fae380a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient(hunting): picked mon.noname-i con 0x7fae3813f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae380a3b40 con 0x7fae380a67c0 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae380a8130 con 0x7fae3813f5c0 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae3806ede0 con 0x7fae380a6ba0 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.423+0000 7fae4004f640 10 monclient(hunting): authenticate will time out at 682.500934s 2024-08-19T09:50:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.424+0000 7fae3d5ea640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fae380a6ba0 0x7fae380a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:44856/0 (socket says 172.21.15.1:44856) 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.424+0000 7fae3d5ea640 1 -- 172.21.15.1:0/1192504195 learned_addr learned my addr 172.21.15.1:0/1192504195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3d5ea640 1 -- 172.21.15.1:0/1192504195 >> v1:172.21.15.1:6793/0 conn(0x7fae380a6ba0 legacy=0x7fae380a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3e5ec640 1 -- 172.21.15.1:0/1192504195 >> v1:172.21.15.82:6792/0 conn(0x7fae3813f5c0 legacy=0x7fae381419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3ddeb640 1 -- 172.21.15.1:0/1192504195 >> v1:172.21.15.82:6791/0 conn(0x7fae380a67c0 legacy=0x7fae380a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640344160 0 0) 0x7fae3806ede0 con 0x7fae380a6ba0 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): my global_id is 4472 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae24003520 con 0x7fae380a6ba0 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028558824 0 0) 0x7fae380a8130 con 0x7fae3813f5c0 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): my global_id is 4426 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae3806ede0 con 0x7fae3813f5c0 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069404088 0 0) 0x7fae380a3b40 con 0x7fae380a67c0 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient(hunting): my global_id is 4559 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae380a8130 con 0x7fae380a67c0 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2081792086 0 0) 0x7fae24003520 con 0x7fae380a6ba0 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 >> v1:172.21.15.82:6792/0 conn(0x7fae3813f5c0 legacy=0x7fae381419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 >> v1:172.21.15.82:6791/0 conn(0x7fae380a67c0 legacy=0x7fae380a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: _finish_hunting 0 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 monclient: found mon.noname-e 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae380a92a0 con 0x7fae380a6ba0 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: _finish_auth 0 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: _check_auth_tickets 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae28001ad0 con 0x7fae380a6ba0 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae4004f640 5 monclient: authenticate success, global_id 4472 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae4004f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: dump: 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae4004f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae280048a0 con 0x7fae380a6ba0 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae4004f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 1 -- 172.21.15.1:0/1192504195 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae28004e10 con 0x7fae380a6ba0 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae4004f640 4 set_mon_vals no callback set 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:26.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.426+0000 7fae3cde9640 10 monclient: dump: 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 10 monclient: shutdown 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 1 -- 172.21.15.1:0/1192504195 >> v1:172.21.15.1:6793/0 conn(0x7fae380a6ba0 legacy=0x7fae380a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 1 -- 172.21.15.1:0/1192504195 shutdown_connections 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 1 -- 172.21.15.1:0/1192504195 >> 172.21.15.1:0/1192504195 conn(0x7fae380ac4b0 msgr2=0x7fae380a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 1 -- 172.21.15.1:0/1192504195 shutdown_connections 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 1 -- 172.21.15.1:0/1192504195 wait complete. 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 10 monclient: build_initial_monmap 2024-08-19T09:50:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.427+0000 7fae4004f640 10 monclient: monmap: 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 librados: starting msgr at 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 librados: starting objecter 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 Processor -- start 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 -- start start 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 librados: setting wanted keys 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 librados: calling monclient init 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient: init 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient: _add_conns ranks=[1,4,3,0,2,5,6,8,7] 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient(hunting): picked mon.noname-f con 0x7fae380a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient(hunting): picked mon.noname-c con 0x7fae380a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient(hunting): picked mon.noname-g con 0x7fae3813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae3813ad70 con 0x7fae380a67c0 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae38139880 con 0x7fae3813f5c0 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae3813bbe0 con 0x7fae380a6ba0 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae4004f640 10 monclient(hunting): authenticate will time out at 682.505832s 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae3d5ea640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fae380a6ba0 0x7fae3813c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57674/0 (socket says 172.21.15.1:57674) 2024-08-19T09:50:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.428+0000 7fae3d5ea640 1 -- 172.21.15.1:0/413748121 learned_addr learned my addr 172.21.15.1:0/413748121 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae3d5ea640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.1:6791/0 conn(0x7fae380a6ba0 legacy=0x7fae3813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 client.?.objecter ms_handle_connect 0x7fae380a6ba0 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae3ddeb640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6789/0 conn(0x7fae380a67c0 legacy=0x7fae3813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 client.?.objecter ms_handle_connect 0x7fae380a67c0 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae3e5ec640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6790/0 conn(0x7fae3813f5c0 legacy=0x7fae381d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 client.?.objecter ms_handle_connect 0x7fae3813f5c0 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3492643362 0 0) 0x7fae3813bbe0 con 0x7fae380a6ba0 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): my global_id is 4540 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae1c004ee0 con 0x7fae380a6ba0 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122196768 0 0) 0x7fae3813ad70 con 0x7fae380a67c0 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): my global_id is 4420 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae3813bbe0 con 0x7fae380a67c0 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368283690 0 0) 0x7fae38139880 con 0x7fae3813f5c0 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): my global_id is 4494 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae3813ad70 con 0x7fae3813f5c0 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1539874506 0 0) 0x7fae1c004ee0 con 0x7fae380a6ba0 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae38139880 con 0x7fae380a6ba0 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae28004d90 con 0x7fae380a6ba0 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3379971324 0 0) 0x7fae3813bbe0 con 0x7fae380a67c0 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae1c004ee0 con 0x7fae380a67c0 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae34003610 con 0x7fae380a67c0 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3227442542 0 0) 0x7fae3813ad70 con 0x7fae3813f5c0 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae3813bbe0 con 0x7fae3813f5c0 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae30002070 con 0x7fae3813f5c0 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient(hunting): dump: 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.455 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 262735226 0 0) 0x7fae38139880 con 0x7fae380a6ba0 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6790/0 conn(0x7fae3813f5c0 legacy=0x7fae381d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6789/0 conn(0x7fae380a67c0 legacy=0x7fae3813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 monclient: found mon.c 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3813b340 con 0x7fae380a6ba0 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:26.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 5 monclient: authenticate success, global_id 4540 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 10 monclient: _renew_subs 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fae3813a680 con 0x7fae380a6ba0 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 10 monclient: _renew_subs 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fae3813cf00 con 0x7fae380a6ba0 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae4004f640 1 librados: init done 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae280048a0 con 0x7fae380a6ba0 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae28001d20 con 0x7fae380a6ba0 2024-08-19T09:50:26.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2f7fe640 4 set_mon_vals no callback set 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.431+0000 7fae2e7fc640 10 monclient: dump: 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.458 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae4004f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fae200065c0 con 0x7fae20002860 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fae28011a00 con 0x7fae380a6ba0 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fae2804daa0 con 0x7fae380a6ba0 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae2e7fc640 10 client.4540.objecter ms_dispatch 0x7fae38000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:26.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae2e7fc640 3 client.4540.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.432+0000 7fae2e7fc640 3 client.4540.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.433+0000 7fae3e5ec640 10 monclient: get_auth_request con 0x7fae1c0419a0 auth_method 0 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.433+0000 7fae2e7fc640 20 client.4540.objecter dump_active .. 0 homeless 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae3ddeb640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6792/0 conn(0x7fae20002860 legacy=0x7fae20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 10 client.4540.objecter ms_handle_connect 0x7fae20002860 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 10 client.4540.objecter resend_mon_ops 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2529279124 0 0) 0x7fae200065c0 con 0x7fae20002860 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 10 monclient(hunting): my global_id is 4435 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.434+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae38139880 con 0x7fae20002860 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3119062902 0 0) 0x7fae38139880 con 0x7fae20002860 2024-08-19T09:50:26.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae200065c0 con 0x7fae20002860 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 10 client.4540.objecter ms_handle_connect 0x7fae1c0419a0 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae34003d60 con 0x7fae20002860 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 10 monclient: dump: 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.461 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3946564954 0 0) 0x7fae200065c0 con 0x7fae20002860 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.435+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae2000a160 con 0x7fae20002860 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.436+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fae34003e50 con 0x7fae20002860 2024-08-19T09:50:26.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.436+0000 7fae2e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.436+0000 7fae2e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.436+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6792/0 conn(0x7fae20002860 legacy=0x7fae20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.436+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6792/0 conn(0x7fae20002860 legacy=0x7fae20004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.442+0000 7fae4004f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.442+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fae20009e50 con 0x7fae20009110 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae3ddeb640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6792/0 conn(0x7fae20009110 legacy=0x7fae200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 10 client.4540.objecter ms_handle_connect 0x7fae20009110 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 10 client.4540.objecter resend_mon_ops 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701748586 0 0) 0x7fae20009e50 con 0x7fae20009110 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 10 monclient(hunting): my global_id is 4444 2024-08-19T09:50:26.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.444+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae200065c0 con 0x7fae20009110 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2807969189 0 0) 0x7fae200065c0 con 0x7fae20009110 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae20009e50 con 0x7fae20009110 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae34004300 con 0x7fae20009110 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 10 monclient: dump: 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:26.464 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2205000739 0 0) 0x7fae20009e50 con 0x7fae20009110 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae2000c3f0 con 0x7fae20009110 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 4195588070) 0x7fae34005b80 con 0x7fae20009110 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:26.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6792/0 conn(0x7fae20009110 legacy=0x7fae200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae2e7fc640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6792/0 conn(0x7fae20009110 legacy=0x7fae200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 4, 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 32705, 2024-08-19T09:50:26.466 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 31, 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:26.467 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.468 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.469 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:26.470 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:26.471 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.473 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:26.474 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.475 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:26.480 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:26.481 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:26.485 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:50:26.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:26.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:26.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:26.486 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:26.486 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 10 librados: watch_flush enter 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 10 librados: watch_flush exit 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 20 client.4540.objecter shutdown clearing up homeless session... 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 10 client.4540.objecter successfully canceled tick 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.82:6812/2549104415 conn(0x7fae1c0419a0 legacy=0x7fae1c043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 10 monclient: shutdown 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.445+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 >> v1:172.21.15.1:6791/0 conn(0x7fae380a6ba0 legacy=0x7fae3813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae3e5ec640 1 -- 172.21.15.1:0/413748121 reap_dead start 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae3e5ec640 1 -- 172.21.15.1:0/413748121 reap_dead start 2024-08-19T09:50:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 shutdown_connections 2024-08-19T09:50:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 >> 172.21.15.1:0/413748121 conn(0x7fae380ac4b0 msgr2=0x7fae3813ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 shutdown_connections 2024-08-19T09:50:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae4004f640 1 -- 172.21.15.1:0/413748121 wait complete. 2024-08-19T09:50:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:26.446+0000 7fae4004f640 1 librados: shutdown 2024-08-19T09:50:26.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:50:26.489 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g', 'a', 'i'] 2024-08-19T09:50:26.489 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-08-19T09:50:26.490 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-08-19T09:50:26.490 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-08-19T09:50:26.490 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T09:50:26.490 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:50:26.502 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T09:50:26.502 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T09:50:26.502 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T09:50:26.502 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T09:50:26.502 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:50:26.513 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T09:50:26.513 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-08-19T09:50:26.513 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-08-19T09:50:26.513 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-08-19T09:50:26.513 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:50:26.553 INFO:tasks.ceph.mon.i:Stopped 2024-08-19T09:50:26.553 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-08-19T09:50:26.554 INFO:tasks.ceph.mon.b:Sent signal 19 2024-08-19T09:50:26.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:50:41.554 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-08-19T09:50:41.555 INFO:tasks.ceph.mon.b:Sent signal 18 2024-08-19T09:50:41.555 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-08-19T09:50:41.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:50:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.665+0000 7fb201b7f640 10 monclient: get_monmap_and_config 2024-08-19T09:50:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.665+0000 7fb201b7f640 10 monclient: build_initial_monmap 2024-08-19T09:50:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.665+0000 7fb201b7f640 10 monclient: monmap: 2024-08-19T09:50:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:41.666546+0000 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:41.666546+0000 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:41.668 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:41.668 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:41.668 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:41.668 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:41.668 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 1 Processor -- start 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 1 -- start start 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 10 monclient: init 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 10 monclient: _add_conns ranks=[0,4,5,2,7,1,6,3,8] 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 10 monclient(hunting): picked mon.noname-a con 0x7fb1fc0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 10 monclient(hunting): picked mon.noname-c con 0x7fb1fc0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 10 monclient(hunting): picked mon.noname-h con 0x7fb1fc13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0a3b40 con 0x7fb1fc0a67c0 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0a8130 con 0x7fb1fc0a6ba0 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.666+0000 7fb201b7f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc06ede0 con 0x7fb1fc13f5c0 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.667+0000 7fb201b7f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.667+0000 7fb201b7f640 10 monclient(hunting): authenticate will time out at 697.743009s 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.667+0000 7fb1fb7fe640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb1fc0a67c0 legacy=0x7fb1fc0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.667+0000 7fb1faffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb1fc0a6ba0 0x7fb1fc0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:33894/0 (socket says 172.21.15.1:33894) 2024-08-19T09:50:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.667+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 learned_addr learned my addr 172.21.15.1:0/133300801 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:41.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.669+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6791/0 conn(0x7fb1fc0a6ba0 legacy=0x7fb1fc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:41.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.669+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6791/0 conn(0x7fb1fc13f5c0 legacy=0x7fb1fc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:41.868+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1fc0a67c0 legacy=0x7fb1fc0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:42.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:42.269+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1fc0a67c0 legacy=0x7fb1fc0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:43.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:43.070+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1fc0a67c0 legacy=0x7fb1fc0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): tick 2024-08-19T09:50:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:50:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 monclient(hunting): continuing hunt 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6791/0 conn(0x7fb1fc13f5c0 legacy=0x7fb1fc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6791/0 conn(0x7fb1fc0a6ba0 legacy=0x7fb1fc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1fc0a67c0 legacy=0x7fb1fc0a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient: _add_conns ranks=[7,3,2,6,8,1] 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-i con 0x7fb1e80038c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-g con 0x7fb1e8007530 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e800b2a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0a92a0 con 0x7fb1e800b2a0 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0a3b40 con 0x7fb1e8007530 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6790/0 conn(0x7fb1e8007530 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6792/0 conn(0x7fb1e80038c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e8011110 con 0x7fb1e80038c0 2024-08-19T09:50:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.667+0000 7fb1f97fa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.669+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6790/0 conn(0x7fb1e800b2a0 legacy=0x7fb1e800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.868+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6790/0 conn(0x7fb1e8007530 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:44.868+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6792/0 conn(0x7fb1e80038c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:45.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:45.269+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6790/0 conn(0x7fb1e8007530 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:45.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:45.269+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6792/0 conn(0x7fb1e80038c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:46.070+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6790/0 conn(0x7fb1e8007530 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:46.070+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6792/0 conn(0x7fb1e80038c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): tick 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 monclient(hunting): continuing hunt 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6792/0 conn(0x7fb1e80038c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6790/0 conn(0x7fb1e8007530 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6790/0 conn(0x7fb1e800b2a0 legacy=0x7fb1e800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient: _add_conns ranks=[8,6,1] 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 reap_dead start 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-e con 0x7fb1e8011460 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 reap_dead start 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-d con 0x7fb1e800a1a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-f con 0x7fb1e800a5a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e80123b0 con 0x7fb1e800a5a0 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0a3b40 con 0x7fb1e800a1a0 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e8011110 con 0x7fb1e8011460 2024-08-19T09:50:47.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.667+0000 7fb1f97fa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:47.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.670+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6792/0 conn(0x7fb1e800a1a0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:47.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.670+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6793/0 conn(0x7fb1e8011460 legacy=0x7fb1e8011840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:47.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:47.670+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6789/0 conn(0x7fb1e800a5a0 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): tick 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 monclient(hunting): continuing hunt 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6793/0 conn(0x7fb1e8011460 legacy=0x7fb1e8011840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6792/0 conn(0x7fb1e800a1a0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.82:6789/0 conn(0x7fb1e800a5a0 legacy=0x7fb1e80099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient: _add_conns ranks=[4,6,0,1,8,2,7,5,3] 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e800df30 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-d con 0x7fb1e800e370 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80064c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e800aa70 con 0x7fb1e80064c0 2024-08-19T09:50:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e80068d0 con 0x7fb1e800df30 2024-08-19T09:50:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1e80064c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e8006aa0 con 0x7fb1e800e370 2024-08-19T09:50:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.668+0000 7fb1f97fa640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.670+0000 7fb1fb7fe640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6791/0 conn(0x7fb1e800df30 legacy=0x7fb1e800d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.670+0000 7fb1faffd640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6792/0 conn(0x7fb1e800e370 legacy=0x7fb1e800e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:50.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:50.869+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1e80064c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:51.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.270+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1e80064c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.562+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722155360 0 0) 0x7fb1e80068d0 con 0x7fb1e800df30 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient(hunting): my global_id is 4549 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e4003520 con 0x7fb1e800df30 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2013808604 0 0) 0x7fb1e4003520 con 0x7fb1e800df30 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6792/0 conn(0x7fb1e800e370 legacy=0x7fb1e800e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6789/0 conn(0x7fb1e80064c0 legacy=0x7fb1e8005d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:51.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 monclient: found mon.noname-c 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e8006d30 con 0x7fb1e800df30 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb1ec0045b0 con 0x7fb1e800df30 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: dump: 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.566 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.567 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb201b7f640 5 monclient: authenticate success, global_id 4549 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb201b7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1ec004800 con 0x7fb1e800df30 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 1 -- 172.21.15.1:0/133300801 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb1ec004d70 con 0x7fb1e800df30 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb1fa7fc640 10 monclient: dump: 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.568 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb201b7f640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:51.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.563+0000 7fb201b7f640 4 set_mon_vals no callback set 2024-08-19T09:50:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 10 monclient: shutdown 2024-08-19T09:50:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 1 -- 172.21.15.1:0/133300801 >> v1:172.21.15.1:6791/0 conn(0x7fb1e800df30 legacy=0x7fb1e800d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb1fbfff640 1 -- 172.21.15.1:0/133300801 reap_dead start 2024-08-19T09:50:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 1 -- 172.21.15.1:0/133300801 shutdown_connections 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 1 -- 172.21.15.1:0/133300801 >> 172.21.15.1:0/133300801 conn(0x7fb1fc0ac4b0 msgr2=0x7fb1fc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 1 -- 172.21.15.1:0/133300801 shutdown_connections 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.564+0000 7fb201b7f640 1 -- 172.21.15.1:0/133300801 wait complete. 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.565+0000 7fb201b7f640 10 monclient: build_initial_monmap 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.565+0000 7fb201b7f640 10 monclient: monmap: 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 1 librados: starting msgr at 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 1 librados: starting objecter 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 1 Processor -- start 2024-08-19T09:50:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 1 -- start start 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 1 librados: setting wanted keys 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 1 librados: calling monclient init 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.566+0000 7fb201b7f640 10 monclient: init 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient: _add_conns ranks=[7,3,1,4,6,5,0,8,2] 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient(hunting): picked mon.noname-i con 0x7fb1e800df30 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient(hunting): picked mon.noname-g con 0x7fb1fc13c0d0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient(hunting): picked mon.noname-f con 0x7fb1fc1d46d0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc127460 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0a2070 con 0x7fb1fc13c0d0 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0aa6e0 con 0x7fb1e800df30 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb1fb7fe640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7fb1e800df30 legacy=0x7fb1fc13b9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb201b7f640 10 monclient(hunting): authenticate will time out at 707.643720s 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb1faffd640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fb1fc13c0d0 legacy=0x7fb1fc1d2fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb1fbfff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb1fc1d46d0 0x7fb1fc1d6ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:47884/0 (socket says 172.21.15.1:47884) 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.567+0000 7fb1fbfff640 1 -- 172.21.15.1:0/1858844290 learned_addr learned my addr 172.21.15.1:0/1858844290 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1fbfff640 1 -- 172.21.15.1:0/1858844290 >> v1:172.21.15.82:6789/0 conn(0x7fb1fc1d46d0 legacy=0x7fb1fc1d6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 client.?.objecter ms_handle_connect 0x7fb1fc1d46d0 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316708735 0 0) 0x7fb1fc127460 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): my global_id is 4429 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e800aa70 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1683646617 0 0) 0x7fb1e800aa70 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1fc127460 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb1f4001ed0 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): dump: 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 466996964 0 0) 0x7fb1fc127460 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 >> v1:172.21.15.82:6792/0 conn(0x7fb1e800df30 legacy=0x7fb1fc13b9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 >> v1:172.21.15.82:6790/0 conn(0x7fb1fc13c0d0 legacy=0x7fb1fc1d2fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 monclient: found mon.f 2024-08-19T09:50:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc125100 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 5 monclient: authenticate success, global_id 4429 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 10 monclient: _renew_subs 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc127180 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 10 monclient: _renew_subs 2024-08-19T09:50:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1fc125c60 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb201b7f640 1 librados: init done 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1f4003c40 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb1f4004130 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1f9ffb640 4 set_mon_vals no callback set 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.570+0000 7fb1e3fff640 10 monclient: dump: 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb1f40043d0 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb201b7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb201b7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb201b7f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1dc005080 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fb1f404da90 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1e3fff640 10 client.4429.objecter ms_dispatch 0x7fb1fc000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1e3fff640 3 client.4429.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1e3fff640 3 client.4429.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1fb7fe640 10 monclient: get_auth_request con 0x7fb1e8043860 auth_method 0 2024-08-19T09:50:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.572+0000 7fb1e3fff640 20 client.4429.objecter dump_active .. 0 homeless 2024-08-19T09:50:51.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.575+0000 7fb1e3fff640 10 client.4429.objecter ms_handle_connect 0x7fb1e8043860 2024-08-19T09:50:51.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.582+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb1f40175b0 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.582+0000 7fb1e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:51.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.582+0000 7fb1e3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.846+0000 7fb201b7f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:50:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.846+0000 7fb201b7f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:50:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.846+0000 7fb201b7f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:50:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.846+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb1dc005370 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.847+0000 7fb1e3fff640 1 -- 172.21.15.1:0/1858844290 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2684 (unknown 254746657 0 4219545496) 0x7fb1f4016eb0 con 0x7fb1fc1d46d0 2024-08-19T09:50:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.847+0000 7fb1e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:50:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.847+0000 7fb1e3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:51.849 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":10,"quorum":[1,2,4,5,6,8],"quorum_names":["f","b","c","h","d","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:50:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 10 librados: watch_flush enter 2024-08-19T09:50:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 10 librados: watch_flush exit 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 20 client.4429.objecter shutdown clearing up homeless session... 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 10 client.4429.objecter successfully canceled tick 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb1e8043860 legacy=0x7fb1e8045d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 10 monclient: shutdown 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 >> v1:172.21.15.82:6789/0 conn(0x7fb1fc1d46d0 legacy=0x7fb1fc1d6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 shutdown_connections 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 >> 172.21.15.1:0/1858844290 conn(0x7fb1fc0ac4b0 msgr2=0x7fb1fc0aab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 shutdown_connections 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 -- 172.21.15.1:0/1858844290 wait complete. 2024-08-19T09:50:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.849+0000 7fb201b7f640 1 librados: shutdown 2024-08-19T09:50:51.861 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-08-19T09:50:51.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-08-19T09:50:51.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:50:51.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.959+0000 7fe555e60640 10 monclient: get_monmap_and_config 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.959+0000 7fe555e60640 10 monclient: build_initial_monmap 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.959+0000 7fe555e60640 10 monclient: monmap: 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:51.960755+0000 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:51.960755+0000 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:51.962 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.960+0000 7fe555e60640 1 Processor -- start 2024-08-19T09:50:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.960+0000 7fe555e60640 1 -- start start 2024-08-19T09:50:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.960+0000 7fe555e60640 10 monclient: init 2024-08-19T09:50:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient: _add_conns ranks=[1,3,6,7,0,8,2,5,4] 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient(hunting): picked mon.noname-f con 0x7fe5500a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient(hunting): picked mon.noname-g con 0x7fe5500a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient(hunting): picked mon.noname-d con 0x7fe55013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5500a3b40 con 0x7fe5500a67c0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5500a8130 con 0x7fe5500a6ba0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55006ede0 con 0x7fe55013f5c0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe555e60640 10 monclient(hunting): authenticate will time out at 708.037113s 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe54effd640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fe5500a6ba0 legacy=0x7fe5500a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe54ffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe55013f5c0 0x7fe5501419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:58310/0 (socket says 172.21.15.1:58310) 2024-08-19T09:50:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.961+0000 7fe54ffff640 1 -- 172.21.15.1:0/1865343273 learned_addr learned my addr 172.21.15.1:0/1865343273 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54ffff640 1 -- 172.21.15.1:0/1865343273 >> v1:172.21.15.1:6792/0 conn(0x7fe55013f5c0 legacy=0x7fe5501419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54f7fe640 1 -- 172.21.15.1:0/1865343273 >> v1:172.21.15.82:6789/0 conn(0x7fe5500a67c0 legacy=0x7fe5500a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74822898 0 0) 0x7fe55006ede0 con 0x7fe55013f5c0 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): my global_id is 4443 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe53c003520 con 0x7fe55013f5c0 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148309833 0 0) 0x7fe5500a3b40 con 0x7fe5500a67c0 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient(hunting): my global_id is 4438 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe55006ede0 con 0x7fe5500a67c0 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 446248251 0 0) 0x7fe53c003520 con 0x7fe55013f5c0 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 >> v1:172.21.15.82:6790/0 conn(0x7fe5500a6ba0 legacy=0x7fe5500a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 >> v1:172.21.15.82:6789/0 conn(0x7fe5500a67c0 legacy=0x7fe5500a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 monclient: found mon.noname-d 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5500a92a0 con 0x7fe55013f5c0 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe555e60640 5 monclient: authenticate success, global_id 4443 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe548001e50 con 0x7fe55013f5c0 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe555e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: dump: 2024-08-19T09:50:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe555e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe548004c20 con 0x7fe55013f5c0 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe555e60640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 1 -- 172.21.15.1:0/1865343273 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe548005150 con 0x7fe55013f5c0 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe555e60640 4 set_mon_vals no callback set 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.963+0000 7fe54e7fc640 10 monclient: dump: 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 10 monclient: shutdown 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 1 -- 172.21.15.1:0/1865343273 >> v1:172.21.15.1:6792/0 conn(0x7fe55013f5c0 legacy=0x7fe5501419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 1 -- 172.21.15.1:0/1865343273 shutdown_connections 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 1 -- 172.21.15.1:0/1865343273 >> 172.21.15.1:0/1865343273 conn(0x7fe5500ac4b0 msgr2=0x7fe5500a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 1 -- 172.21.15.1:0/1865343273 shutdown_connections 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.964+0000 7fe555e60640 1 -- 172.21.15.1:0/1865343273 wait complete. 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 10 monclient: build_initial_monmap 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 10 monclient: monmap: 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 1 librados: starting msgr at 2024-08-19T09:50:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 1 librados: starting objecter 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 1 Processor -- start 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 1 -- start start 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 1 librados: setting wanted keys 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 1 librados: calling monclient init 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.965+0000 7fe555e60640 10 monclient: init 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient: _add_conns ranks=[5,6,3,2,0,8,7,4,1] 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient(hunting): picked mon.noname-h con 0x7fe5500a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient(hunting): picked mon.noname-d con 0x7fe5500a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient(hunting): picked mon.noname-g con 0x7fe55013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55013a9b0 con 0x7fe55013f5c0 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe550142db0 con 0x7fe5500a67c0 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5500a1680 con 0x7fe5500a6ba0 2024-08-19T09:50:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe555e60640 10 monclient(hunting): authenticate will time out at 708.042174s 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe54ffff640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fe55013f5c0 legacy=0x7fe5500a0c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe54effd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe5500a6ba0 0x7fe5500a0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:58318/0 (socket says 172.21.15.1:58318) 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.966+0000 7fe54effd640 1 -- 172.21.15.1:0/3243270615 learned_addr learned my addr 172.21.15.1:0/3243270615 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe54effd640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6792/0 conn(0x7fe5500a6ba0 legacy=0x7fe5500a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 client.?.objecter ms_handle_connect 0x7fe5500a6ba0 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 client.?.objecter ms_handle_connect 0x7fe5500a67c0 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe54f7fe640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.82:6791/0 conn(0x7fe5500a67c0 legacy=0x7fe55009fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171154535 0 0) 0x7fe5500a1680 con 0x7fe5500a6ba0 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): my global_id is 4452 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe530004eb0 con 0x7fe5500a6ba0 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860809803 0 0) 0x7fe550142db0 con 0x7fe5500a67c0 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): my global_id is 4568 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5500a1680 con 0x7fe5500a67c0 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1873448831 0 0) 0x7fe530004eb0 con 0x7fe5500a6ba0 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe550142db0 con 0x7fe5500a6ba0 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe538004730 con 0x7fe5500a6ba0 2024-08-19T09:50:51.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): dump: 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1944208792 0 0) 0x7fe5500a1680 con 0x7fe5500a67c0 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe530004eb0 con 0x7fe5500a67c0 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe540003610 con 0x7fe5500a67c0 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient(hunting): dump: 2024-08-19T09:50:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.985 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2830028069 0 0) 0x7fe550142db0 con 0x7fe5500a6ba0 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.82:6791/0 conn(0x7fe5500a67c0 legacy=0x7fe55009fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.82:6790/0 conn(0x7fe55013f5c0 legacy=0x7fe5500a0c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 monclient: found mon.d 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55013a6d0 con 0x7fe5500a6ba0 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe547fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe555e60640 5 monclient: authenticate success, global_id 4452 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe555e60640 10 monclient: _renew_subs 2024-08-19T09:50:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe555e60640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.968+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5501d7260 con 0x7fe5500a6ba0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 10 monclient: _renew_subs 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5501d7610 con 0x7fe5500a6ba0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 1 librados: init done 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe538004990 con 0x7fe5500a6ba0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe547fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe538004e80 con 0x7fe5500a6ba0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe547fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe547fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe547fff640 10 monclient: dump: 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.988 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe54dffb640 4 set_mon_vals no callback set 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.969+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe5340065c0 con 0x7fe534002860 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe5380050d0 con 0x7fe5500a6ba0 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fe53804db90 con 0x7fe5500a6ba0 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe547fff640 10 client.4452.objecter ms_dispatch 0x7fe550000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe547fff640 3 client.4452.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe547fff640 3 client.4452.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe547fff640 20 client.4452.objecter dump_active .. 0 homeless 2024-08-19T09:50:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.970+0000 7fe54ffff640 10 monclient: get_auth_request con 0x7fe5300414e0 auth_method 0 2024-08-19T09:50:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 client.4452.objecter ms_handle_connect 0x7fe534002860 2024-08-19T09:50:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 client.4452.objecter resend_mon_ops 2024-08-19T09:50:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe54f7fe640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6790/0 conn(0x7fe534002860 legacy=0x7fe534004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972124763 0 0) 0x7fe5340065c0 con 0x7fe534002860 2024-08-19T09:50:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient(hunting): my global_id is 4466 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe550142db0 con 0x7fe534002860 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2050659553 0 0) 0x7fe550142db0 con 0x7fe534002860 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5340065c0 con 0x7fe534002860 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe540003e40 con 0x7fe534002860 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient: dump: 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.993 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.994 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3679845352 0 0) 0x7fe5340065c0 con 0x7fe534002860 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe53400a160 con 0x7fe534002860 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 client.4452.objecter ms_handle_connect 0x7fe5300414e0 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe540004110 con 0x7fe534002860 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6790/0 conn(0x7fe534002860 legacy=0x7fe534004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.972+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6790/0 conn(0x7fe534002860 legacy=0x7fe534004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.978+0000 7fe555e60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.978+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe534009e70 con 0x7fe534009130 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe54f7fe640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6790/0 conn(0x7fe534009130 legacy=0x7fe5340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 client.4452.objecter ms_handle_connect 0x7fe534009130 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 client.4452.objecter resend_mon_ops 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4060087124 0 0) 0x7fe534009e70 con 0x7fe534009130 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient(hunting): my global_id is 4475 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5340065c0 con 0x7fe534009130 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1259260126 0 0) 0x7fe5340065c0 con 0x7fe534009130 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe534009e70 con 0x7fe534009130 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe540005b00 con 0x7fe534009130 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient: dump: 2024-08-19T09:50:51.996 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:51.997 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3360405545 0 0) 0x7fe534009e70 con 0x7fe534009130 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe53400c410 con 0x7fe534009130 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7650 (unknown 0 0 3548457581) 0x7fe540007df0 con 0x7fe534009130 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6790/0 conn(0x7fe534009130 legacy=0x7fe5340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.981+0000 7fe547fff640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6790/0 conn(0x7fe534009130 legacy=0x7fe5340095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:51.998 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 10, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 58272, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:51.999 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.000 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.001 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:52.002 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.003 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.006 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.007 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.008 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:52.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:52.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.015 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:52.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.018 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:52.019 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 10 librados: watch_flush enter 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 10 librados: watch_flush exit 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 20 client.4452.objecter shutdown clearing up homeless session... 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 10 client.4452.objecter successfully canceled tick 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe5300414e0 legacy=0x7fe5300439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 10 monclient: shutdown 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 >> v1:172.21.15.1:6792/0 conn(0x7fe5500a6ba0 legacy=0x7fe5500a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe54ffff640 1 -- 172.21.15.1:0/3243270615 reap_dead start 2024-08-19T09:50:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe54ffff640 1 -- 172.21.15.1:0/3243270615 reap_dead start 2024-08-19T09:50:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 shutdown_connections 2024-08-19T09:50:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 >> 172.21.15.1:0/3243270615 conn(0x7fe5500ac4b0 msgr2=0x7fe5500ab4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 shutdown_connections 2024-08-19T09:50:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 -- 172.21.15.1:0/3243270615 wait complete. 2024-08-19T09:50:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:51.982+0000 7fe555e60640 1 librados: shutdown 2024-08-19T09:50:52.022 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:50:52.022 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon c 2024-08-19T09:50:52.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.121+0000 7fa6fa036640 10 monclient: get_monmap_and_config 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.121+0000 7fa6fa036640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.121+0000 7fa6fa036640 10 monclient: monmap: 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:52.122091+0000 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:52.122091+0000 2024-08-19T09:50:52.122 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.123 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.121+0000 7fa6fa036640 1 Processor -- start 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.121+0000 7fa6fa036640 1 -- start start 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.121+0000 7fa6fa036640 10 monclient: init 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient: _add_conns ranks=[4,0,2,1,8,6,5,7,3] 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f40a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f40a3b40 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f40a8130 con 0x7fa6f413f5c0 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f406ede0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6fa036640 10 monclient(hunting): authenticate will time out at 708.198464s 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6f2ffd640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fa6f40a6ba0 legacy=0x7fa6f40a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6f3fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa6f413f5c0 0x7fa6f41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:60792/0 (socket says 172.21.15.1:60792) 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6f37fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fa6f40a67c0 0x7fa6f40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:51502/0 (socket says 172.21.15.1:51502) 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.122+0000 7fa6f3fff640 1 -- 172.21.15.1:0/1276551607 learned_addr learned my addr 172.21.15.1:0/1276551607 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f3fff640 1 -- 172.21.15.1:0/1276551607 >> v1:172.21.15.1:6790/0 conn(0x7fa6f413f5c0 legacy=0x7fa6f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f37fe640 1 -- 172.21.15.1:0/1276551607 >> v1:172.21.15.1:6791/0 conn(0x7fa6f40a67c0 legacy=0x7fa6f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286226224 0 0) 0x7fa6f406ede0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): my global_id is 4558 2024-08-19T09:50:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6dc003520 con 0x7fa6f40a67c0 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583881488 0 0) 0x7fa6f40a8130 con 0x7fa6f413f5c0 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 10 monclient(hunting): my global_id is 4484 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.124+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f406ede0 con 0x7fa6f413f5c0 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3058205900 0 0) 0x7fa6dc003520 con 0x7fa6f40a67c0 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 >> v1:172.21.15.1:6790/0 conn(0x7fa6f413f5c0 legacy=0x7fa6f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 >> v1:172.21.15.1:6789/0 conn(0x7fa6f40a6ba0 legacy=0x7fa6f40a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 monclient: found mon.noname-c 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f40a92a0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e4001a00 con 0x7fa6f40a67c0 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 5 monclient: authenticate success, global_id 4558 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: dump: 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6e4001cd0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 1 -- 172.21.15.1:0/1276551607 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e4004b20 con 0x7fa6f40a67c0 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 4 set_mon_vals no callback set 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6f27fc640 10 monclient: dump: 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 10 monclient: shutdown 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 1 -- 172.21.15.1:0/1276551607 >> v1:172.21.15.1:6791/0 conn(0x7fa6f40a67c0 legacy=0x7fa6f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 1 -- 172.21.15.1:0/1276551607 shutdown_connections 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 1 -- 172.21.15.1:0/1276551607 >> 172.21.15.1:0/1276551607 conn(0x7fa6f40ac4b0 msgr2=0x7fa6f40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 1 -- 172.21.15.1:0/1276551607 shutdown_connections 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.125+0000 7fa6fa036640 1 -- 172.21.15.1:0/1276551607 wait complete. 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 10 monclient: monmap: 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.136 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 1 librados: starting msgr at 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 1 librados: starting objecter 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 1 Processor -- start 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 1 -- start start 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 1 librados: setting wanted keys 2024-08-19T09:50:52.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 1 librados: calling monclient init 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.126+0000 7fa6fa036640 10 monclient: init 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient: _add_conns ranks=[1,6,5,2,8,7,4,3,0] 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient(hunting): picked mon.noname-f con 0x7fa6f40a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient(hunting): picked mon.noname-d con 0x7fa6f40a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient(hunting): picked mon.noname-h con 0x7fa6f413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f41429d0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f41cf020 con 0x7fa6f413f5c0 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f40a3120 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6fa036640 10 monclient(hunting): authenticate will time out at 708.203423s 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6f2ffd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa6f40a6ba0 0x7fa6f41ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:58326/0 (socket says 172.21.15.1:58326) 2024-08-19T09:50:52.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.127+0000 7fa6f2ffd640 1 -- 172.21.15.1:0/1085283553 learned_addr learned my addr 172.21.15.1:0/1085283553 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6f2ffd640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6792/0 conn(0x7fa6f40a6ba0 legacy=0x7fa6f41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 client.?.objecter ms_handle_connect 0x7fa6f40a6ba0 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6f37fe640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.82:6789/0 conn(0x7fa6f40a67c0 legacy=0x7fa6f41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 client.?.objecter ms_handle_connect 0x7fa6f40a67c0 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 client.?.objecter ms_handle_connect 0x7fa6f413f5c0 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6f3fff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.82:6791/0 conn(0x7fa6f413f5c0 legacy=0x7fa6f41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153507617 0 0) 0x7fa6f40a3120 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): my global_id is 4461 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6e0004eb0 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744208837 0 0) 0x7fa6f40a92a0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): my global_id is 4447 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f40a3120 con 0x7fa6f40a67c0 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591611048 0 0) 0x7fa6f41cf020 con 0x7fa6f413f5c0 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 10 monclient(hunting): my global_id is 4577 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.129+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f40a92a0 con 0x7fa6f413f5c0 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3125260187 0 0) 0x7fa6e0004eb0 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6f41cf020 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e80046a0 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): dump: 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 384703972 0 0) 0x7fa6f40a3120 con 0x7fa6f40a67c0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6e0004eb0 con 0x7fa6f40a67c0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3811855954 0 0) 0x7fa6f40a92a0 con 0x7fa6f413f5c0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6f40a3120 con 0x7fa6f413f5c0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e4004b20 con 0x7fa6f40a67c0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient(hunting): dump: 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 970264080 0 0) 0x7fa6f41cf020 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.82:6791/0 conn(0x7fa6f413f5c0 legacy=0x7fa6f41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.82:6789/0 conn(0x7fa6f40a67c0 legacy=0x7fa6f41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 monclient: found mon.d 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f40a15a0 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 5 monclient: authenticate success, global_id 4461 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 10 monclient: _renew_subs 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f413a150 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 10 monclient: _renew_subs 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6f4139d70 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6fa036640 1 librados: init done 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6e8004970 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e8004ee0 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6f1ffb640 4 set_mon_vals no callback set 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.130+0000 7fa6dbfff640 10 monclient: dump: 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.153 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6fa036640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa6d40065c0 con 0x7fa6d4002860 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa6e8005080 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fa6e804daf0 con 0x7fa6f40a6ba0 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6dbfff640 10 client.4461.objecter ms_dispatch 0x7fa6f4000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6dbfff640 3 client.4461.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6dbfff640 3 client.4461.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6dbfff640 20 client.4461.objecter dump_active .. 0 homeless 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.131+0000 7fa6f3fff640 10 monclient: get_auth_request con 0x7fa6e0041720 auth_method 0 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 client.4461.objecter ms_handle_connect 0x7fa6d4002860 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 client.4461.objecter resend_mon_ops 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6f37fe640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6791/0 conn(0x7fa6d4002860 legacy=0x7fa6d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092608461 0 0) 0x7fa6d40065c0 con 0x7fa6d4002860 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient(hunting): my global_id is 4567 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f41cf020 con 0x7fa6d4002860 2024-08-19T09:50:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1391590821 0 0) 0x7fa6f41cf020 con 0x7fa6d4002860 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6d40065c0 con 0x7fa6d4002860 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e40051a0 con 0x7fa6d4002860 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 monclient: dump: 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1631703833 0 0) 0x7fa6d40065c0 con 0x7fa6d4002860 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6d400a160 con 0x7fa6d4002860 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.133+0000 7fa6dbfff640 10 client.4461.objecter ms_handle_connect 0x7fa6e0041720 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.134+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa6e40051a0 con 0x7fa6d4002860 2024-08-19T09:50:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.134+0000 7fa6dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.134+0000 7fa6dbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.134+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6791/0 conn(0x7fa6d4002860 legacy=0x7fa6d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.134+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6791/0 conn(0x7fa6d4002860 legacy=0x7fa6d4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.140+0000 7fa6fa036640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.140+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa6d4009e70 con 0x7fa6d4009130 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.142+0000 7fa6dbfff640 10 client.4461.objecter ms_handle_connect 0x7fa6d4009130 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.142+0000 7fa6f37fe640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6791/0 conn(0x7fa6d4009130 legacy=0x7fa6d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.142+0000 7fa6dbfff640 10 client.4461.objecter resend_mon_ops 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598351902 0 0) 0x7fa6d4009e70 con 0x7fa6d4009130 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient(hunting): my global_id is 4576 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6d40065c0 con 0x7fa6d4009130 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3321649 0 0) 0x7fa6d40065c0 con 0x7fa6d4009130 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6d4009e70 con 0x7fa6d4009130 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa6e4005900 con 0x7fa6d4009130 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient: dump: 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 136472459 0 0) 0x7fa6d4009e70 con 0x7fa6d4009130 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6d400c3f0 con 0x7fa6d4009130 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7650 (unknown 0 0 1113712361) 0x7fa6e4007d70 con 0x7fa6d4009130 2024-08-19T09:50:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6791/0 conn(0x7fa6d4009130 legacy=0x7fa6d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6dbfff640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6791/0 conn(0x7fa6d4009130 legacy=0x7fa6d40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 10, 2024-08-19T09:50:52.161 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 58415, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:52.162 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:52.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:52.165 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.166 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:52.171 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.172 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.174 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.175 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.176 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.177 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.180 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:52.181 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6fa036640 10 librados: watch_flush enter 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6fa036640 10 librados: watch_flush exit 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6fa036640 20 client.4461.objecter shutdown clearing up homeless session... 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.143+0000 7fa6fa036640 10 client.4461.objecter successfully canceled tick 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa6e0041720 legacy=0x7fa6e0043be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 10 monclient: shutdown 2024-08-19T09:50:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 >> v1:172.21.15.1:6792/0 conn(0x7fa6f40a6ba0 legacy=0x7fa6f41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6f3fff640 1 -- 172.21.15.1:0/1085283553 reap_dead start 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6f3fff640 1 -- 172.21.15.1:0/1085283553 reap_dead start 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 shutdown_connections 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 >> 172.21.15.1:0/1085283553 conn(0x7fa6f40ac4b0 msgr2=0x7fa6f40a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 shutdown_connections 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 -- 172.21.15.1:0/1085283553 wait complete. 2024-08-19T09:50:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.144+0000 7fa6fa036640 1 librados: shutdown 2024-08-19T09:50:52.184 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:50:52.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon d 2024-08-19T09:50:52.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:50:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.283+0000 7f0e04304640 10 monclient: get_monmap_and_config 2024-08-19T09:50:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.283+0000 7f0e04304640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.283+0000 7f0e04304640 10 monclient: monmap: 2024-08-19T09:50:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:52.284189+0000 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:52.284189+0000 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.285 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.286 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.283+0000 7f0e04304640 1 Processor -- start 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.283+0000 7f0e04304640 1 -- start start 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.283+0000 7f0e04304640 10 monclient: init 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient: _add_conns ranks=[4,7,1,6,8,0,2,5,3] 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient(hunting): picked mon.noname-c con 0x7f0dfc0a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient(hunting): picked mon.noname-i con 0x7f0dfc0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient(hunting): picked mon.noname-f con 0x7f0dfc0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:52.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc0a3c70 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc0a33d0 con 0x7f0dfc0a6830 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc0a36b0 con 0x7f0dfc0694b0 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e04304640 10 monclient(hunting): authenticate will time out at 708.360517s 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e0189f640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f0dfc0694b0 legacy=0x7f0dfc0a7320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e020a0640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f0dfc0a6830 0x7f0dfc0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:51532/0 (socket says 172.21.15.1:51532) 2024-08-19T09:50:52.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.284+0000 7f0e020a0640 1 -- 172.21.15.1:0/3847259846 learned_addr learned my addr 172.21.15.1:0/3847259846 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e020a0640 1 -- 172.21.15.1:0/3847259846 >> v1:172.21.15.1:6791/0 conn(0x7f0dfc0a6830 legacy=0x7f0dfc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e028a1640 1 -- 172.21.15.1:0/3847259846 >> v1:172.21.15.82:6789/0 conn(0x7f0dfc0a46a0 legacy=0x7f0dfc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427668359 0 0) 0x7f0dfc0a33d0 con 0x7f0dfc0a6830 2024-08-19T09:50:52.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e0109e640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e0109e640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e0109e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e0109e640 10 monclient(hunting): my global_id is 4585 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.286+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0de4003520 con 0x7f0dfc0a6830 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674296766 0 0) 0x7f0dfc0a3c70 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient(hunting): my global_id is 4456 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dfc0a33d0 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 97202043 0 0) 0x7f0de4003520 con 0x7f0dfc0a6830 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 >> v1:172.21.15.82:6792/0 conn(0x7f0dfc0694b0 legacy=0x7f0dfc0a7320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 >> v1:172.21.15.82:6789/0 conn(0x7f0dfc0a46a0 legacy=0x7f0dfc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 monclient: found mon.noname-c 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc06dbb0 con 0x7f0dfc0a6830 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dec001e60 con 0x7f0dfc0a6830 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 5 monclient: authenticate success, global_id 4585 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:50:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: dump: 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0dec004c20 con 0x7f0dfc0a6830 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 1 -- 172.21.15.1:0/3847259846 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dec005190 con 0x7f0dfc0a6830 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 4 set_mon_vals no callback set 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e0109e640 10 monclient: dump: 2024-08-19T09:50:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 10 monclient: shutdown 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 1 -- 172.21.15.1:0/3847259846 >> v1:172.21.15.1:6791/0 conn(0x7f0dfc0a6830 legacy=0x7f0dfc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 1 -- 172.21.15.1:0/3847259846 shutdown_connections 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 1 -- 172.21.15.1:0/3847259846 >> 172.21.15.1:0/3847259846 conn(0x7f0dfc0a9020 msgr2=0x7f0dfc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 1 -- 172.21.15.1:0/3847259846 shutdown_connections 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.287+0000 7f0e04304640 1 -- 172.21.15.1:0/3847259846 wait complete. 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 10 monclient: monmap: 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.302 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 1 librados: starting msgr at 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 1 librados: starting objecter 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 1 Processor -- start 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 1 -- start start 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 1 librados: setting wanted keys 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 1 librados: calling monclient init 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.288+0000 7f0e04304640 10 monclient: init 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient: _add_conns ranks=[5,2,1,8,4,0,3,6,7] 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient(hunting): picked mon.noname-h con 0x7f0dfc0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient(hunting): picked mon.noname-b con 0x7f0dfc0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient(hunting): picked mon.noname-f con 0x7f0dfc0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:52.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc13ab30 con 0x7f0dfc0a6830 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc1cf260 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc0aa030 con 0x7f0dfc0694b0 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e04304640 10 monclient(hunting): authenticate will time out at 708.365429s 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e0189f640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0dfc0a46a0 0x7f0dfc1ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:60798/0 (socket says 172.21.15.1:60798) 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.289+0000 7f0e0189f640 1 -- 172.21.15.1:0/3076303012 learned_addr learned my addr 172.21.15.1:0/3076303012 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0e0189f640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6790/0 conn(0x7f0dfc0a46a0 legacy=0x7f0dfc1ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 client.?.objecter ms_handle_connect 0x7f0dfc0a46a0 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0e020a0640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.82:6791/0 conn(0x7f0dfc0694b0 legacy=0x7f0dfc1ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 client.?.objecter ms_handle_connect 0x7f0dfc0694b0 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 client.?.objecter ms_handle_connect 0x7f0dfc0a6830 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0e028a1640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.82:6789/0 conn(0x7f0dfc0a6830 legacy=0x7f0dfc1d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1111302361 0 0) 0x7f0dfc1cf260 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): my global_id is 4493 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0de8004eb0 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590264583 0 0) 0x7f0dfc0aa030 con 0x7f0dfc0694b0 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): my global_id is 4586 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dfc1cf260 con 0x7f0dfc0694b0 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786108372 0 0) 0x7f0dfc13ab30 con 0x7f0dfc0a6830 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 10 monclient(hunting): my global_id is 4465 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.291+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dfc0aa030 con 0x7f0dfc0a6830 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2950332371 0 0) 0x7f0de8004eb0 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dfc13ab30 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df0004730 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): dump: 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 805545216 0 0) 0x7f0dfc0aa030 con 0x7f0dfc0a6830 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0de8004eb0 con 0x7f0dfc0a6830 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3990043410 0 0) 0x7f0dfc1cf260 con 0x7f0dfc0694b0 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dfc0aa030 con 0x7f0dfc0694b0 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df4003610 con 0x7f0dfc0a6830 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): dump: 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dec005110 con 0x7f0dfc0694b0 2024-08-19T09:50:52.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient(hunting): dump: 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2871988274 0 0) 0x7f0dfc13ab30 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.82:6791/0 conn(0x7f0dfc0694b0 legacy=0x7f0dfc1ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.82:6789/0 conn(0x7f0dfc0a6830 legacy=0x7f0dfc1d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 monclient: found mon.b 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc142b20 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 5 monclient: authenticate success, global_id 4493 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 10 monclient: _renew_subs 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc13a850 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 10 monclient: _renew_subs 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0dfc138250 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e04304640 1 librados: init done 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0df00049d0 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df0004f00 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0e0089d640 4 set_mon_vals no callback set 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.292+0000 7f0dfaffd640 10 monclient: dump: 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0e04304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ddc0065c0 con 0x7f0ddc002860 2024-08-19T09:50:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0df00051d0 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f0df004dc00 con 0x7f0dfc0a46a0 2024-08-19T09:50:52.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0dfaffd640 10 client.4493.objecter ms_dispatch 0x7f0dfc000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:52.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0dfaffd640 3 client.4493.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:52.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0dfaffd640 3 client.4493.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:52.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0dfaffd640 20 client.4493.objecter dump_active .. 0 homeless 2024-08-19T09:50:52.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.293+0000 7f0e028a1640 10 monclient: get_auth_request con 0x7f0de80419e0 auth_method 0 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 client.4493.objecter ms_handle_connect 0x7f0ddc002860 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 client.4493.objecter resend_mon_ops 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0e020a0640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6792/0 conn(0x7f0ddc002860 legacy=0x7f0ddc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462536352 0 0) 0x7f0ddc0065c0 con 0x7f0ddc002860 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient(hunting): my global_id is 4470 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dfc13ab30 con 0x7f0ddc002860 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 988230213 0 0) 0x7f0dfc13ab30 con 0x7f0ddc002860 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ddc0065c0 con 0x7f0ddc002860 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dec001900 con 0x7f0ddc002860 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 monclient: dump: 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1725588232 0 0) 0x7f0ddc0065c0 con 0x7f0ddc002860 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ddc00a160 con 0x7f0ddc002860 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.295+0000 7f0dfaffd640 10 client.4493.objecter ms_handle_connect 0x7f0de80419e0 2024-08-19T09:50:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.296+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0dec0020b0 con 0x7f0ddc002860 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.296+0000 7f0dfaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.296+0000 7f0dfaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.296+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6792/0 conn(0x7f0ddc002860 legacy=0x7f0ddc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.296+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6792/0 conn(0x7f0ddc002860 legacy=0x7f0ddc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.302+0000 7f0e04304640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.302+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ddc009e70 con 0x7f0ddc009130 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0e020a0640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6792/0 conn(0x7f0ddc009130 legacy=0x7f0ddc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 client.4493.objecter ms_handle_connect 0x7f0ddc009130 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 client.4493.objecter resend_mon_ops 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 281460050 0 0) 0x7f0ddc009e70 con 0x7f0ddc009130 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient(hunting): my global_id is 4479 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ddc0065c0 con 0x7f0ddc009130 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1234443410 0 0) 0x7f0ddc0065c0 con 0x7f0ddc009130 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ddc009e70 con 0x7f0ddc009130 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dec005d80 con 0x7f0ddc009130 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 10 monclient: dump: 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 788295820 0 0) 0x7f0ddc009e70 con 0x7f0ddc009130 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.304+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ddc00c410 con 0x7f0ddc009130 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7650 (unknown 0 0 2996766101) 0x7f0dec004c20 con 0x7f0ddc009130 2024-08-19T09:50:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0dfaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0dfaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6792/0 conn(0x7f0ddc009130 legacy=0x7f0ddc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0dfaffd640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6792/0 conn(0x7f0ddc009130 legacy=0x7f0ddc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 10, 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 58562, 2024-08-19T09:50:52.324 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:52.325 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:52.326 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:52.328 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.329 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.337 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.338 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.339 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.340 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.343 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.345 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.345 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:52.345 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 10 librados: watch_flush enter 2024-08-19T09:50:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 10 librados: watch_flush exit 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 20 client.4493.objecter shutdown clearing up homeless session... 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 10 client.4493.objecter successfully canceled tick 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0de80419e0 legacy=0x7f0de8043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 10 monclient: shutdown 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 >> v1:172.21.15.1:6790/0 conn(0x7f0dfc0a46a0 legacy=0x7f0dfc1ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e028a1640 1 -- 172.21.15.1:0/3076303012 reap_dead start 2024-08-19T09:50:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e028a1640 1 -- 172.21.15.1:0/3076303012 reap_dead start 2024-08-19T09:50:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 shutdown_connections 2024-08-19T09:50:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 >> 172.21.15.1:0/3076303012 conn(0x7f0dfc0a9020 msgr2=0x7f0dfc0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 shutdown_connections 2024-08-19T09:50:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 -- 172.21.15.1:0/3076303012 wait complete. 2024-08-19T09:50:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.305+0000 7f0e04304640 1 librados: shutdown 2024-08-19T09:50:52.348 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:50:52.348 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon e 2024-08-19T09:50:52.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.447+0000 7f36a1de9640 10 monclient: get_monmap_and_config 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.447+0000 7f36a1de9640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.447+0000 7f36a1de9640 10 monclient: monmap: 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:52.448868+0000 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:52.448868+0000 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.448+0000 7f36a1de9640 1 Processor -- start 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.448+0000 7f36a1de9640 1 -- start start 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.448+0000 7f36a1de9640 10 monclient: init 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient: _add_conns ranks=[8,1,2,5,7,4,3,0,6] 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient(hunting): picked mon.noname-e con 0x7f369c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient(hunting): picked mon.noname-f con 0x7f369c0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient(hunting): picked mon.noname-b con 0x7f369c13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c0a3b40 con 0x7f369c0a6ba0 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c0a8130 con 0x7f369c13f5c0 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c06ede0 con 0x7f369c0a67c0 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f36a1de9640 10 monclient(hunting): authenticate will time out at 708.525234s 2024-08-19T09:50:52.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f369bfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f369c13f5c0 0x7f369c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:60804/0 (socket says 172.21.15.1:60804) 2024-08-19T09:50:52.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f369bfff640 1 -- 172.21.15.1:0/3174263207 learned_addr learned my addr 172.21.15.1:0/3174263207 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.449+0000 7f369b7fe640 1 --1- 172.21.15.1:0/3174263207 >> v1:172.21.15.1:6793/0 conn(0x7f369c0a67c0 0x7f369c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48418/0 (socket says 172.21.15.1:48418) 2024-08-19T09:50:52.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369b7fe640 1 -- 172.21.15.1:0/3174263207 >> v1:172.21.15.1:6793/0 conn(0x7f369c0a67c0 legacy=0x7f369c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369bfff640 1 -- 172.21.15.1:0/3174263207 >> v1:172.21.15.1:6790/0 conn(0x7f369c13f5c0 legacy=0x7f369c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369affd640 1 -- 172.21.15.1:0/3174263207 >> v1:172.21.15.82:6789/0 conn(0x7f369c0a6ba0 legacy=0x7f369c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747344846 0 0) 0x7f369c06ede0 con 0x7f369c0a67c0 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): my global_id is 4481 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3684003520 con 0x7f369c0a67c0 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429155707 0 0) 0x7f369c0a8130 con 0x7f369c13f5c0 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): my global_id is 4502 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c06ede0 con 0x7f369c13f5c0 2024-08-19T09:50:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339135917 0 0) 0x7f369c0a3b40 con 0x7f369c0a6ba0 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient(hunting): my global_id is 4474 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c0a8130 con 0x7f369c0a6ba0 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2280084912 0 0) 0x7f3684003520 con 0x7f369c0a67c0 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 >> v1:172.21.15.1:6790/0 conn(0x7f369c13f5c0 legacy=0x7f369c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 >> v1:172.21.15.82:6789/0 conn(0x7f369c0a6ba0 legacy=0x7f369c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 monclient: found mon.noname-e 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:52.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c0a92a0 con 0x7f369c0a67c0 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f368c001ad0 con 0x7f369c0a67c0 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f36a1de9640 5 monclient: authenticate success, global_id 4481 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.451+0000 7f36a1de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: dump: 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f368c0047e0 con 0x7f369c0a67c0 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 1 -- 172.21.15.1:0/3174263207 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f368c004d50 con 0x7f369c0a67c0 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f369a7fc640 10 monclient: dump: 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 4 set_mon_vals no callback set 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 10 monclient: shutdown 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 1 -- 172.21.15.1:0/3174263207 >> v1:172.21.15.1:6793/0 conn(0x7f369c0a67c0 legacy=0x7f369c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 1 -- 172.21.15.1:0/3174263207 shutdown_connections 2024-08-19T09:50:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 1 -- 172.21.15.1:0/3174263207 >> 172.21.15.1:0/3174263207 conn(0x7f369c0ac4b0 msgr2=0x7f369c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 1 -- 172.21.15.1:0/3174263207 shutdown_connections 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.452+0000 7f36a1de9640 1 -- 172.21.15.1:0/3174263207 wait complete. 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 10 monclient: monmap: 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.463 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 1 librados: starting msgr at 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 1 librados: starting objecter 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 1 Processor -- start 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 1 -- start start 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 1 librados: setting wanted keys 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 1 librados: calling monclient init 2024-08-19T09:50:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.453+0000 7f36a1de9640 10 monclient: init 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient: _add_conns ranks=[6,4,1,5,2,0,7,8,3] 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient(hunting): picked mon.noname-d con 0x7f369c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient(hunting): picked mon.noname-c con 0x7f369c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient(hunting): picked mon.noname-f con 0x7f369c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c13a560 con 0x7f369c13f5c0 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c13a840 con 0x7f369c0a6ba0 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c139b50 con 0x7f369c0a67c0 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f36a1de9640 10 monclient(hunting): authenticate will time out at 708.530208s 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f369affd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f369c0a6ba0 0x7f369c0a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:51540/0 (socket says 172.21.15.1:51540) 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f369b7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f369c0a67c0 0x7f369c09fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:58348/0 (socket says 172.21.15.1:58348) 2024-08-19T09:50:52.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.454+0000 7f369affd640 1 -- 172.21.15.1:0/359465785 learned_addr learned my addr 172.21.15.1:0/359465785 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f369b7fe640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6792/0 conn(0x7f369c0a67c0 legacy=0x7f369c09fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f369affd640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6791/0 conn(0x7f369c0a6ba0 legacy=0x7f369c0a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f3683fff640 10 client.?.objecter ms_handle_connect 0x7f369c0a67c0 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f3683fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f3683fff640 10 client.?.objecter ms_handle_connect 0x7f369c0a6ba0 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f3683fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.456+0000 7f369bfff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.82:6789/0 conn(0x7f369c13f5c0 legacy=0x7f369c0a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 client.?.objecter ms_handle_connect 0x7f369c13f5c0 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276679794 0 0) 0x7f369c13a840 con 0x7f369c0a6ba0 2024-08-19T09:50:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): my global_id is 4594 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3688004eb0 con 0x7f369c0a6ba0 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518517736 0 0) 0x7f369c13a560 con 0x7f369c13f5c0 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): my global_id is 4483 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c13a840 con 0x7f369c13f5c0 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75232677 0 0) 0x7f369c139b50 con 0x7f369c0a67c0 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): my global_id is 4488 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c13a560 con 0x7f369c0a67c0 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1943751127 0 0) 0x7f369c13a560 con 0x7f369c0a67c0 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f369c139b50 con 0x7f369c0a67c0 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f368c004cd0 con 0x7f369c0a67c0 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): dump: 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3504341853 0 0) 0x7f3688004eb0 con 0x7f369c0a6ba0 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f369c13a560 con 0x7f369c0a6ba0 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.458+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 790254464 0 0) 0x7f369c13a840 con 0x7f369c13f5c0 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3688004eb0 con 0x7f369c13f5c0 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3690003610 con 0x7f369c0a6ba0 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient(hunting): dump: 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3694001f80 con 0x7f369c13f5c0 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient(hunting): dump: 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 273002980 0 0) 0x7f369c139b50 con 0x7f369c0a67c0 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6791/0 conn(0x7f369c0a6ba0 legacy=0x7f369c0a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.82:6789/0 conn(0x7f369c13f5c0 legacy=0x7f369c0a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 monclient: found mon.d 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c142c60 con 0x7f369c0a67c0 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 5 monclient: authenticate success, global_id 4488 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 10 monclient: _renew_subs 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f369c0a2850 con 0x7f369c0a67c0 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 10 monclient: _renew_subs 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f369c139fa0 con 0x7f369c0a67c0 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f36a1de9640 1 librados: init done 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f368c0047e0 con 0x7f369c0a67c0 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f368c001d20 con 0x7f369c0a67c0 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3699ffb640 4 set_mon_vals no callback set 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.459+0000 7f3683fff640 10 monclient: dump: 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f36a1de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f367c0065c0 con 0x7f367c002860 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f368c011940 con 0x7f369c0a67c0 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f368c04d990 con 0x7f369c0a67c0 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f3683fff640 10 client.4488.objecter ms_dispatch 0x7f369c000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f3683fff640 3 client.4488.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f3683fff640 3 client.4488.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f3683fff640 20 client.4488.objecter dump_active .. 0 homeless 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.460+0000 7f369bfff640 10 monclient: get_auth_request con 0x7f3688041be0 auth_method 0 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 client.4488.objecter ms_handle_connect 0x7f367c002860 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 client.4488.objecter resend_mon_ops 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f369affd640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6793/0 conn(0x7f367c002860 legacy=0x7f367c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859924213 0 0) 0x7f367c0065c0 con 0x7f367c002860 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient(hunting): my global_id is 4490 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c139b50 con 0x7f367c002860 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 client.4488.objecter ms_handle_connect 0x7f3688041be0 2024-08-19T09:50:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2328613540 0 0) 0x7f369c139b50 con 0x7f367c002860 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f367c0065c0 con 0x7f367c002860 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3690003e10 con 0x7f367c002860 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.462+0000 7f3683fff640 10 monclient: dump: 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.483 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1235446303 0 0) 0x7f367c0065c0 con 0x7f367c002860 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f367c00a160 con 0x7f367c002860 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3690004080 con 0x7f367c002860 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:52.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6793/0 conn(0x7f367c002860 legacy=0x7f367c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.463+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6793/0 conn(0x7f367c002860 legacy=0x7f367c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.469+0000 7f36a1de9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:52.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.469+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f367c009e50 con 0x7f367c009110 2024-08-19T09:50:52.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.471+0000 7f3683fff640 10 client.4488.objecter ms_handle_connect 0x7f367c009110 2024-08-19T09:50:52.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.471+0000 7f3683fff640 10 client.4488.objecter resend_mon_ops 2024-08-19T09:50:52.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.471+0000 7f369affd640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6793/0 conn(0x7f367c009110 legacy=0x7f367c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339554312 0 0) 0x7f367c009e50 con 0x7f367c009110 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient(hunting): my global_id is 4499 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f367c0065c0 con 0x7f367c009110 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1724935678 0 0) 0x7f367c0065c0 con 0x7f367c009110 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f367c009e50 con 0x7f367c009110 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3690004460 con 0x7f367c009110 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient: dump: 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.487 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.488 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3483305554 0 0) 0x7f367c009e50 con 0x7f367c009110 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f367c00c3f0 con 0x7f367c009110 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7650 (unknown 0 0 2553183053) 0x7f3690007ba0 con 0x7f367c009110 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6793/0 conn(0x7f367c009110 legacy=0x7f367c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f3683fff640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6793/0 conn(0x7f367c009110 legacy=0x7f367c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 10, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 58745, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:52.509 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f36a1de9640 10 librados: watch_flush enter 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f36a1de9640 10 librados: watch_flush exit 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f36a1de9640 20 client.4488.objecter shutdown clearing up homeless session... 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.472+0000 7f36a1de9640 10 client.4488.objecter successfully canceled tick 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3688041be0 legacy=0x7f36880440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 10 monclient: shutdown 2024-08-19T09:50:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 >> v1:172.21.15.1:6792/0 conn(0x7f369c0a67c0 legacy=0x7f369c09fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f369bfff640 1 -- 172.21.15.1:0/359465785 reap_dead start 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f369bfff640 1 -- 172.21.15.1:0/359465785 reap_dead start 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 shutdown_connections 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 >> 172.21.15.1:0/359465785 conn(0x7f369c0ac4b0 msgr2=0x7f369c0aa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 shutdown_connections 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 -- 172.21.15.1:0/359465785 wait complete. 2024-08-19T09:50:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.473+0000 7f36a1de9640 1 librados: shutdown 2024-08-19T09:50:52.512 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:50:52.512 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-08-19T09:50:52.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.612+0000 7fa914098640 10 monclient: get_monmap_and_config 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.612+0000 7fa914098640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.612+0000 7fa914098640 10 monclient: monmap: 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:52.613142+0000 2024-08-19T09:50:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:52.613142+0000 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.614 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.612+0000 7fa914098640 1 Processor -- start 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 1 -- start start 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient: init 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient: _add_conns ranks=[6,5,7,8,1,4,3,0,2] 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient(hunting): picked mon.noname-d con 0x7fa90c0a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient(hunting): picked mon.noname-h con 0x7fa90c0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:52.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient(hunting): picked mon.noname-i con 0x7fa90c0a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c0a3c70 con 0x7fa90c0694b0 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c0a33d0 con 0x7fa90c0a6830 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c0a36b0 con 0x7fa90c0a46a0 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa914098640 10 monclient(hunting): authenticate will time out at 708.689524s 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa912635640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7fa90c0a46a0 legacy=0x7fa90c141850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa911e34640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa90c0a6830 0x7fa90c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:58354/0 (socket says 172.21.15.1:58354) 2024-08-19T09:50:52.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.613+0000 7fa911e34640 1 -- 172.21.15.1:0/892996230 learned_addr learned my addr 172.21.15.1:0/892996230 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.615+0000 7fa911e34640 1 -- 172.21.15.1:0/892996230 >> v1:172.21.15.1:6792/0 conn(0x7fa90c0a6830 legacy=0x7fa90c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.615+0000 7fa911633640 1 -- 172.21.15.1:0/892996230 >> v1:172.21.15.82:6791/0 conn(0x7fa90c0694b0 legacy=0x7fa90c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1581555015 0 0) 0x7fa90c0a3c70 con 0x7fa90c0694b0 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): my global_id is 4595 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f8003520 con 0x7fa90c0694b0 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025253062 0 0) 0x7fa90c0a33d0 con 0x7fa90c0a6830 2024-08-19T09:50:52.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient(hunting): my global_id is 4497 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa90c0a3c70 con 0x7fa90c0a6830 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1515515990 0 0) 0x7fa8f8003520 con 0x7fa90c0694b0 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 >> v1:172.21.15.82:6792/0 conn(0x7fa90c0a46a0 legacy=0x7fa90c141850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 >> v1:172.21.15.1:6792/0 conn(0x7fa90c0a6830 legacy=0x7fa90c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 monclient: found mon.noname-h 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c06dbb0 con 0x7fa90c0694b0 2024-08-19T09:50:52.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa914098640 5 monclient: authenticate success, global_id 4595 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa8fc001bf0 con 0x7fa90c0694b0 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: dump: 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa914098640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8fc003610 con 0x7fa90c0694b0 2024-08-19T09:50:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa914098640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa914098640 4 set_mon_vals no callback set 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 1 -- 172.21.15.1:0/892996230 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa8fc003bc0 con 0x7fa90c0694b0 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.616+0000 7fa910e32640 10 monclient: dump: 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 10 monclient: shutdown 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 -- 172.21.15.1:0/892996230 >> v1:172.21.15.82:6791/0 conn(0x7fa90c0694b0 legacy=0x7fa90c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 -- 172.21.15.1:0/892996230 shutdown_connections 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 -- 172.21.15.1:0/892996230 >> 172.21.15.1:0/892996230 conn(0x7fa90c0a9020 msgr2=0x7fa90c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 -- 172.21.15.1:0/892996230 shutdown_connections 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 -- 172.21.15.1:0/892996230 wait complete. 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 10 monclient: monmap: 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 librados: starting msgr at 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.617+0000 7fa914098640 1 librados: starting objecter 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 Processor -- start 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 -- start start 2024-08-19T09:50:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 librados: setting wanted keys 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 librados: calling monclient init 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient: init 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient: _add_conns ranks=[6,8,3,1,2,7,4,0,5] 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient(hunting): picked mon.noname-d con 0x7fa90c0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient(hunting): picked mon.noname-e con 0x7fa90c0a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient(hunting): picked mon.noname-g con 0x7fa90c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c13b910 con 0x7fa90c0a6830 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c068900 con 0x7fa90c0694b0 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c13a330 con 0x7fa90c0a46a0 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa914098640 10 monclient(hunting): authenticate will time out at 708.694586s 2024-08-19T09:50:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa912635640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fa90c0a6830 legacy=0x7fa90c1d6430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa911e34640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa90c0694b0 0x7fa90c141e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:58356/0 (socket says 172.21.15.1:58356) 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.618+0000 7fa911e34640 1 -- 172.21.15.1:0/1463890082 learned_addr learned my addr 172.21.15.1:0/1463890082 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa911633640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.1:6793/0 conn(0x7fa90c0a46a0 legacy=0x7fa90c1d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 client.?.objecter ms_handle_connect 0x7fa90c0a46a0 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 client.?.objecter ms_handle_connect 0x7fa90c0694b0 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa911e34640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.1:6792/0 conn(0x7fa90c0694b0 legacy=0x7fa90c141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961242580 0 0) 0x7fa90c13a330 con 0x7fa90c0a46a0 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): my global_id is 4508 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f0004eb0 con 0x7fa90c0a46a0 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362239194 0 0) 0x7fa90c068900 con 0x7fa90c0694b0 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 10 monclient(hunting): my global_id is 4506 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.620+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa90c13a330 con 0x7fa90c0694b0 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4115467289 0 0) 0x7fa90c13a330 con 0x7fa90c0694b0 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa90c068900 con 0x7fa90c0694b0 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa908001c90 con 0x7fa90c0694b0 2024-08-19T09:50:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): dump: 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2698806667 0 0) 0x7fa8f0004eb0 con 0x7fa90c0a46a0 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa90c13a330 con 0x7fa90c0a46a0 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa8fc001bb0 con 0x7fa90c0a46a0 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient(hunting): dump: 2024-08-19T09:50:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.637 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1374702435 0 0) 0x7fa90c068900 con 0x7fa90c0694b0 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.1:6793/0 conn(0x7fa90c0a46a0 legacy=0x7fa90c1d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6790/0 conn(0x7fa90c0a6830 legacy=0x7fa90c1d6430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 monclient: found mon.d 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c1425a0 con 0x7fa90c0694b0 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 5 monclient: authenticate success, global_id 4506 2024-08-19T09:50:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 10 monclient: _renew_subs 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c1ca440 con 0x7fa90c0694b0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 10 monclient: _renew_subs 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa90c1d7b50 con 0x7fa90c0694b0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa914098640 1 librados: init done 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa908004a90 con 0x7fa90c0694b0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa908004f80 con 0x7fa90c0694b0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa9027fc640 10 monclient: dump: 2024-08-19T09:50:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.640 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.621+0000 7fa903fff640 4 set_mon_vals no callback set 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.622+0000 7fa914098640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.622+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8f40065c0 con 0x7fa8f4002860 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.622+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa9080051a0 con 0x7fa90c0694b0 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.623+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fa90804d990 con 0x7fa90c0694b0 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.623+0000 7fa9027fc640 10 client.4506.objecter ms_dispatch 0x7fa90c000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.623+0000 7fa9027fc640 3 client.4506.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.623+0000 7fa9027fc640 3 client.4506.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.623+0000 7fa912635640 10 monclient: get_auth_request con 0x7fa8f0041480 auth_method 0 2024-08-19T09:50:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.623+0000 7fa9027fc640 20 client.4506.objecter dump_active .. 0 homeless 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 client.4506.objecter ms_handle_connect 0x7fa8f4002860 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 client.4506.objecter resend_mon_ops 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa911633640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6789/0 conn(0x7fa8f4002860 legacy=0x7fa8f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749960741 0 0) 0x7fa8f40065c0 con 0x7fa8f4002860 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient(hunting): my global_id is 4492 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa90c068900 con 0x7fa8f4002860 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 client.4506.objecter ms_handle_connect 0x7fa8f0041480 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 462248101 0 0) 0x7fa90c068900 con 0x7fa8f4002860 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8f40065c0 con 0x7fa8f4002860 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa8fc0015d0 con 0x7fa8f4002860 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 10 monclient: dump: 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 394126910 0 0) 0x7fa8f40065c0 con 0x7fa8f4002860 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.625+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8f400a160 con 0x7fa8f4002860 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.626+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa8fc0041c0 con 0x7fa8f4002860 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.626+0000 7fa9027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.626+0000 7fa9027fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.626+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6789/0 conn(0x7fa8f4002860 legacy=0x7fa8f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.626+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6789/0 conn(0x7fa8f4002860 legacy=0x7fa8f4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.632+0000 7fa914098640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.632+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8f4009e70 con 0x7fa8f4009130 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.634+0000 7fa911633640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6789/0 conn(0x7fa8f4009130 legacy=0x7fa8f40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.634+0000 7fa9027fc640 10 client.4506.objecter ms_handle_connect 0x7fa8f4009130 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.634+0000 7fa9027fc640 10 client.4506.objecter resend_mon_ops 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585337789 0 0) 0x7fa8f4009e70 con 0x7fa8f4009130 2024-08-19T09:50:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient(hunting): my global_id is 4501 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f40065c0 con 0x7fa8f4009130 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 55831553 0 0) 0x7fa8f40065c0 con 0x7fa8f4009130 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8f4009e70 con 0x7fa8f4009130 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa8fc005b00 con 0x7fa8f4009130 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient: dump: 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1305343418 0 0) 0x7fa8f4009e70 con 0x7fa8f4009130 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8f400c410 con 0x7fa8f4009130 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7652 (unknown 0 0 188760452) 0x7fa8fc006fb0 con 0x7fa8f4009130 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6789/0 conn(0x7fa8f4009130 legacy=0x7fa8f40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa9027fc640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6789/0 conn(0x7fa8f4009130 legacy=0x7fa8f40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.650 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 10, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 58895, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:50:52.651 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.652 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.653 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.654 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.655 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.656 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.657 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.661 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.664 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.666 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.667 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.668 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.670 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:50:52.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.671 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:52.671 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:52.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.635+0000 7fa914098640 10 librados: watch_flush enter 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 10 librados: watch_flush exit 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 20 client.4506.objecter shutdown clearing up homeless session... 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 10 client.4506.objecter successfully canceled tick 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa8f0041480 legacy=0x7fa8f0043940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 10 monclient: shutdown 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 >> v1:172.21.15.1:6792/0 conn(0x7fa90c0694b0 legacy=0x7fa90c141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa912635640 1 -- 172.21.15.1:0/1463890082 reap_dead start 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa912635640 1 -- 172.21.15.1:0/1463890082 reap_dead start 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 shutdown_connections 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 >> 172.21.15.1:0/1463890082 conn(0x7fa90c0a9020 msgr2=0x7fa90c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 shutdown_connections 2024-08-19T09:50:52.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 -- 172.21.15.1:0/1463890082 wait complete. 2024-08-19T09:50:52.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.636+0000 7fa914098640 1 librados: shutdown 2024-08-19T09:50:52.674 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:50:52.674 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon h 2024-08-19T09:50:52.674 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.773+0000 7f47b7415640 10 monclient: get_monmap_and_config 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.773+0000 7f47b7415640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.773+0000 7f47b7415640 10 monclient: monmap: 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:50:52.774596+0000 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:50:52.774596+0000 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 1 Processor -- start 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 1 -- start start 2024-08-19T09:50:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient: init 2024-08-19T09:50:52.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient: _add_conns ranks=[5,8,0,7,6,1,3,2,4] 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient(hunting): picked mon.noname-h con 0x7f47b00a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient(hunting): picked mon.noname-e con 0x7f47b00a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient(hunting): picked mon.noname-a con 0x7f47b013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b00a3b40 con 0x7f47b013f5c0 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b00a8130 con 0x7f47b00a67c0 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b006ede0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b7415640 10 monclient(hunting): authenticate will time out at 708.850934s 2024-08-19T09:50:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.774+0000 7f47b59b2640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f47b013f5c0 legacy=0x7f47b01419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.775+0000 7f47b49b0640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f47b00a6ba0 0x7f47b00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48472/0 (socket says 172.21.15.1:48472) 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.775+0000 7f47b49b0640 1 -- 172.21.15.1:0/1801273051 learned_addr learned my addr 172.21.15.1:0/1801273051 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b49b0640 1 -- 172.21.15.1:0/1801273051 >> v1:172.21.15.1:6793/0 conn(0x7f47b00a6ba0 legacy=0x7f47b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b51b1640 1 -- 172.21.15.1:0/1801273051 >> v1:172.21.15.82:6791/0 conn(0x7f47b00a67c0 legacy=0x7f47b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2918961144 0 0) 0x7f47b006ede0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): my global_id is 4517 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f479c003520 con 0x7f47b00a6ba0 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2762550020 0 0) 0x7f47b00a8130 con 0x7f47b00a67c0 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient(hunting): my global_id is 4604 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b006ede0 con 0x7f47b00a67c0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2805065520 0 0) 0x7f479c003520 con 0x7f47b00a6ba0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 >> v1:172.21.15.82:6791/0 conn(0x7f47b00a67c0 legacy=0x7f47b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 >> v1:172.21.15.1:6789/0 conn(0x7f47b013f5c0 legacy=0x7f47b01419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 monclient: found mon.noname-e 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b00a92a0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a0001ad0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b7415640 5 monclient: authenticate success, global_id 4517 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b7415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: dump: 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.785 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b7415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47a00048a0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 1 -- 172.21.15.1:0/1801273051 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a0004dd0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b7415640 10 monclient: get_monmap_and_config success 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47b7415640 4 set_mon_vals no callback set 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.777+0000 7f47a7fff640 10 monclient: dump: 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 10 monclient: shutdown 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 1 -- 172.21.15.1:0/1801273051 >> v1:172.21.15.1:6793/0 conn(0x7f47b00a6ba0 legacy=0x7f47b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 1 -- 172.21.15.1:0/1801273051 shutdown_connections 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 1 -- 172.21.15.1:0/1801273051 >> 172.21.15.1:0/1801273051 conn(0x7f47b00ac4b0 msgr2=0x7f47b00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 1 -- 172.21.15.1:0/1801273051 shutdown_connections 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 1 -- 172.21.15.1:0/1801273051 wait complete. 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 10 monclient: build_initial_monmap 2024-08-19T09:50:52.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.778+0000 7f47b7415640 10 monclient: monmap: 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:50:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 librados: starting msgr at 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 librados: starting objecter 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 Processor -- start 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 -- start start 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 librados: setting wanted keys 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 librados: calling monclient init 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient: init 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient: _reopen_session rank -1 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient: _add_conns ranks=[8,5,2,6,1,7,4,3,0] 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient(hunting): picked mon.noname-e con 0x7f47b00a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient(hunting): picked mon.noname-h con 0x7f47b00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:50:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient(hunting): picked mon.noname-b con 0x7f47b013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b013a730 con 0x7f47b013f5c0 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b01cb260 con 0x7f47b00a6ba0 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b00a9cc0 con 0x7f47b00a67c0 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient(hunting): _renew_subs 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b7415640 10 monclient(hunting): authenticate will time out at 708.855825s 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b51b1640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f47b00a67c0 0x7f47b01ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48480/0 (socket says 172.21.15.1:48480) 2024-08-19T09:50:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.779+0000 7f47b51b1640 1 -- 172.21.15.1:0/3330478432 learned_addr learned my addr 172.21.15.1:0/3330478432 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b51b1640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.1:6793/0 conn(0x7f47b00a67c0 legacy=0x7f47b01ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 client.?.objecter ms_handle_connect 0x7f47b00a67c0 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 client.?.objecter ms_handle_connect 0x7f47b013f5c0 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b59b2640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.1:6790/0 conn(0x7f47b013f5c0 legacy=0x7f47b01d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b49b0640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b00a6ba0 legacy=0x7f47b01cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 client.?.objecter ms_handle_connect 0x7f47b00a6ba0 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2003077733 0 0) 0x7f47b00a9cc0 con 0x7f47b00a67c0 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): my global_id is 4526 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4794004eb0 con 0x7f47b00a67c0 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99835328 0 0) 0x7f47b01cb260 con 0x7f47b00a6ba0 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): my global_id is 4613 2024-08-19T09:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b00a9cc0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 201309324 0 0) 0x7f47b013a730 con 0x7f47b013f5c0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): my global_id is 4511 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b01cb260 con 0x7f47b013f5c0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2190071393 0 0) 0x7f4794004eb0 con 0x7f47b00a67c0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47b013a730 con 0x7f47b00a67c0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47ac003610 con 0x7f47b00a67c0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:50:52.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.801 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2162941486 0 0) 0x7f47b00a9cc0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4794004eb0 con 0x7f47b00a6ba0 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a0004d90 con 0x7f47b00a6ba0 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4112989965 0 0) 0x7f47b01cb260 con 0x7f47b013f5c0 2024-08-19T09:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47b00a9cc0 con 0x7f47b013f5c0 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a8004790 con 0x7f47b013f5c0 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient(hunting): dump: 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1314622853 0 0) 0x7f47b013a730 con 0x7f47b00a67c0 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b00a6ba0 legacy=0x7f47b01cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.1:6790/0 conn(0x7f47b013f5c0 legacy=0x7f47b01d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 monclient: found mon.e 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b00a15c0 con 0x7f47b00a67c0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: _finish_auth 0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 5 monclient: authenticate success, global_id 4526 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 10 monclient: _renew_subs 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47b013a450 con 0x7f47b00a67c0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47ac003bd0 con 0x7f47b00a67c0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47ac004100 con 0x7f47b00a67c0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a5ffb640 10 monclient: dump: 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.807 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47a77fe640 4 set_mon_vals no callback set 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 10 monclient: _renew_subs 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f47b00a2120 con 0x7f47b00a67c0 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.782+0000 7f47b7415640 1 librados: init done 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f47ac0042a0 con 0x7f47b00a67c0 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47b7415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f47b00008d0 con 0x7f47b013fa80 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7f47ac04db30 con 0x7f47b00a67c0 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47a5ffb640 10 client.4526.objecter ms_dispatch 0x7f47b0000b90 osd_map(20..20 src has 1..20) 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47a5ffb640 3 client.4526.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.783+0000 7f47a5ffb640 3 client.4526.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.784+0000 7f47a5ffb640 20 client.4526.objecter dump_active .. 0 homeless 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.784+0000 7f47b49b0640 10 monclient: get_auth_request con 0x7f47940419e0 auth_method 0 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47b59b2640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b013fa80 legacy=0x7f47b013fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 client.4526.objecter ms_handle_connect 0x7f47b013fa80 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 client.4526.objecter resend_mon_ops 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2303560238 0 0) 0x7f47b00008d0 con 0x7f47b013fa80 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient(hunting): my global_id is 4622 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b013a730 con 0x7f47b013fa80 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 client.4526.objecter ms_handle_connect 0x7f47940419e0 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 231489659 0 0) 0x7f47b013a730 con 0x7f47b013fa80 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47b00008d0 con 0x7f47b013fa80 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a8004fa0 con 0x7f47b013fa80 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 10 monclient: dump: 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.811 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 760109238 0 0) 0x7f47b00008d0 con 0x7f47b013fa80 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.786+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47b0140570 con 0x7f47b013fa80 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.787+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f47a80072c0 con 0x7f47b013fa80 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.787+0000 7f47a5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.787+0000 7f47a5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.787+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b013fa80 legacy=0x7f47b013fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.787+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b013fa80 legacy=0x7f47b013fe60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.793+0000 7f47b7415640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:50:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.793+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f47b002e1d0 con 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47b59b2640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b0140720 legacy=0x7f47b01db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 client.4526.objecter ms_handle_connect 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 client.4526.objecter resend_mon_ops 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 581373564 0 0) 0x7f47b002e1d0 con 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient(hunting): my global_id is 4631 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b00008d0 con 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3384834097 0 0) 0x7f47b00008d0 con 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47b002e1d0 con 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a80059c0 con 0x7f47b0140720 2024-08-19T09:50:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.795+0000 7f47a5ffb640 10 monclient: dump: 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:50:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3255883902 0 0) 0x7f47b002e1d0 con 0x7f47b0140720 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f47b01decb0 con 0x7f47b0140720 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7650 (unknown 0 0 429870861) 0x7f47a8006ed0 con 0x7f47b0140720 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b0140720 legacy=0x7f47b01db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47a5ffb640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6791/0 conn(0x7f47b0140720 legacy=0x7f47b01db400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 10, 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 59062, 2024-08-19T09:50:52.816 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.817 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.818 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.819 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:50:52.820 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:50:52.821 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:50:52.822 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.824 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.825 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:50:52.826 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.829 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.830 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.831 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.833 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:50:52.834 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.835 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:50:52.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:50:52.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:50:52.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:50:52.837 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:50:52.837 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:50:52.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 10 librados: watch_flush enter 2024-08-19T09:50:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 10 librados: watch_flush exit 2024-08-19T09:50:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 20 client.4526.objecter shutdown clearing up homeless session... 2024-08-19T09:50:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 10 client.4526.objecter successfully canceled tick 2024-08-19T09:50:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.82:6812/2549104415 conn(0x7f47940419e0 legacy=0x7f4794043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 10 monclient: shutdown 2024-08-19T09:50:52.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:50:52.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 >> v1:172.21.15.1:6793/0 conn(0x7f47b00a67c0 legacy=0x7f47b01ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:50:52.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b59b2640 1 -- 172.21.15.1:0/3330478432 reap_dead start 2024-08-19T09:50:52.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b59b2640 1 -- 172.21.15.1:0/3330478432 reap_dead start 2024-08-19T09:50:52.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 shutdown_connections 2024-08-19T09:50:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.796+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 >> 172.21.15.1:0/3330478432 conn(0x7f47b00ac4b0 msgr2=0x7f47b00ab2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:50:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.797+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 shutdown_connections 2024-08-19T09:50:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.797+0000 7f47b7415640 1 -- 172.21.15.1:0/3330478432 wait complete. 2024-08-19T09:50:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:50:52.797+0000 7f47b7415640 1 librados: shutdown 2024-08-19T09:50:52.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:50:52.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr:Note: switching to '964adda23da9f7b057cb34fdc0b82aeb7ed65324'. 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr:state without impacting any branches by switching back to a branch. 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-19T09:51:18.381 INFO:tasks.workunit.client.0.smithi082.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: git switch -c 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr:Or undo this operation with: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: git switch - 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-08-19T09:51:18.382 INFO:tasks.workunit.client.0.smithi082.stderr:HEAD is now at 964adda23da Merge pull request #59290 from anthonyeleven/mountpoint 2024-08-19T09:51:18.383 DEBUG:teuthology.orchestra.run.smithi082:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-19T09:51:18.473 INFO:tasks.workunit.client.0.smithi082.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-19T09:51:18.475 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-19T09:51:18.475 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-19T09:51:19.032 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-19T09:51:19.106 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-19T09:51:19.146 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-19T09:51:19.148 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-19T09:51:19.148 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-19T09:51:19.186 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-19T09:51:19.190 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-08-19T09:51:19.190 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-19T09:51:19.205 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-08-19T09:51:19.207 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-08-19T09:51:19.207 DEBUG:teuthology.orchestra.run.smithi082:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ parallel=1 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' '' = --serial ']' 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ crimson=0 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' '' = --crimson ']' 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ color= 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -t 1 ']' 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-08-19T09:51:19.265 INFO:tasks.workunit.client.0.smithi082.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-19T09:51:19.266 INFO:tasks.workunit.client.0.smithi082.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-19T09:51:19.266 INFO:tasks.workunit.client.0.smithi082.stderr:+ declare -A pids 2024-08-19T09:51:19.267 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.267 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.267 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_aio 2024-08-19T09:51:19.267 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_aio' 2024-08-19T09:51:19.267 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_aio 2024-08-19T09:51:19.267 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.268 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_aio 2024-08-19T09:51:19.268 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25656 2024-08-19T09:51:19.268 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_aio on pid 25656' 2024-08-19T09:51:19.268 INFO:tasks.workunit.client.0.smithi082.stdout:test api_aio on pid 25656 2024-08-19T09:51:19.271 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25656 2024-08-19T09:51:19.271 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.271 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.271 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-08-19T09:51:19.271 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_aio_pp 2024-08-19T09:51:19.271 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_aio_pp' 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_aio_pp 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.272 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.273 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.274 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.274 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.274 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_aio_pp 2024-08-19T09:51:19.274 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25663 2024-08-19T09:51:19.274 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_aio_pp on pid 25663' 2024-08-19T09:51:19.274 INFO:tasks.workunit.client.0.smithi082.stdout:test api_aio_pp on pid 25663 2024-08-19T09:51:19.278 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25663 2024-08-19T09:51:19.278 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_io 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_io' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_io 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.279 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.280 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.281 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.282 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.282 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.282 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.282 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.282 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.283 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.284 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_io 2024-08-19T09:51:19.284 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25672 2024-08-19T09:51:19.284 INFO:tasks.workunit.client.0.smithi082.stdout:test api_io on pid 25672 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_io on pid 25672' 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25672 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_io_pp 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_io_pp' 2024-08-19T09:51:19.289 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_io_pp 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.290 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.291 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.292 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.293 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_io_pp 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25683 2024-08-19T09:51:19.294 INFO:tasks.workunit.client.0.smithi082.stdout:test api_io_pp on pid 25683 2024-08-19T09:51:19.300 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_io_pp on pid 25683' 2024-08-19T09:51:19.300 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25683 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_asio 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_asio' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_asio 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.301 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.302 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.303 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.304 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.305 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.306 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_asio 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25698 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_asio on pid 25698' 2024-08-19T09:51:19.307 INFO:tasks.workunit.client.0.smithi082.stdout:test api_asio on pid 25698 2024-08-19T09:51:19.316 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25698 2024-08-19T09:51:19.316 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.316 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_list 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_list' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_list 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.317 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.318 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.319 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.320 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.321 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.322 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.323 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.324 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_list 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25656/exe 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25715 2024-08-19T09:51:19.325 INFO:tasks.workunit.client.0.smithi082.stdout:test api_list on pid 25715 2024-08-19T09:51:19.334 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_list on pid 25715' 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25715 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_lock 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_lock' 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_lock 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.335 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.336 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.337 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.338 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.339 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.340 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.341 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25663/exe 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-08-19T09:51:19.342 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_aio.log 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_aio: /' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_lock 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.343 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.344 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25734 2024-08-19T09:51:19.345 INFO:tasks.workunit.client.0.smithi082.stdout:test api_lock on pid 25734 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_lock on pid 25734' 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25734 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.354 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_lock_pp 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_lock_pp' 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_lock_pp 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.355 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.356 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.357 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.358 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.359 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_aio_pp: /' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.362 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.363 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.364 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.364 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.364 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.364 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.364 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.365 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.365 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.365 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.365 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_lock_pp 2024-08-19T09:51:19.365 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25672/exe 2024-08-19T09:51:19.365 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25756 2024-08-19T09:51:19.366 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_lock_pp on pid 25756' 2024-08-19T09:51:19.366 INFO:tasks.workunit.client.0.smithi082.stdout:test api_lock_pp on pid 25756 2024-08-19T09:51:19.376 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25756 2024-08-19T09:51:19.376 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.376 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_misc 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_misc' 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.377 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.378 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.379 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_misc 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.380 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.381 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.382 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.382 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.382 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.382 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.385 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.385 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.385 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_io: /' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.386 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.387 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25683/exe 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_io.log 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.388 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.389 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_misc 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25774 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_misc on pid 25774' 2024-08-19T09:51:19.390 INFO:tasks.workunit.client.0.smithi082.stdout:test api_misc on pid 25774 2024-08-19T09:51:19.417 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25774 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.418 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.419 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.420 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_misc_pp 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.421 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_misc_pp' 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.422 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.423 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.424 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_io_pp: /' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.425 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.426 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25698/exe 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.427 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.428 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.429 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_asio.log 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.430 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.431 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_asio: /' 2024-08-19T09:51:19.432 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.433 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.434 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.435 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.436 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.437 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.438 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.439 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.440 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_misc_pp 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25734/exe 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.441 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.442 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_misc_pp 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25820 2024-08-19T09:51:19.443 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-08-19T09:51:19.444 INFO:tasks.workunit.client.0.smithi082.stdout:test api_misc_pp on pid 25820 2024-08-19T09:51:19.446 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_misc_pp on pid 25820' 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25820 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_tier_pp 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_tier_pp' 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_tier_pp 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.447 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25715/exe 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_tier_pp 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25830 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.448 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.449 INFO:tasks.workunit.client.0.smithi082.stdout:test api_tier_pp on pid 25830 2024-08-19T09:51:19.462 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_tier_pp on pid 25830' 2024-08-19T09:51:19.462 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25830 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.463 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.464 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_list: /' 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25756/exe 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.465 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_lock.log 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_lock_pp: /' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_lock: /' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_pool 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_list.log 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_pool' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.466 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.467 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.468 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.468 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.468 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.468 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.468 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.469 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_pool 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25774/exe 2024-08-19T09:51:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_pool 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25897 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_pool on pid 25897' 2024-08-19T09:51:19.474 INFO:tasks.workunit.client.0.smithi082.stdout:test api_pool on pid 25897 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25897 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_snapshots 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.480 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_snapshots' 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.481 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_snapshots 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.482 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.483 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.484 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [----------] Global test environment set-up. 2024-08-19T09:51:19.485 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [----------] 12 tests from AsioRados 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.344+0000 7fbe3c38c780 10 monclient: get_monmap_and_config 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.344+0000 7fbe3c38c780 10 monclient: build_initial_monmap 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.344+0000 7fbe3c38c780 10 monclient: monmap: 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: epoch 0 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: last_changed 0.000000 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: created 0.000000 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: min_mon_release 0 (unknown) 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: election_strategy: 1 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:51:19.486 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 1 Processor -- start 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 1 -- start start 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient: init 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient: _reopen_session rank -1 2024-08-19T09:51:19.487 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient: _add_conns ranks=[7,5,1,4,0,3,6,8,2] 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient(hunting): picked mon.noname-i con 0x559d53e12240 addr v1:172.21.15.82:6792/0 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient(hunting): picked mon.noname-h con 0x559d53e12620 addr v1:172.21.15.82:6791/0 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient(hunting): picked mon.noname-f con 0x559d53e12a30 addr v1:172.21.15.82:6789/0 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x559d53e32170 con 0x559d53e12a30 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x559d53e30170 con 0x559d53e12620 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x559d53e31dc0 con 0x559d53e12240 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient(hunting): _renew_subs 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.345+0000 7fbe3c38c780 10 monclient(hunting): authenticate will time out at 730.373960s 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.346+0000 7fbe39e7b640 1 -- >> v1:172.21.15.82:6792/0 conn(0x559d53e12240 legacy=0x559d53f50df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.346+0000 7fbe3967a640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x559d53e12620 0x559d53f55c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.82:42138/0 (socket says 172.21.15.82:42138) 2024-08-19T09:51:19.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.346+0000 7fbe3967a640 1 -- 172.21.15.82:0/4128224154 learned_addr learned my addr 172.21.15.82:0/4128224154 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:51:19.489 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_snapshots 2024-08-19T09:51:19.489 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-08-19T09:51:19.489 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25931 2024-08-19T09:51:19.489 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_misc.log 2024-08-19T09:51:19.489 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_snapshots on pid 25931' 2024-08-19T09:51:19.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.348+0000 7fbe3967a640 1 -- 172.21.15.82:0/4128224154 >> v1:172.21.15.82:6791/0 conn(0x559d53e12620 legacy=0x559d53f55c50 unknown :-1test api_snapshots on pid 25931 2024-08-19T09:51:19.491 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25931 2024-08-19T09:51:19.491 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.491 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.491 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-08-19T09:51:19.491 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stdout: s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.349+0000 7fbe3a67c640 1 -- 172.21.15.82:0/4128224154 >> v1:172.21.15.82:6789/0 conn(0x559d53e12a30 legacy=0x559d53f59380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:51:19.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.349+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:51:19.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.351+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65086521 0 0) 0x559d53e30170 con 0x559d53e12620 2024-08-19T09:51:19.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.351+0000 7fbe2bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:51:19.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.351+0000 7fbe2bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:51:19.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.351+0000 7fbe2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:51:19.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.351+0000 7fbe2bfff640 10 monclient(hunting): my global_id is 4658 2024-08-19T09:51:19.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.351+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe24001e10 con 0x559d53e12620 2024-08-19T09:51:19.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3349063793 0 0) 0x7fbe24001e10 con 0x559d53e12620 2024-08-19T09:51:19.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 >> v1:172.21.15.82:6792/0 conn(0x559d53e12240 legacy=0x559d53f50df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:51:19.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 >> v1:172.21.15.82:6789/0 conn(0x559d53e12a30 legacy=0x559d53f59380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:51:19.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:51:19.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 monclient: found mon.noname-h 2024-08-19T09:51:19.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d53e2ad60 con 0x559d53e12620 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: _finish_auth 0 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbe34002f20 con 0x559d53e12620 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: dump: 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: epoch 1 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:19.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:19.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T09:51:19.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: election_strategy: 1 2024-08-19T09:51:19.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:51:19.498 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.498 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_snapshots_pp 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_snapshots_pp' 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.499 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_misc: /' 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.500 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:51:19.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe3c38c780 5 monclient: authenticate success, global_id 4658 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe3c38c780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbe34001250 con 0x559d53e12620 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 1 -- 172.21.15.82:0/4128224154 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbe34001780 con 0x559d53e12620 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe2bfff640 10 monclient: dump: 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: epoch 1 2024-08-19T09:51:19.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: election_strategy: 1 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:51:19.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe3c38c780 10 monclient: get_monmap_and_config success 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.352+0000 7fbe3c38c780 4 set_mon_vals no callback set 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.353+0000 7fbe3c38c780 10 monclient: shutdown 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.353+0000 7fbe3c38c780 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.353+0000 7fbe3c38c780 1 -- 172.21.15.82:0/4128224154 >> v1:172.21.15.82:6791/0 conn(0x559d53e12620 legacy=0x559d53f55c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 1 -- 172.21.15.82:0/4128224154 shutdown_connections 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 1 -- 172.21.15.82:0/4128224154 >> 172.21.15.82:0/4128224154 conn(0x559d53e330e0 msgr2=0x559d53e2b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 1 -- 172.21.15.82:0/4128224154 shutdown_connections 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 1 -- 172.21.15.82:0/4128224154 wait complete. 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 10 monclient: build_initial_monmap 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 10 monclient: monmap: 2024-08-19T09:51:19.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: epoch 0 2024-08-19T09:51:19.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:51:19.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: last_changed 0.000000 2024-08-19T09:51:19.507 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.507 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.507 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.508 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.509 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: created 0.000000 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: min_mon_release 0 (unknown) 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: election_strategy: 1 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:51:19.510 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 1 librados: starting msgr at 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.354+0000 7fbe3c38c780 1 librados: starting objecter 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 Processor -- start 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 -- start start 2024-08-19T09:51:19.511 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 librados: setting wanted keys 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 librados: calling monclient init 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient: init 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient: _reopen_session rank -1 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient: _add_conns ranks=[1,0,3,8,5,7,6,4,2] 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient(hunting): picked mon.noname-f con 0x559d53e12240 addr v1:172.21.15.82:6789/0 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient(hunting): picked mon.noname-a con 0x559d53e12620 addr v1:172.21.15.1:6789/0 2024-08-19T09:51:19.512 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient(hunting): picked mon.noname-g con 0x559d53e12a30 addr v1:172.21.15.82:6790/0 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x559d53e2eeb0 con 0x559d53e12620 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x559d53f59140 con 0x559d53e12240 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x559d53f4cd80 con 0x559d53e12a30 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient(hunting): _renew_subs 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3c38c780 10 monclient(hunting): authenticate will time out at 730.383827s 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3a67c640 1 -- >> v1:172.21.15.82:6790/0 conn(0x559d53e12a30 legacy=0x559d53f608d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.355+0000 7fbe3967a640 1 -- >> v1:172.21.15.1:6789/0 conn(0x559d53e12620 legacy=0x559d53f58a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.356+0000 7fbe39e7b640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x559d53e12240 0x559d53f58320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:51764/0 (socket says 172.21.15.82:51764) 2024-08-19T09:51:19.513 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.356+0000 7fbe39e7b640 1 -- 172.21.15.82:0/1475860754 learned_addr learned my addr 172.21.15.82:0/1475860754 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.519 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_snapshots_pp 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.520 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.521 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.521 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.521 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.521 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.521 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.521 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.522 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.522 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.522 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.522 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.522 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.523 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.524 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.525 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_snapshots_pp 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25975 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.526 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.359+0000 7fbe39e7b640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6789/0 conn(0x559d53e12240 legacy=0x559d53f58320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delaytest api_snapshots_pp on pid 25975 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_snapshots_pp on pid 25975' 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=25975 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.537 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25830/exe 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25820/exe 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.539 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_stat 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_stat' 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.540 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.541 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.542 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.543 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.544 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_stat 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.545 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.546 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_stat 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26007 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_stat on pid 26007' 2024-08-19T09:51:19.547 INFO:tasks.workunit.client.0.smithi082.stdout:test api_stat on pid 26007 2024-08-19T09:51:19.551 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26007 2024-08-19T09:51:19.551 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.551 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.551 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-08-19T09:51:19.551 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_stat_pp 2024-08-19T09:51:19.551 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_stat_pp' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_stat_pp 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.552 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.553 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_stat_pp 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26022 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_stat_pp on pid 26022' 2024-08-19T09:51:19.554 INFO:tasks.workunit.client.0.smithi082.stdout:test api_stat_pp on pid 26022 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26022 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_misc_pp: /' 2024-08-19T09:51:19.564 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_tier_pp: /' 2024-08-19T09:51:19.565 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_watch_notify 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_watch_notify' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.566 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.567 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.568 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_watch_notify 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.569 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.570 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.571 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.572 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_watch_notify 2024-08-19T09:51:19.573 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26051 2024-08-19T09:51:19.573 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_watch_notify on pid 26051' 2024-08-19T09:51:19.573 INFO:tasks.workunit.client.0.smithi082.stdout:test api_watch_notify on pid 26051 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26051 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.588 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.589 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.590 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.590 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.590 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.590 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.590 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.590 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.591 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.592 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.593 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25897/exe 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.594 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_watch_notify_pp 2024-08-19T09:51:19.595 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_watch_notify_pp' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.596 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.597 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.598 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_watch_notify_pp 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.599 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.600 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_pool.log 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.601 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.602 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.603 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_watch_notify_pp 2024-08-19T09:51:19.603 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26086 2024-08-19T09:51:19.603 INFO:tasks.workunit.client.0.smithi082.stdout:test api_watch_notify_pp on pid 26086 2024-08-19T09:51:19.619 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_watch_notify_pp on pid 26086' 2024-08-19T09:51:19.619 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26086 2024-08-19T09:51:19.619 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.619 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.619 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-08-19T09:51:19.619 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_cmd 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.620 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_cmd' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_pool: /' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25931/exe 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.621 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.622 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.623 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.624 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.624 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.624 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.624 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.624 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.624 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-08-19T09:51:19.625 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.625 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.625 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.625 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.625 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.625 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.626 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/25975/exe 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.627 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.630 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.631 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.632 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.633 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.634 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.634 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26007/exe 2024-08-19T09:51:19.634 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_cmd 2024-08-19T09:51:19.634 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.634 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.635 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.635 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.635 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.635 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.635 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.635 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.636 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.636 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.636 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_cmd 2024-08-19T09:51:19.636 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.636 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26132 2024-08-19T09:51:19.636 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_cmd on pid 26132' 2024-08-19T09:51:19.637 INFO:tasks.workunit.client.0.smithi082.stdout:test api_cmd on pid 26132 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26132 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_cmd_pp 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_cmd_pp' 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.642 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.643 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.643 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.643 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.643 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.643 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.644 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.644 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.644 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.644 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_snapshots: /' 2024-08-19T09:51:19.644 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.644 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_cmd_pp 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.645 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-08-19T09:51:19.646 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_cmd_pp 2024-08-19T09:51:19.647 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26151 2024-08-19T09:51:19.647 INFO:tasks.workunit.client.0.smithi082.stdout:test api_cmd_pp on pid 26151 2024-08-19T09:51:19.675 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_cmd_pp on pid 26151' 2024-08-19T09:51:19.675 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26151 2024-08-19T09:51:19.675 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.676 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.676 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.676 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.676 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.676 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.677 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.677 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_service 2024-08-19T09:51:19.677 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.677 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.677 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.677 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.678 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.678 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.678 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_service' 2024-08-19T09:51:19.678 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.678 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.678 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.679 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.679 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.679 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.679 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-08-19T09:51:19.679 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.679 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.680 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.681 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.681 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-08-19T09:51:19.681 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_stat.log 2024-08-19T09:51:19.681 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.681 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.681 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.682 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.682 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.682 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.682 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.682 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.682 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_stat: /' 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.683 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.684 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.685 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.685 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.685 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.685 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.685 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.686 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.686 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.686 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.686 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.686 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.687 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.688 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.689 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.690 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.690 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.690 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.690 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.690 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.690 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.691 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.691 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.691 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.691 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.691 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.691 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26022/exe 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.692 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.693 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.694 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26086/exe 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.697 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.698 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.699 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.700 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.701 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.702 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_service 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26051/exe 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.703 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_stat_pp: /' 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-08-19T09:51:19.704 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_service 2024-08-19T09:51:19.705 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26210 2024-08-19T09:51:19.705 INFO:tasks.workunit.client.0.smithi082.stdout:test api_service on pid 26210 2024-08-19T09:51:19.711 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_service on pid 26210' 2024-08-19T09:51:19.711 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26210 2024-08-19T09:51:19.711 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.711 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.711 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-08-19T09:51:19.711 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_service_pp 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_service_pp' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_watch_notify: /' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.712 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_service_pp 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.713 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.714 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_service_pp 2024-08-19T09:51:19.715 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26253 2024-08-19T09:51:19.716 INFO:tasks.workunit.client.0.smithi082.stdout:test api_service_pp on pid 26253 2024-08-19T09:51:19.726 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_service_pp on pid 26253' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26253 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.727 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_c_write_operations 2024-08-19T09:51:19.728 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.729 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.730 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_c_write_operations' 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.731 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.732 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_c_write_operations 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.733 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.734 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_c_write_operations 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.735 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26294 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_c_write_operations on pid 26294' 2024-08-19T09:51:19.736 INFO:tasks.workunit.client.0.smithi082.stdout:test api_c_write_operations on pid 26294 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26294 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.751 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.752 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26132/exe 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.753 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.754 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26151/exe 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s api_c_read_operations 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.755 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.756 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' api_c_read_operations' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.757 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.758 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo api_c_read_operations 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.759 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.760 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_cmd: /' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.761 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-08-19T09:51:19.762 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_cmd.log 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.763 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=api_c_read_operations 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26368 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test api_c_read_operations on pid 26368' 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-08-19T09:51:19.764 INFO:tasks.workunit.client.0.smithi082.stdout:test api_c_read_operations on pid 26368 2024-08-19T09:51:19.769 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26368 2024-08-19T09:51:19.769 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.769 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.770 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s list_parallel 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.771 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' list_parallel' 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo list_parallel 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.772 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.773 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=list_parallel 2024-08-19T09:51:19.774 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.774 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.774 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26386 2024-08-19T09:51:19.774 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test list_parallel on pid 26386' 2024-08-19T09:51:19.774 INFO:tasks.workunit.client.0.smithi082.stdout:test list_parallel on pid 26386 2024-08-19T09:51:19.779 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26386 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-08-19T09:51:19.780 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s open_pools_parallel 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' open_pools_parallel' 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26210/exe 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.781 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo open_pools_parallel 2024-08-19T09:51:19.782 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.783 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=open_pools_parallel 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_service: /' 2024-08-19T09:51:19.784 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26408 2024-08-19T09:51:19.785 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test open_pools_parallel on pid 26408' 2024-08-19T09:51:19.785 INFO:tasks.workunit.client.0.smithi082.stdout:test open_pools_parallel on pid 26408 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26408 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.800 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.801 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_service.log 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.802 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.803 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.804 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.805 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.806 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.807 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.808 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26253/exe 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s delete_pools_parallel 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' delete_pools_parallel' 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.809 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.810 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo delete_pools_parallel 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.811 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.812 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_service_pp: /' 2024-08-19T09:51:19.813 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=delete_pools_parallel 2024-08-19T09:51:19.813 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26459 2024-08-19T09:51:19.813 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test delete_pools_parallel on pid 26459' 2024-08-19T09:51:19.813 INFO:tasks.workunit.client.0.smithi082.stdout:test delete_pools_parallel on pid 26459 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26459 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s cls 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' cls' 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.820 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo cls 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.821 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.822 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.823 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=cls 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26471 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.824 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test cls on pid 26471' 2024-08-19T09:51:19.825 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.825 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.825 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.825 INFO:tasks.workunit.client.0.smithi082.stdout:test cls on pid 26471 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26471 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.843 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s cmd 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.844 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' cmd' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.845 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.846 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo cmd 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.847 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.848 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.849 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.850 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26386/exe 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26294/exe 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.851 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.852 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.853 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.854 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.855 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ list_parallel: /' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.856 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.857 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.858 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.859 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.860 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_list_parallel.log 2024-08-19T09:51:19.860 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=cmd 2024-08-19T09:51:19.860 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26524 2024-08-19T09:51:19.860 INFO:tasks.workunit.client.0.smithi082.stdout:test cmd on pid 26524 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test cmd on pid 26524' 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26524 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.879 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.880 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s handler_error 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' handler_error' 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.881 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo handler_error 2024-08-19T09:51:19.882 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.883 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.884 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.885 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.886 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.887 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.888 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26471/exe 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.889 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.890 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26368/exe 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.891 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ cls: /' 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_cls 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.892 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.893 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_cls.log 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.894 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.895 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-08-19T09:51:19.896 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-08-19T09:51:19.897 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=handler_error 2024-08-19T09:51:19.897 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26589 2024-08-19T09:51:19.897 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test handler_error on pid 26589' 2024-08-19T09:51:19.897 INFO:tasks.workunit.client.0.smithi082.stdout:test handler_error on pid 26589 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26589 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.902 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26459/exe 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s io 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' io' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo io 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26408/exe 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.903 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.904 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=io 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26610 2024-08-19T09:51:19.905 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test io on pid 26610' 2024-08-19T09:51:19.906 INFO:tasks.workunit.client.0.smithi082.stdout:test io on pid 26610 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26610 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s ec_io 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' ec_io' 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo ec_io 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.911 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.912 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.913 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-08-19T09:51:19.914 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=ec_io 2024-08-19T09:51:19.915 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26633 2024-08-19T09:51:19.916 INFO:tasks.workunit.client.0.smithi082.stdout:test ec_io on pid 26633 2024-08-19T09:51:19.921 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test ec_io on pid 26633' 2024-08-19T09:51:19.921 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26633 2024-08-19T09:51:19.921 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.922 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.923 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s list 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' list' 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo list 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.924 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.925 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=list 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26660 2024-08-19T09:51:19.926 INFO:tasks.workunit.client.0.smithi082.stdout:test list on pid 26660 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test list on pid 26660' 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26660 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.937 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26524/exe 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.938 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:19.939 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s ec_list 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.940 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.941 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' ec_list' 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo ec_list 2024-08-19T09:51:19.942 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_cmd 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_cmd.log 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ cmd: /' 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.943 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.944 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.945 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=ec_list 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26727 2024-08-19T09:51:19.946 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test ec_list on pid 26727' 2024-08-19T09:51:19.947 INFO:tasks.workunit.client.0.smithi082.stdout:test ec_list on pid 26727 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26727 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s misc 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' misc' 2024-08-19T09:51:19.949 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo misc 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=misc 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26751 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test misc on pid 26751' 2024-08-19T09:51:19.950 INFO:tasks.workunit.client.0.smithi082.stdout:test misc on pid 26751 2024-08-19T09:51:19.978 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26751 2024-08-19T09:51:19.978 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:19.978 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:19.978 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s pool 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' pool' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.979 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.980 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:19.981 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.982 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.983 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.984 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.985 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.986 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:19.987 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.988 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.989 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:19.991 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:19.992 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:19.993 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:19.994 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.995 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:19.996 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.997 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo pool 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:19.998 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:19.999 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.000 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.001 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.002 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.003 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26633/exe 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26589/exe 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26610/exe 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.004 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=pool 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:20.005 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.006 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:20.006 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.006 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26889 2024-08-19T09:51:20.006 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.006 INFO:tasks.workunit.client.0.smithi082.stdout:test pool on pid 26889 2024-08-19T09:51:20.018 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:20.018 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test pool on pid 26889' 2024-08-19T09:51:20.018 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26889 2024-08-19T09:51:20.018 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.019 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ ec_io: /' 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.020 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s read_operations 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.021 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' read_operations' 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.022 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_ec_io.log 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_ec_io 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ handler_error: /' 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_handler_error 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.023 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_handler_error.log 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo read_operations 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_io 2024-08-19T09:51:20.024 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_io.log 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ io: /' 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26751/exe 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26660/exe 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.025 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.026 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:20.027 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_misc.log 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.028 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ misc: /' 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_misc 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_list 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ list: /' 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_list.log 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.029 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.030 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:20.031 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.031 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:20.031 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.031 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=read_operations 2024-08-19T09:51:20.031 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26949 2024-08-19T09:51:20.031 INFO:tasks.workunit.client.0.smithi082.stdout:test read_operations on pid 26949 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test read_operations on pid 26949' 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26949 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.042 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s snapshots 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' snapshots' 2024-08-19T09:51:20.043 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:20.044 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.045 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.046 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.047 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.048 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26727/exe 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo snapshots 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.049 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:20.050 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:20.051 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=snapshots 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26979 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test snapshots on pid 26979' 2024-08-19T09:51:20.052 INFO:tasks.workunit.client.0.smithi082.stdout:test snapshots on pid 26979 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=26979 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s watch_notify 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' watch_notify' 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.061 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.062 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ ec_list: /' 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_ec_list.log 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_ec_list 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26889/exe 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo watch_notify 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:20.063 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.064 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:20.066 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_pool.log 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_pool 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.067 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ pool: /' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:20.068 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.069 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=watch_notify 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=27030 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test watch_notify on pid 27030' 2024-08-19T09:51:20.070 INFO:tasks.workunit.client.0.smithi082.stdout:test watch_notify on pid 27030 2024-08-19T09:51:20.077 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=27030 2024-08-19T09:51:20.077 INFO:tasks.workunit.client.0.smithi082.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T09:51:20.077 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:20.077 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.077 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.077 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ printf %25s write_operations 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.078 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:+ r=' write_operations' 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++ echo write_operations 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++ awk '{print $1}' 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.079 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.080 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.081 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:20.082 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:+ ff=write_operations 2024-08-19T09:51:20.083 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=27062 2024-08-19T09:51:20.084 INFO:tasks.workunit.client.0.smithi082.stdout:test write_operations on pid 27062 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ echo 'test write_operations on pid 27062' 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ pids[$f]=27062 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ ret=0 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25830 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25830 2024-08-19T09:51:20.108 INFO:tasks.workunit.client.0.smithi082.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26949/exe 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.109 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:+ . /etc/bashrc 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -z '' ']' 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ BASHRCSOURCED=Y 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ shopt -q login_shell 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:+++ umask 2024-08-19T09:51:20.110 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ SHELL=/bin/bash 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.111 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T09:51:20.112 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ read_operations: /' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_read_operations.log 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_read_operations 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:20.113 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.114 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T09:51:20.115 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:+++ return 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export S_COLORS=auto 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:+++ S_COLORS=auto 2024-08-19T09:51:20.116 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.117 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:20.118 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.119 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.120 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.121 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/26979/exe 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.122 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T09:51:20.123 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup 2024-08-19T09:51:20.124 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/locale 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_snapshots 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_snapshots.log 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ snapshots: /' 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.125 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.126 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:++++ LANG=en_US.UTF-8 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ LANG=en_US.UTF-8 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.127 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:+++ unset LANG_backup config 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n '' ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' dumb = linux ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/less.sh 2024-08-19T09:51:20.128 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ '[' '' ']' 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/27030/exe 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:+++++ readlink /proc/27062/exe 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.129 INFO:tasks.workunit.client.0.smithi082.stderr:++++ basename /usr/bin/bash 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_declare='declare -f' 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ which_opt=-f 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export which_declare 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:+++ export -f which 2024-08-19T09:51:20.130 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset i 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:++ unset -f pathmunge 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ export PATH 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ unset rc 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_write_operations.log 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_write_operations 2024-08-19T09:51:20.131 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ write_operations: /' 2024-08-19T09:51:20.132 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph_test_neorados_watch_notify 2024-08-19T09:51:20.132 INFO:tasks.workunit.client.0.smithi082.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-08-19T09:51:20.132 INFO:tasks.workunit.client.0.smithi082.stderr:+ sed 's/^/ watch_notify: /' 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: queue 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.359+0000 7fbe2affd640 10 client.?.objecter ms_handle_connect 0x559d53e12240 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.359+0000 7fbe2affd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.359+0000 7fbe39e7b640 1 --1- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6789/0 conn(0x559d53e12240 0x559d53f58320 :-1 s=OPENED pgs=140 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x559d53f59140 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.556+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6790/0 conn(0x559d53e12a30 legacy=0x559d53f608d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.557+0000 7fbe3967a640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.1:6789/0 conn(0x559d53e12620 legacy=0x559d53f58a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:51:20.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.957+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6790/0 conn(0x559d53e12a30 legacy=0x559d53f608d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:19.959+0000 7fbe3967a640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.1:6789/0 conn(0x559d53e12620 legacy=0x559d53f58a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765394229 0 0) 0x559d53f59140 con 0x559d53e12240 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): _negotiate 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): my global_id is 4546 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe24001e10 con 0x559d53e12240 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3958708635 0 0) 0x7fbe24001e10 con 0x559d53e12240 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x559d53f59140 con 0x559d53e12240 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbe2c003680 con 0x559d53e12240 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:51:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): dump: 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: epoch 1 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: election_strategy: 1 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:51:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.187+0000 7fbe2affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1335171198 0 0) 0x559d53f59140 con 0x559d53e12240 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6790/0 conn(0x559d53e12a30 legacy=0x559d53f608d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.1:6789/0 conn(0x559d53e12620 legacy=0x559d53f58a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: _finish_hunting 0 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 monclient: found mon.f 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:51:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d53f4b700 con 0x559d53e12240 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: _finish_auth 0 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: _check_auth_tickets 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 5 monclient: authenticate success, global_id 4546 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 10 monclient: _renew_subs 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x559d53f4a590 con 0x559d53e12240 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 10 monclient: _renew_subs 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x559d53e2e5d0 con 0x559d53e12240 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 1 librados: init done 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe3c38c780 10 librados: wait_for_osdmap waiting 2024-08-19T09:51:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbe2c001320 con 0x559d53e12240 2024-08-19T09:51:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:51:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbe2c003f70 con 0x559d53e12240 2024-08-19T09:51:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:51:21.418 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:51:21.418 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2affd640 10 monclient: dump: 2024-08-19T09:51:21.418 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: epoch 1 2024-08-19T09:51:21.418 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: election_strategy: 1 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:51:21.419 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.188+0000 7fbe2b7fe640 4 set_mon_vals no callback set 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbe2c0103d0 con 0x559d53e12240 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 2088056616 0 0) 0x7fbe2c04c5c0 con 0x559d53e12240 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_map(20..20 src has 1..20) 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map decoding full epoch 20 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3967a640 10 monclient: get_auth_request con 0x7fbe2403f860 auth_method 0 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 20 client.4546.objecter dump_active .. 0 homeless 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 10 librados: wait_for_osdmap done waiting 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbe2c003190 con 0x559d53e12240 2024-08-19T09:51:21.420 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7fbe2c015820 con 0x559d53e12240 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2affd640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe2bfff640 10 client.4546.objecter _get_latest_version latest 20, have it 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 10 client.4546.objecter create_pool name=ceph_test_rados_api_asio 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 10 client.4546.objecter pool_op_submit 1 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.189+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x559d53dbab10 con 0x559d53e12240 2024-08-19T09:51:21.421 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:20.191+0000 7fbe2affd640 10 client.4546.objecter ms_handle_connect 0x7fbe2403f860 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 10 ==== pool_op_reply(tid 1 (0) Success v22) ==== 43+0+0 (unknown 2947578034 0 0) 0x7fbe2c0160e0 con 0x559d53e12240 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 client.4546.objecter ms_dispatch 0x559d53dad570 pool_op_reply(tid 1 (0) Success v22) 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 client.4546.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v22) 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 client.4546.objecter have request 1 at 0x559d53f4b220 Op: create 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 20 client.4546.objecter waiting for client to reach epoch 22 before calling back 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 monclient: _renew_subs 2024-08-19T09:51:21.444 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=21}) -- 0x559d53f59140 con 0x559d53e12240 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.417+0000 7fbe2affd640 10 client.4546.objecter done 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== mon.1 v1:172.21.15.82:6789/0 11 ==== osd_map(21..22 src has 1..22) ==== 21486+0+0 (unknown 3457359203 0 0) 0x7fbe2c01e540 con 0x559d53e12240 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe2affd640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_map(21..22 src has 1..22) 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map got epochs [21,22] > 20 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map decoding incremental epoch 21 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map decoding incremental epoch 22 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe2affd640 20 client.4546.objecter dump_active .. 0 homeless 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 10 librados: set snap read head -> 1 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 10 librados: writefull oid=exist nspace= 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53f63000 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 22 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:21.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:21.446 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:21.446 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:21.446 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [writefull 0~5 in=5b] tid 2 osd.3 2024-08-19T09:51:21.446 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:21.446 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 2 2024-08-19T09:51:21.446 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 15 client.4546.objecter _send_op 2 to 33.1 on osd.3 2024-08-19T09:51:22.075 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:2 33.1 33:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_red api_list: [==========] Running 11 tests from 3 test suites. 2024-08-19T09:51:22.075 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] Global test environment set-up. 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] 7 tests from LibRadosList 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.ListObjects (309 ms) 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (36 ms) 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset foo1,foo2,foo3 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo1 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo2 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo3 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset foo1,foo4,foo5 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo4 2024-08-19T09:51:22.076 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo5 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo1 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset foo6,foo7 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo7 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo6 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns1:foo4 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns1:foo5 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns2:foo7 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns2:foo6 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns1:foo1 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: :foo1 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: :foo2 2024-08-19T09:51:22.077 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: :foo3 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (115 ms) 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 1 0 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 10 0 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 13 0 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 7 0 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 14 0 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 0 0 2024-08-19T09:51:22.078 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 15 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 11 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 5 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 8 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 6 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 3 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 4 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 12 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 9 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 2 0 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (87 ms) 2024-08-19T09:51:22.082 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: x cursor=MIN 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-08-19T09:51:22.083 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: FIRST> seek to MIN oid=1 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-08-19T09:51:22.084 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:566253c9:::13:head 2024-08-19T09:51:22.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-08-19T09:51:22.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-08-19T09:51:22.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-08-19T09:51:22.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-08-19T09:51:22.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:863748b0:::15:head 2024-08-19T09:51:22.145 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-08-19T09:51:22.146 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:e960b815:::9:head 2024-08-19T09:51:22.147 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:566253c9:::13:head 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=13 expected=13 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:863748b0:::15:head 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=15 expected=15 2024-08-19T09:51:22.148 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=0 expected=0 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=14 expected=14 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=7 expected=7 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-08-19T09:51:22.149 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=10 expected=10 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=2 expected=2 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:e960b815:::9:head 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=9 expected=9 2024-08-19T09:51:22.150 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-08-19T09:51:22.488 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (267 ms) 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (13 ms) 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-08-19T09:51:22.489 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (8 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (27 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (13 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (24 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (12 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (11 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (11 ms) 2024-08-19T09:51:22.490 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (11 ms) 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (10 ms) 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (18 ms) 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (24 ms) 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (23 ms) 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (7 ms) 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-08-19T09:51:22.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (486 ms total) 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2911 ms total) 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: Running main() from gmock_main.cc 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [----------] Global test environment set-up. 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2806 ms) 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [----------] 1 test from neocls_handler_error (2806 ms total) 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: 2024-08-19T09:51:22.492 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [----------] Global test environment tear-down 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2806 ms total) 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: handler_error: [ PASSED ] 1 test. 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-08-19T09:51:22.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (65 ms) 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (64 ms) 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2853 ms) 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2982 ms total) 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2982 ms total) 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: cls: Running main() from gmock_main.cc 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-08-19T09:51:22.494 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [----------] Global test environment set-up. 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [----------] 1 test from NeoRadosCls 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [ OK ] NeoRadosCls.DNE (2918 ms) 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [----------] 1 test from NeoRadosCls (2918 ms total) 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [----------] Global test environment tear-down 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [==========] 1 test from 1 test suite ran. (2918 ms total) 2024-08-19T09:51:22.495 INFO:tasks.workunit.client.0.smithi082.stdout: cls: [ PASSED ] 1 test. 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout:irected+supports_pool_eio e22) -- 0x559d53f674d0 con 0x559d53f639c0 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.442+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.455+0000 7fbe3a67c640 10 monclient: get_auth_request con 0x559d53f639c0 auth_method 0 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:21.461+0000 7fbe2affd640 10 client.4546.objecter ms_handle_connect 0x559d53f639c0 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.355+0000 7fbe29ffb640 10 monclient: tick 2024-08-19T09:51:22.608 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.355+0000 7fbe29ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.355+0000 7fbe29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.355+0000 7fbe29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 1 ==== osd_map(23..23 src has 1..23) ==== 6265+0+0 (unknown 3365137266 0 0) 0x7fbe30003820 con 0x559d53f639c0 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_map(23..23 src has 1..23) 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map got epochs [23,23] > 22 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 3 client.4546.objecter handle_osd_map decoding incremental epoch 23 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 10 client.4546.objecter checking op 2 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 20 client.4546.objecter dump_active .. 0 homeless 2024-08-19T09:51:22.609 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.589+0000 7fbe2affd640 20 client.4546.objecter 2 33.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 2 ==== osd_op_reply(2 exist [writefull 0~5] v23'1 uv1 ondisk = 0) ==== 149+0+0 (unknown 511897997 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(2 exist [writefull 0~5] v23'1 uv1 ondisk = 0) 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 2 ondisk uv 1 in 33.1 attempt 0 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 2 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 2 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 2 2024-08-19T09:51:22.610 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.611 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-08-19T09:51:22.611 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 10 librados: Objecter returned from writefull r=0 2024-08-19T09:51:22.611 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53f61ff0 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.606+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [read 0~256] tid 3 osd.3 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.627 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 3 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 15 client.4546.objecter _send_op 3 to 33.1 on osd.3 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:3 33.1 33:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53f624a0 con 0x559d53f639c0 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53f62880 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base noexist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target noexist @33 -> pgid 33.a7bbaa1 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.a7bbaa1 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid noexist '@33' '@33' [read 0~256] tid 4 osd.3 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 4 2024-08-19T09:51:22.628 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 15 client.4546.objecter _send_op 4 to 33.1 on osd.3 2024-08-19T09:51:22.629 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:4 33.1 33:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53f62d30 con 0x559d53f639c0 2024-08-19T09:51:22.629 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.629 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.607+0000 7fbe3c38c780 5 client.4546.objecter 2 in flight 2024-08-19T09:51:22.629 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 579880347 0 3741568377) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.629 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-19T09:51:22.629 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.632 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 3 ondisk uv 1 in 33.1 attempt 0 2024-08-19T09:51:22.632 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 5 2024-08-19T09:51:22.632 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 3 2024-08-19T09:51:22.632 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 3 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 3 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.626+0000 7fbe3a67c640 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (25 ms) 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3041636674 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 4 ondisk uv 0 in 33.1 attempt 0 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 10 client.4546.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.633 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 4 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 4 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 4 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53f62880 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [read 0~256] tid 5 osd.3 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.634 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 5 2024-08-19T09:51:22.635 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 15 client.4546.objecter _send_op 5 to 33.1 on osd.3 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:5 33.1 33:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53f61ff0 con 0x559d53f639c0 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53f67870 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base noexist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target noexist @33 -> pgid 33.a7bbaa1 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.a7bbaa1 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.658 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid noexist '@33' '@33' [read 0~256] tid 6 osd.3 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 6 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 15 client.4546.objecter _send_op 6 to 33.1 on osd.3 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:6 33.1 33:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53f67d20 con 0x559d53f639c0 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.631+0000 7fbe3c38c780 5 client.4546.objecter 2 in flight 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 579880347 0 3741568377) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 5 ondisk uv 1 in 33.1 attempt 0 2024-08-19T09:51:22.659 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 5 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 5 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 5 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 5 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3041636674 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 6 ondisk uv 0 in 33.1 attempt 0 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:51:22.660 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 6 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 6 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 6 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.654+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (23 ms) 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53f67870 2024-08-19T09:51:22.661 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [read 0~256] tid 7 osd.3 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 7 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 15 client.4546.objecter _send_op 7 to 33.1 on osd.3 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:7 33.1 33:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53f62880 con 0x559d53f639c0 2024-08-19T09:51:22.662 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base noexist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target noexist @33 -> pgid 33.a7bbaa1 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.a7bbaa1 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid noexist '@33' '@33' [read 0~256] tid 8 osd.3 2024-08-19T09:51:22.663 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.664 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 8 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 15 client.4546.objecter _send_op 8 to 33.1 on osd.3 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:8 33.1 33:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e13f90 con 0x559d53f639c0 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.657+0000 7fbe3c38c780 5 client.4546.objecter 2 in flight 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 579880347 0 3741568377) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-19T09:51:22.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 7 ondisk uv 1 in 33.1 attempt 0 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 5 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 7 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 7 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 7 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3041636674 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 8 ondisk uv 0 in 33.1 attempt 0 2024-08-19T09:51:22.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 8 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 8 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 8 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (14 ms) 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-08-19T09:51:22.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-19T09:51:22.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 librados: queue_aio_write 0x559d53d725e0 completion 0x559d53f63230 write_seq 1 2024-08-19T09:51:22.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [write 0~5 in=5b] tid 9 osd.3 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 9 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 15 client.4546.objecter _send_op 9 to 33.1 on osd.3 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:9 33.1 33:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x559d53f67870 con 0x559d53f639c0 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.668+0000 7fbe3c38c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 9 ==== osd_op_reply(9 exist [write 0~5] v23'2 uv2 ondisk = 0) ==== 149+0+0 (unknown 506760037 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(9 exist [write 0~5] v23'2 uv2 ondisk = 0) 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 9 ondisk uv 2 in 33.1 attempt 0 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 9 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 9 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 9 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.676+0000 7fbe3a67c640 20 librados: complete_aio_write 0x559d53f63230 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (9 ms) 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-19T09:51:22.680 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 librados: queue_aio_write 0x559d53d725e0 completion 0x559d53f63340 write_seq 2 2024-08-19T09:51:22.681 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.694 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.694 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.694 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.694 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.694 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [write 0~5 in=5b] tid 10 osd.3 2024-08-19T09:51:22.694 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 10 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 15 client.4546.objecter _send_op 10 to 33.1 on osd.3 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:10 33.1 33:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x559d53f67870 con 0x559d53f639c0 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.677+0000 7fbe3c38c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 10 ==== osd_op_reply(10 exist [write 0~5] v23'3 uv3 ondisk = 0) ==== 149+0+0 (unknown 1790193841 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(10 exist [write 0~5] v23'3 uv3 ondisk = 0) 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 10 ondisk uv 3 in 33.1 attempt 0 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.695 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 10 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 10 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 10 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.692+0000 7fbe3a67c640 20 librados: complete_aio_write 0x559d53f63340 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (16 ms) 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 librados: queue_aio_write 0x559d53d725e0 completion 0x559d53f67a50 write_seq 3 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.696 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.704 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.704 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [write 0~5 in=5b] tid 11 osd.3 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 11 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 15 client.4546.objecter _send_op 11 to 33.1 on osd.3 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:11 33.1 33:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x559d53e17480 con 0x559d53f639c0 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.693+0000 7fbe3c38c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 11 ==== osd_op_reply(11 exist [write 0~5] v23'4 uv4 ondisk = 0) ==== 149+0+0 (unknown 551249260 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(11 exist [write 0~5] v23'4 uv4 ondisk = 0) 2024-08-19T09:51:22.705 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 11 ondisk uv 4 in 33.1 attempt 0 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 11 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 11 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 11 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3a67c640 20 librados: complete_aio_write 0x559d53f67a50 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (10 ms) 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.706 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.707 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.707 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.707 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.707 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [read 0~0] tid 12 osd.3 2024-08-19T09:51:22.707 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.707 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 12 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 15 client.4546.objecter _send_op 12 to 33.1 on osd.3 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:12 33.1 33:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e17480 con 0x559d53f639c0 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e14480 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base noexist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target noexist @33 -> pgid 33.a7bbaa1 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.a7bbaa1 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid noexist '@33' '@33' [read 0~0] tid 13 osd.3 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.708 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 13 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 15 client.4546.objecter _send_op 13 to 33.1 on osd.3 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:13 33.1 33:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14930 con 0x559d53f639c0 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.703+0000 7fbe3c38c780 5 client.4546.objecter 2 in flight 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 1354934122 0 3741568377) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 12 ondisk uv 4 in 33.1 attempt 0 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 5 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 12 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 12 2024-08-19T09:51:22.709 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 12 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3636694906 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 13 ondisk uv 0 in 33.1 attempt 0 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 13 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 13 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.710 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 13 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (3 ms) 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [read 0~0] tid 14 osd.3 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 14 2024-08-19T09:51:22.711 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 15 client.4546.objecter _send_op 14 to 33.1 on osd.3 2024-08-19T09:51:22.712 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:14 33.1 33:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14460 con 0x559d53f639c0 2024-08-19T09:51:22.712 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.712 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.712 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e149e0 2024-08-19T09:51:22.712 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base noexist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.712 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target noexist @33 -> pgid 33.a7bbaa1 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.a7bbaa1 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid noexist '@33' '@33' [read 0~0] tid 15 osd.3 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 15 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 15 client.4546.objecter _send_op 15 to 33.1 on osd.3 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:15 33.1 33:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14e90 con 0x559d53f639c0 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.706+0000 7fbe3c38c780 5 client.4546.objecter 2 in flight 2024-08-19T09:51:22.713 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 1354934122 0 3741568377) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 14 ondisk uv 4 in 33.1 attempt 0 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 5 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 14 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 14 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 14 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3636694906 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.714 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 15 ondisk uv 0 in 33.1 attempt 0 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 15 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 15 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 15 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-08-19T09:51:22.715 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [read 0~0] tid 16 osd.3 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 16 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 15 client.4546.objecter _send_op 16 to 33.1 on osd.3 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:16 33.1 33:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14460 con 0x559d53f639c0 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e149c0 2024-08-19T09:51:22.716 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base noexist @33 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target noexist @33 -> pgid 33.a7bbaa1 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.a7bbaa1 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid noexist '@33' '@33' [read 0~0] tid 17 osd.3 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 17 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 15 client.4546.objecter _send_op 17 to 33.1 on osd.3 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:17 33.1 33:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14e70 con 0x559d53f639c0 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.707+0000 7fbe3c38c780 5 client.4546.objecter 2 in flight 2024-08-19T09:51:22.717 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 1354934122 0 3741568377) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 16 ondisk uv 4 in 33.1 attempt 0 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 5 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 16 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 16 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 5 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 16 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3636694906 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.718 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 17 ondisk uv 0 in 33.1 attempt 0 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 17 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 17 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 17 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (3 ms) 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-08-19T09:51:22.719 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 20 librados: queue_aio_write 0x559d53d725e0 completion 0x559d53e176a0 write_seq 4 2024-08-19T09:51:22.720 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.720 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.720 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.720 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.720 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [writefull 0~5 in=5b] tid 18 osd.3 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 18 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 15 client.4546.objecter _send_op 18 to 33.1 on osd.3 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:18 33.1 33:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14460 con 0x559d53f639c0 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.710+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 18 ==== osd_op_reply(18 exist [writefull 0~5] v23'5 uv5 ondisk = 0) ==== 149+0+0 (unknown 3379324972 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(18 exist [writefull 0~5] v23'5 uv5 ondisk = 0) 2024-08-19T09:51:22.721 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 18 ondisk uv 5 in 33.1 attempt 0 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 18 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 18 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 18 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3a67c640 20 librados: complete_aio_write 0x559d53e176a0 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4 ms) 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-08-19T09:51:22.722 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 20 librados: queue_aio_write 0x559d53d725e0 completion 0x559d53e149c0 write_seq 5 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [writefull 0~5 in=5b] tid 19 osd.3 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 19 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 15 client.4546.objecter _send_op 19 to 33.1 on osd.3 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:19 33.1 33:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14460 con 0x559d53f639c0 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.723 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.714+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 19 ==== osd_op_reply(19 exist [writefull 0~5] v23'6 uv6 ondisk = 0) ==== 149+0+0 (unknown 1424860240 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(19 exist [writefull 0~5] v23'6 uv6 ondisk = 0) 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 19 ondisk uv 6 in 33.1 attempt 0 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 19 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 19 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 19 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3a67c640 20 librados: complete_aio_write 0x559d53e149c0 2024-08-19T09:51:22.724 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 20 librados: queue_aio_write 0x559d53d725e0 completion 0x559d53e17710 write_seq 6 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 10 client.4546.objecter _op_submit op 0x559d53e13ae0 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 20 client.4546.objecter _calc_target epoch 23 base exist @33 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 20 client.4546.objecter _calc_target target exist @33 -> pgid 33.d3a59201 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 10 client.4546.objecter _calc_target raw pgid 33.d3a59201 -> actual 33.1 acting [3,2] primary 3 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 20 client.4546.objecter _get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 10 client.4546.objecter _op_submit oid exist '@33' '@33' [writefull 0~5 in=5b] tid 20 osd.3 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 20 client.4546.objecter get_session s=0x559d53f4a350 osd=3 3 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 15 client.4546.objecter _session_op_assign 3 20 2024-08-19T09:51:22.725 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 15 client.4546.objecter _send_op 20 to 33.1 on osd.3 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 --> v1:172.21.15.82:6804/394826682 -- osd_op(unknown.0.0:20 33.1 33:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x559d53e14460 con 0x559d53f639c0 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.715+0000 7fbe3c38c780 5 client.4546.objecter 1 in flight 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 <== osd.3 v1:172.21.15.82:6804/394826682 20 ==== osd_op_reply(20 exist [writefull 0~5] v23'7 uv7 ondisk = 0) ==== 149+0+0 (unknown 544022404 0 0) 0x7fbe30003b20 con 0x559d53f639c0 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 10 client.4546.objecter ms_dispatch 0x559d53dad570 osd_op_reply(20 exist [writefull 0~5] v23'7 uv7 ondisk = 0) 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 10 client.4546.objecter in handle_osd_op_reply 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 7 client.4546.objecter handle_osd_op_reply 20 ondisk uv 7 in 33.1 attempt 0 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 10 client.4546.objecter op 0 rval 0 len 0 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 15 client.4546.objecter handle_osd_op_reply completed tid 20 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 15 client.4546.objecter _finish_op 20 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 4 2024-08-19T09:51:22.726 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 15 client.4546.objecter _session_op_remove 3 20 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 5 client.4546.objecter 0 in flight 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 20 librados: complete_aio_write 0x559d53e17710 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 10 librados: watch_flush enter 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 10 librados: watch_flush exit 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 10 client.4546.objecter close_session for osd.3 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6804/394826682 conn(0x559d53f639c0 legacy=0x559d53f65db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 20 client.4546.objecter put_session s=0x559d53f4a350 osd=3 1 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 20 client.4546.objecter shutdown clearing up homeless session... 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 10 client.4546.objecter successfully canceled tick 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbe2403f860 legacy=0x7fbe24041d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:51:22.727 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 10 monclient: shutdown 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 >> v1:172.21.15.82:6789/0 conn(0x559d53e12240 legacy=0x559d53f58320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3a67c640 1 -- 172.21.15.82:0/1475860754 reap_dead start 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 shutdown_connections 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 >> 172.21.15.82:0/1475860754 conn(0x559d53e330e0 msgr2=0x559d53f4d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 shutdown_connections 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 -- 172.21.15.82:0/1475860754 wait complete. 2024-08-19T09:51:22.728 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.717+0000 7fbe3c38c780 1 librados: shutdown 2024-08-19T09:51:22.729 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [----------] 12 tests from AsioRados (111 ms total) 2024-08-19T09:51:22.729 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: 2024-08-19T09:51:22.729 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [----------] Global test environment tear-down 2024-08-19T09:51:22.729 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (3373 ms total) 2024-08-19T09:51:22.729 INFO:tasks.workunit.client.0.smithi082.stdout: api_asio: [ PASSED ] 12 tests. 2024-08-19T09:51:23.419 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: starting. 2024-08-19T09:51:23.419 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: creating pool ceph_test_rados_list_parallel.smithi082-26493 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: created object 0... 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: created object 25... 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: created object 49... 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: finishing. 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_1_[26582]: shutting down. 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_2_[26587]: starting. 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_2_[26587]: listing objects. 2024-08-19T09:51:23.420 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_2_[26587]: listed object 0... 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_2_[26587]: listed object 25... 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_2_[26587]: saw 50 objects 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_2_[26587]: shutting down. 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: starting. 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: creating pool ceph_test_rados_list_parallel.smithi082-26493 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: created object 0... 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: created object 25... 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: created object 49... 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: finishing. 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_3_[27616]: shutting down. 2024-08-19T09:51:23.421 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_4_[27617]: starting. 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_4_[27617]: listing objects. 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_4_[27617]: listed object 0... 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_4_[27617]: listed object 25... 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_4_[27617]: saw 45 objects 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_4_[27617]: shutting down. 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_5_[27618]: starting. 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_5_[27618]: removed 25 objects... 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_5_[27618]: removed half of the objects 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_5_[27618]: removed 50 objects... 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_5_[27618]: removed 50 objects 2024-08-19T09:51:23.422 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_5_[27618]: shutting down. 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: starting. 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: creating pool ceph_test_rados_list_parallel.smithi082-26493 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: created object 0... 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: created object 25... 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: created object 49... 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: finishing. 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_6_[27669]: shutting down. 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: starting. 2024-08-19T09:51:23.423 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: listing objects. 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: listed object 0... 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: listed object 25... 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: listed object 50... 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: saw 53 objects 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_7_[27670]: shutting down. 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_8_[27671]: starting. 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_8_[27671]: added 25 objects... 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_8_[27671]: added half of the objects 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_8_[27671]: added 50 objects... 2024-08-19T09:51:23.424 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_8_[27671]: added 50 objects 2024-08-19T09:51:23.425 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_8_[27671]: shutting down. 2024-08-19T09:51:23.425 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.425 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:23.573 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ********************** api_cmd: Running main() from gmock_main.cc 2024-08-19T09:51:23.573 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [----------] Global test environment set-up. 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (772 ms) 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (51 ms) 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3175 ms) 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.461998+0000 mon.f [INF] from='client.? v1:172.21.15.82:0/42298181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi082-26893-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi082-26893-1"}]': finished 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.462115+0000 mon.f [INF] from='client.4745 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi082-26976-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi082-26976-1"}]': finished 2024-08-19T09:51:23.574 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.462185+0000 mon.f [INF] from='client.4702 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi082-26185-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi082-26185-1"}]': finished 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.462268+0000 mon.f [INF] from='client.4560 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26073-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.462346+0000 mon.f [INF] from='client.4864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26332-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.468325+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/2801551873' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-25997-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.473818+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-25997-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.640269+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/3075410934' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi082-25779-24"} : dispatch 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.642910+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi082-25779-24"} : dispatch 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.646080+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/3075410934' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi082-25779-24"} : dispatch 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.646576+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi082-25779-24"} : dispatch 2024-08-19T09:51:23.575 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.647346+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/3075410934' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi082-25779-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.647762+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi082-25779-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.660217+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2856875345' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.673632+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2890685693' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.676815+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.677251+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.679775+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2856875345' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.680308+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.681358+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2890685693' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.681981+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.687997+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2856875345' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:23.576 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.698048+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:23.577 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.702341+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/1344254999' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:23.577 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.708873+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:23.577 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.710085+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/1344254999' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:23.577 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.711389+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:23.577 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.711961+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/1344254999' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi082-26134-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.713660+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi082-26134-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.763365+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2890685693' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi082-25765-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:22.769165+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi082-25765-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.470656+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-25997-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.470716+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi082-25779-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.470776+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.470849+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi082-26134-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:24.528 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.470908+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi082-25765-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:24.529 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.476605+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2856875345' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi082-26173-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:24.529 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.477953+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/3075410934' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi082-25779-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi082-25779-24"} : dispatch 2024-08-19T09:51:24.529 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.478568+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/1344254999' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi082-26134-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:24.529 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.479804+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2890685693' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi082-25765-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:24.529 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.482421+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi082-26134-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:25.572 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.491007+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/153911870' entity='client.admin' cmd={"prefix": "osd pool application en delete_pools_parallel: process_1_[26689]: starting. 2024-08-19T09:51:25.572 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_1_[26689]: creating pool ceph_test_rados_delete_pools_parallel.smithi082-26602 2024-08-19T09:51:25.572 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_1_[26689]: created object 0... 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_1_[26689]: created object 25... 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_1_[26689]: created object 49... 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_1_[26689]: finishing. 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_1_[26689]: shutting down. 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_2_[26691]: starting. 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_2_[26691]: deleting pool ceph_test_rados_delete_pools_parallel.smithi082-26602 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_2_[26691]: shutting down. 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: ******************************* 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: starting. 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: creating pool ceph_test_rados_delete_pools_parallel.smithi082-26602 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: created object 0... 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: created object 25... 2024-08-19T09:51:25.573 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: created object 49... 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: finishing. 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_3_[27331]: shutting down. 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: ******************************* 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_5_[27333]: starting. 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_5_[27333]: listing objects. 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_5_[27333]: listed object 0... 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_5_[27333]: listed object 25... 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_5_[27333]: saw 50 objects 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_5_[27333]: shutting down. 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: ******************************* 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_4_[27332]: starting. 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_4_[27332]: deleting pool ceph_test_rados_delete_pools_parallel.smithi082-26602 2024-08-19T09:51:25.574 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: process_4_[27332]: shutting down. 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: ******************************* 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: ******************************* 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: starting. 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: creating pool ceph_test_rados_open_pools_parallel.smithi082-26607 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: created object 0... 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: created object 25... 2024-08-19T09:51:25.575 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: created object 49... 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: finishing. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_1_[26674]: shutting down. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_2_[26692]: starting. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_2_[26692]: rados_pool_create. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_2_[26692]: rados_ioctx_create. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_2_[26692]: shutting down. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: ******************************* 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: starting. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: creating pool ceph_test_rados_open_pools_parallel.smithi082-26607 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: created object 0... 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: created object 25... 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: created object 49... 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: finishing. 2024-08-19T09:51:25.576 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_3_[27329]: shutting down. 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: ******************************* 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_4_[27330]: starting. 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_4_[27330]: rados_pool_create. 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_4_[27330]: rados_ioctx_create. 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: process_4_[27330]: shutting down. 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: ******************************* 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: ******************************* 2024-08-19T09:51:25.577 INFO:tasks.workunit.client.0.smithi082.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-08-19T09:51:26.579 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: Running main() from gmock_main.cc 2024-08-19T09:51:26.579 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-08-19T09:51:26.579 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [----------] Global test environment set-up. 2024-08-19T09:51:26.579 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-08-19T09:51:26.579 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1835 ms) 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2052 ms) 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3070 ms) 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6957 ms total) 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [----------] Global test environment tear-down 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6958 ms total) 2024-08-19T09:51:26.580 INFO:tasks.workunit.client.0.smithi082.stdout: cmd: [ PASSED ] 3 tests. 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout:********* 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: starting. 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: creating pool ceph_test_rados_list_parallel.smithi082-26493 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: created object 0... 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: created object 25... 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: created object 49... 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: finishing. 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_9_[27720]: shutting down. 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.645 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: starting. 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: listing objects. 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: listed object 0... 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: listed object 25... 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: listed object 50... 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: listed object 75... 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: saw 100 objects 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_10_[27721]: shutting down. 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.646 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_11_[27722]: starting. 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_11_[27722]: added 25 objects... 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_11_[27722]: added half of the objects 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_11_[27722]: added 50 objects... 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_11_[27722]: added 50 objects 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_11_[27722]: shutting down. 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_12_[27723]: starting. 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_12_[27723]: added 25 objects... 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_12_[27723]: added half of the objects 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_12_[27723]: added 50 objects... 2024-08-19T09:51:26.647 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_12_[27723]: added 50 objects 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_12_[27723]: shutting down. 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_13_[27724]: starting. 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_13_[27724]: removed 25 objects... 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_13_[27724]: removed half of the objects 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_13_[27724]: removed 50 objects... 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_13_[27724]: removed 50 objects 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_13_[27724]: shutting down. 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.648 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: starting. 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: creating pool ceph_test_rados_list_parallel.smithi082-26493 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: created object 0... 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: created object 25... 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: created object 49... 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: finishing. 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_14_[28386]: shutting down. 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.649 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:26.650 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: starting. 2024-08-19T09:51:26.650 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listing objects. 2024-08-19T09:51:26.650 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listed object 0... 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listed object api_io_pp: Running main() from gmock_main.cc 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [----------] Global test environment set-up. 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: seed 25779 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-08-19T09:51:26.683 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (224 ms) 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (20 ms) 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (9 ms) 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (606 ms) 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (7 ms) 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5 ms) 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-08-19T09:51:26.684 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (8 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (2 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (1 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (3 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3 ms) 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-08-19T09:51:26.685 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (18 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (8 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (3 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (5 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (1 ms) 2024-08-19T09:51:26.686 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (937 ms total) 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1090 ms) 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-08-19T09:51:26.687 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (20 ms) 2024-08-19T09:51:27.562 INFO:tasks.workunit.client.0.smithi082.stdout: able","pool": "SimpleWrite_smithi082-25728-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.562 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.494867+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/118930509' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi082-25743-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.562 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.497413+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi082-25779-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi082-25779-24"} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.497717+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi082-26173-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.497980+0000 mon.f [INF] from='client.? v1:172.21.15.82:0/42298181' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi082-26893-1"} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.498169+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi082-25765-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.510891+0000 mon.f [INF] from='client.4796 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi082-25728-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.511324+0000 mon.f [INF] from='client.4850 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi082-25743-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.569128+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/365509660' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.569934+0000 client.admin [INF] onexx 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.570826+0000 mon.f [INF] from='client.4965 ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.575410+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2174040617' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26497-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.563 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:23.580096+0000 mon.f [INF] from='client.4841 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26497-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525274+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi082-26134-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi082-26134-7"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525358+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi082-25779-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi082-25779-24"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525437+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi082-26173-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi082-26173-7"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525520+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi082-25765-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi082-25765-16"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525587+0000 mon.f [INF] from='client.? v1:172.21.15.82:0/42298181' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi082-26893-1"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525635+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-25997-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525695+0000 mon.f [INF] from='client.4745 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi082-26976-1"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525763+0000 mon.f [INF] from='client.4702 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi082-26185-1"}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525829+0000 mon.f [INF] from='client.5062 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26073-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525892+0000 mon.f [INF] from='client.5006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi082-25842-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:27.564 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.525975+0000 mon.f [INF] from='client.5108 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi082-25819-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:27.565 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.529722+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/1779429450' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi082-25819-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi082-25819-10"} : dispatch 2024-08-19T09:51:27.565 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.532954+0000 mon.b [INF] from='client.? v1:172.21.15.82:0/2070353927' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi082-26976-1"} : dispatch 2024-08-19T09:51:27.565 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.534313+0000 mon.f [INF] from='client.5108 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi082-25819-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi082-25819-10"} : dispatch 2024-08-19T09:51:27.565 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.537571+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/795697837' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi082-26185-1"} : dispatch 2024-08-19T09:51:27.565 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.537887+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/118930509' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi082-25743-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.565 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.540409+0000 mon.b [INF] from='client.? v1:172.21.15.82:0/3979547797' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi082-25842-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi082-25842-10"} : dispatch 2024-08-19T09:51:27.766 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.541916+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/2801551873' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi082-25997-4", "tierpool": "test-rados-api-smithi082-25997-4-cache"} : dispatch 2024-08-19T09:51:27.766 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.547241+0000 mon.f [INF] from='client.4965 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-08-19T09:51:27.766 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.555242+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi082-25997-4", "tierpool": "test-rados-api-smithi082-25997-4-cache"} : dispatch 2024-08-19T09:51:27.766 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.555680+0000 mon.f [INF] from='client.4702 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi082-26185-1"} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.563992+0000 mon.f [INF] from='client.4850 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi082-25743-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.564274+0000 mon.f [INF] from='client.4745 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi082-26976-1"} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.564717+0000 mon.f [INF] from='client.5006 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi082-25842-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi082-25842-10"} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.623447+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/353269376' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi082-26893-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:25.640444+0000 mon.f [INF] from='client.5197 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi082-26893-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.537937+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi082-25997-4", "tierpool": "test-rados-api-smithi082-25997-4-cache"}]': finished 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.538063+0000 mon.f [INF] from='client.4702 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi082-26185-1"}]': finished 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.538146+0000 mon.f [INF] from='client.4850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi082-25743-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.538215+0000 mon.f [INF] from='client.4745 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi082-26976-1"}]': finished 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.538271+0000 mon.f [INF] from='client.5197 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi082-26893-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.546748+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/353269376' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi082-26893-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi082-26893-2"} : dispatch 2024-08-19T09:51:27.767 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.550781+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/118930509' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi082-25743-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.551402+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/2801551873' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi082-25997-4", "overlaypool": "test-rados-api-smithi082-25997-4-cache"} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.572496+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/823595302' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi082-25728-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.575730+0000 mon.f [INF] from='client.5197 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi082-26893-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi082-26893-2"} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.576072+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi082-25997-4", "overlaypool": "test-rados-api-smithi082-25997-4-cache"} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.576277+0000 mon.f [INF] from='client.5109 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi082-25728-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.576739+0000 mon.f [INF] from='client.4850 ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi082-25743-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.577565+0000 mon.b [INF] from='client.? v1:172.21.15.82:0/1582608883' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26497-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.589914+0000 mon.f [INF] from='client.5168 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26497-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.619504+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/1541923894' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi082-26976-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.624934+0000 mon.f [INF] from='client.5306 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi082-26976-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.763444+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/3075410934' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi082-25779-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-08-19T09:51:27.768 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:26.763785+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi082-25779-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542366+0000 mon.f [INF] from='client.5108 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi082-25819-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi082-25819-10"}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542444+0000 mon.f [INF] from='client.5006 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi082-25842-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi082-25842-10"}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542501+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi082-25997-4", "overlaypool": "test-rados-api-smithi082-25997-4-cache"}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542553+0000 mon.f [INF] from='client.5109 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi082-25728-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542603+0000 mon.f [INF] from='client.4850 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi082-25743-3", "field": "max_bytes", "val": "4096"}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542650+0000 mon.f [INF] from='client.5168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26497-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542727+0000 mon.f [INF] from='client.5306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi082-26976-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.542797+0000 mon.f [INF] from='client.4803 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi082-25779-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.546941+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/2801551873' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi082-25997-4-cache", "mode": "writeback"} : dispatch 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.547401+0000 mon.h [INF] from='client.? v1:172.21.15.82:0/1541923894' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi082-26976-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi082-26976-2"} : dispatch 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.557216+0000 mon.f [INF] from='client.4892 ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi082-25997-4-cache", "mode": "writeback"} : dispatch 2024-08-19T09:51:27.769 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.557546+0000 mon.f [INF] from='client.5306 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi082-26976-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi082-26976-2"} : dispatch 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.559855+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2890685693' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.561273+0000 mon.f [INF] from='client.4832 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi082-25765-16"} : dispatch 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.561845+0000 mon.d [INF] from='client.? v1:172.21.15.82:0/365509660' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.562543+0000 mon.c [INF] from='client.? v1:172.21.15.82:0/1344254999' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.563323+0000 client.admin [INF] threexx 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.564392+0000 mon.f [INF] from='client.4965 ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-08-19T09:51:27.770 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.564652+0000 mon.f [INF] from='client.4981 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi082-26134-7"} : dispatch 2024-08-19T09:51:29.666 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.565776+0000 mon.e [IN list: Running main() from gmock_main.cc 2024-08-19T09:51:29.666 INFO:tasks.workunit.client.0.smithi082.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [----------] Global test environment set-up. 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [----------] 3 tests from NeoradosList 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ OK ] NeoradosList.ListObjects (2806 ms) 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ OK ] NeoradosList.ListObjectsNS (4023 ms) 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3154 ms) 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [----------] 3 tests from NeoradosList (9983 ms total) 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [----------] Global test environment tear-down 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [==========] 3 tests from 1 test suite ran. (9983 ms total) 2024-08-19T09:51:29.667 INFO:tasks.workunit.client.0.smithi082.stdout: list: [ PASSED ] 3 tests. 2024-08-19T09:51:29.669 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: Running main() from gmock_main.cc 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [----------] Global test environment set-up. 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStat.Stat (220 ms) 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (13 ms) 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStat.StatNS (51 ms) 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-08-19T09:51:29.670 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [----------] 5 tests from LibRadosStat (286 ms total) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1088 ms) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (35 ms) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (4 ms) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1127 ms total) 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: 2024-08-19T09:51:29.671 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [----------] Global test environment tear-down 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10221 ms total) 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat: [ PASSED ] 9 tests. 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: Running main() from gmock_main.cc 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [----------] Global test environment set-up. 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (279 ms) 2024-08-19T09:51:29.672 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: no timeout :/ 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: no timeout :/ 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: no timeout :/ 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: no timeout :/ 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: no timeout :/ 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (53 ms) 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (17 ms) 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.Checksum (7 ms) 2024-08-19T09:51:29.673 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (16 ms) 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (10 ms) 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (17 ms) 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (5 ms) 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-08-19T09:51:29.674 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3 ms) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.RmXattr (27 ms) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIo.XattrIter (12 ms) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [----------] 14 tests from LibRadosIo (454 ms total) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1087 ms) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (12 ms) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-08-19T09:51:29.675 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (21 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (7 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (11 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (27 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-08-19T09:51:29.676 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1185 ms total) 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: api_stat_pp: Running main() from gmock_main.cc 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: seed 26173 2024-08-19T09:51:29.677 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (218 ms) 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (29 ms) 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (10 ms) 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (17 ms) 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (275 ms total) 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-08-19T09:51:29.678 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1108 ms) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (6 ms) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16 ms) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1130 ms total) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10215 ms total) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: [----------] Global test environment tear-down 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [==========] 24 tests from 2 test suites ran. (10360 ms total) 2024-08-19T09:51:29.679 INFO:tasks.workunit.client.0.smithi082.stdout: api_io: [ PASSED ] 24 tests. 2024-08-19T09:51:31.155 INFO:tasks.workunit.client.0.smithi082.stdout:25... 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listed object 50... 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listed object 75... 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listed object 100... 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: listed object 125... 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: saw 150 objects 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_15_[28387]: shutting down. 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_16_[28388]: starting. 2024-08-19T09:51:31.156 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_16_[28388]: added 25 objects... 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_16_[28388]: added half of the objects 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_16_[28388]: added 50 objects... 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_16_[28388]: added 50 objects 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: process_16_[28388]: shutting down. 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******************************* 2024-08-19T09:51:31.157 INFO:tasks.workunit.client.0.smithi082.stdout: list_parallel: ******* SUCCESS ********** 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: Running main() from gmock_main.cc 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [----------] Global test environment set-up. 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: handle_notify cookie 94856110635792 notify_id 98784247808 notifier_gid 4708 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (3744 ms) 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: Trying... 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: handle_notify cookie 94856124118352 notify_id 111669149699 notifier_gid 4931 2024-08-19T09:51:31.158 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: Waiting for 3.000000000s 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: Timed out. 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: Flushing... 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: Flushed... 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7663 ms) 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (11407 ms total) 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [----------] Global test environment tear-down 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (11407 ms total) 2024-08-19T09:51:31.159 INFO:tasks.workunit.client.0.smithi082.stdout: watch_notify: [ PASSED ] 2 tests. 2024-08-19T09:51:31.329 INFO:tasks.workunit.client.0.smithi082.stdout:F] from='client.? v1:172.21.15.82:0/1905060361' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi082-26185-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.566532+0000 mon.f [INF] from='client.5138 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi082-26185-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.567813+0000 mon.e [INF] from='client.? v1:172.21.15.82:0/2856875345' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.568489+0000 mon.f [INF] from='client.4805 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi082-26173-7"} : dispatch 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: got: 2024-08-19T09:51:27.568977+0000 mon.f [INF] from='client.? v1:172.21.15.82:0/142709954' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi082-26073-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7832 ms) 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (11830 ms total) 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [----------] Global test environment tear-down 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (11830 ms total) 2024-08-19T09:51:31.330 INFO:tasks.workunit.client.0.smithi082.stdout: api_cmd: [ PASSED ] 4 tests. 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: Running main() from gmock_main.cc 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [----------] Global test environment set-up. 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5852 ms) 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-08-19T09:51:33.394 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7863 ms) 2024-08-19T09:51:33.395 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (13715 ms total) 2024-08-19T09:51:33.395 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: 2024-08-19T09:51:33.395 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [----------] Global test environment tear-down 2024-08-19T09:51:33.395 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (13715 ms total) 2024-08-19T09:51:33.395 INFO:tasks.workunit.client.0.smithi082.stdout: ec_io: [ PASSED ] 2 tests. 2024-08-19T09:51:33.785 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: Running main() from gmock_main.cc 2024-08-19T09:51:33.785 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [----------] Global test environment set-up. 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (260 ms) 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3 ms) 2024-08-19T09:51:33.786 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1078 ms) 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1004 ms) 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (8 ms) 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.Unlock (12 ms) 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-08-19T09:51:33.787 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [----------] 8 tests from LibRadosLock (2372 ms total) 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1116 ms) 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (9 ms) 2024-08-19T09:51:33.788 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1042 ms) 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1015 ms) 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (11 ms) 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (17 ms) 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (25 ms) 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-08-19T09:51:33.789 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3239 ms total) 2024-08-19T09:51:33.790 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: 2024-08-19T09:51:33.790 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [----------] Global test environment tear-down 2024-08-19T09:51:33.790 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (14442 ms total) 2024-08-19T09:51:33.790 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock: [ PASSED ] 16 tests. 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: seed 25842 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (214 ms) 2024-08-19T09:51:33.791 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-08-19T09:51:33.792 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (29 ms) 2024-08-19T09:51:33.792 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-08-19T09:51:33.792 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1081 ms) 2024-08-19T09:51:33.792 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-08-19T09:51:33.792 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1012 ms) 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (8 ms) 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (10 ms) 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-08-19T09:51:33.793 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2362 ms total) 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1114 ms) 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (10 ms) 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-08-19T09:51:33.794 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1046 ms) 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1021 ms) 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (6 ms) 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (13 ms) 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (5 ms) 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-08-19T09:51:33.795 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3219 ms total) 2024-08-19T09:51:33.796 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: 2024-08-19T09:51:33.796 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-08-19T09:51:33.796 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (14434 ms total) 2024-08-19T09:51:33.796 INFO:tasks.workunit.client.0.smithi082.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-08-19T09:51:36.580 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1115 ms) 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (21 ms) 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1136 ms total) 2024-08-19T09:51:36.581 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (106 ms) 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4821 ms) 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (18 ms) 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-08-19T09:51:36.582 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (4 ms) 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710137136 notify_id 141733920770 notifier_gid 5138 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710137136 notify_id 141733920774 notifier_gid 5138 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (10 ms) 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710140432 notify_id 141733920775 notifier_gid 5138 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (10 ms) 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-08-19T09:51:36.583 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710091760 notify_id 141733920768 notifier_gid 5138 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (26 ms) 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710092128 notify_id 141733920776 notifier_gid 5138 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710092128 notify_id 141733920777 notifier_gid 5138 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (9 ms) 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: trying... 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710092128 notify_id 141733920771 notifier_gid 5138 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: timed out 2024-08-19T09:51:36.584 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: flushing 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_ pool: Running main() from gmock_main.cc 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [----------] Global test environment set-up. 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ OK ] NeoRadosPools.PoolList (2772 ms) 2024-08-19T09:51:38.445 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2044 ms) 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2060 ms) 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (7200 ms) 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2017 ms) 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2639 ms) 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [----------] 6 tests from NeoRadosPools (18733 ms total) 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: 2024-08-19T09:51:38.446 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [----------] Global test environment tear-down 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [==========] 6 tests from 1 test suite ran. (18733 ms total) 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: pool: [ PASSED ] 6 tests. 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout:api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (8 ms) 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (9 ms) 2024-08-19T09:51:38.447 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (7 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (7 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (3 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (9 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (15 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (17 ms) 2024-08-19T09:51:38.448 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (7017 ms) 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1106 ms) 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (8 ms) 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (15 ms) 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (9358 ms total) 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [----------] Global test environment tear-down 2024-08-19T09:51:38.449 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (19124 ms total) 2024-08-19T09:51:38.450 INFO:tasks.workunit.client.0.smithi082.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-08-19T09:51:42.528 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: Running main() from gmock_main.cc 2024-08-19T09:51:42.528 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-08-19T09:51:42.528 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [----------] Global test environment set-up. 2024-08-19T09:51:42.528 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-08-19T09:51:42.528 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-08-19T09:51:42.528 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6841 ms) 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7238 ms) 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8735 ms) 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [----------] 3 tests from NeoradosECList (22814 ms total) 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [----------] Global test environment tear-down 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22814 ms total) 2024-08-19T09:51:42.529 INFO:tasks.workunit.client.0.smithi082.stdout: ec_list: [ PASSED ] 3 tests. 2024-08-19T09:51:43.386 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify_test_cb 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (271 ms) 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93893589174048 err -107 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (17 ms) 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-08-19T09:51:43.387 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93893589191824 err -107 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1021 ms) 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 98784247808 cookie 93893589213296 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9 ms) 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 98784247809 cookie 93893589213296 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (5 ms) 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 98784247808 cookie 93893589260608 2024-08-19T09:51:43.388 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6 ms) 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 98784247809 cookie 93893589254368 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 98784247809 cookie 93893589260608 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (24 ms) 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 98784247809 cookie 93893589260608 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 103079215106 cookie 93893589260608 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3004 ms) 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-08-19T09:51:43.389 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93893589260608 err -107 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_cb from 4757 notify_id 128849018884 cookie 93893589260608 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5008 ms) 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93893589260608 err -107 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10371 ms total) 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-08-19T09:51:43.390 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: watch_notify_test_cb 2024-08-19T09:51:43.391 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1054 ms) 2024-08-19T09:51:43.391 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1054 ms total) 2024-08-19T09:51:43.391 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: 2024-08-19T09:51:43.391 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-08-19T09:51:43.391 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (23905 ms total) 2024-08-19T09:51:43.391 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-08-19T09:51:43.802 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: Running main() from gmock_main.cc 2024-08-19T09:51:43.802 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-08-19T09:51:43.802 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [----------] Global test environment set-up. 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (3754 ms) 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3079 ms) 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3088 ms) 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3741 ms) 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2398 ms) 2024-08-19T09:51:43.803 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3644 ms) 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (4341 ms) 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (24045 ms total) 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [----------] Global test environment tear-down 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (24045 ms total) 2024-08-19T09:51:43.804 INFO:tasks.workunit.client.0.smithi082.stdout: write_operations: [ PASSED ] 7 tests. 2024-08-19T09:51:49.825 INFO:tasks.workunit.client.0.smithi082.stdout:notify_pp: flushed 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3008 ms) 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: trying... 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710092128 notify_id 154618822660 notifier_gid 5138 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: timed out 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: flushing 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: flushed 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3011 ms) 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: List watches 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify2 2024-08-19T09:51:49.826 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710092128 notify_id 167503724554 notifier_gid 5138 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify2 done 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: watch_check 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: unwatch2 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: flushing 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: done 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3011 ms) 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: List watches 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify2 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: handle_notify cookie 94910710092128 notify_id 180388626443 notifier_gid 5138 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: notify2 done 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: watch_check 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: unwatch2 2024-08-19T09:51:49.827 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: flushing 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: done 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3017 ms) 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (17063 ms total) 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (30346 ms total) 2024-08-19T09:51:49.828 INFO:tasks.workunit.client.0.smithi082.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-08-19T09:51:49.992 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [----------] Global test environment set-up. 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5079 ms) 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (61 ms) 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20043 ms) 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-08-19T09:51:49.993 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: attempt 0 of 20 2024-08-19T09:51:49.994 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5247 ms) 2024-08-19T09:51:49.994 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30430 ms total) 2024-08-19T09:51:49.994 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: 2024-08-19T09:51:49.994 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [----------] Global test environment tear-down 2024-08-19T09:51:49.994 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30431 ms total) 2024-08-19T09:51:49.994 INFO:tasks.workunit.client.0.smithi082.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-08-19T09:51:51.893 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2946 ms) 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3070 ms) 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3016 ms) 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (6214 ms) 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (5662 ms) 2024-08-19T09:51:51.894 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3048 ms) 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8362 ms) 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (32318 ms total) 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (32318 ms total) 2024-08-19T09:51:51.895 INFO:tasks.workunit.client.0.smithi082.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [----------] Global test environment set-up. 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [----------] 4 tests from LibRadosService 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5062 ms) 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ OK ] LibRadosService.RegisterLate (41 ms) 2024-08-19T09:51:52.292 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: cluster: 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: id: 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: health: HEALTH_WARN 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 3/9 mons down, quorum f,b,c,h,d,e 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 12 pool(s) do not have an application enabled 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: services: 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: mon: 9 daemons, quorum f,b,c,h,d,e (age 39s), out of quorum: a, g, i 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: mgr: x(active, since 90s) 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: osd: 6 osds: 6 up (since 83s), 6 in (since 92s) 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-08-19T09:51:52.293 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: data: 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: pools: 26 pools, 705 pgs 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: objects: 210 objects, 145 MiB 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: usage: 205 MiB used, 540 GiB / 540 GiB avail 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: pgs: 20.000% pgs unknown 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 12.908% pgs not active 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 473 active+clean 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 141 unknown 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 91 creating+peering 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: io: 2024-08-19T09:51:52.294 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: client: 12 MiB/s rd, 37 MiB/s wr, 183 op/s rd, 299 op/s wr 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6602 ms) 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ RUN ] LibRadosService.Status 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ OK ] LibRadosService.Status (21031 ms) 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [----------] 4 tests from LibRadosService (32737 ms total) 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [----------] Global test environment tear-down 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32737 ms total) 2024-08-19T09:51:52.295 INFO:tasks.workunit.client.0.smithi082.stdout: api_service: [ PASSED ] 4 tests. 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: Running main() from gmock_main.cc 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [----------] Global test environment set-up. 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.Version (2807 ms) 2024-08-19T09:51:56.852 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (5087 ms) 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.LongName (3591 ms) 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2624 ms) 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3630 ms) 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3040 ms) 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.Exec (3048 ms) 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-08-19T09:51:56.853 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2289 ms) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3015 ms) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2986 ms) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1990 ms) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3048 ms) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [----------] 12 tests from NeoRadosMisc (37156 ms total) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [----------] Global test environment tear-down 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [==========] 12 tests from 1 test suite ran. (37156 ms total) 2024-08-19T09:51:56.854 INFO:tasks.workunit.client.0.smithi082.stdout: misc: [ PASSED ] 12 tests. 2024-08-19T09:52:02.177 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: Running main() from gmock_main.cc 2024-08-19T09:52:02.177 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [----------] Global test environment set-up. 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4078 ms) 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3095 ms) 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3074 ms) 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3740 ms) 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2398 ms) 2024-08-19T09:52:02.178 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5706 ms) 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4294 ms) 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5043 ms) 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (11327 ms) 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [----------] 9 tests from LibRadosPools (42756 ms total) 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [----------] Global test environment tear-down 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (42757 ms total) 2024-08-19T09:52:02.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_pool: [ PASSED ] 9 tests. 2024-08-19T09:52:02.182 INFO:tasks.workunit.client.0.smithi082.stdout: io: Running main() from gmock_main.cc 2024-08-19T09:52:02.182 INFO:tasks.workunit.client.0.smithi082.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-08-19T09:52:02.182 INFO:tasks.workunit.client.0.smithi082.stdout: io: [----------] Global test environment set-up. 2024-08-19T09:52:02.182 INFO:tasks.workunit.client.0.smithi082.stdout: io: [----------] 14 tests from NeoRadosIo 2024-08-19T09:52:02.182 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.Limits (2805 ms) 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3062 ms) 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.ReadOp (3026 ms) 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.SparseRead (3603 ms) 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2614 ms) 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3664 ms) 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-08-19T09:52:02.183 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3024 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2320 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3000 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.Trunc (3004 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.Remove (3006 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3034 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.RmXattr (2982 ms) 2024-08-19T09:52:02.184 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3350 ms) 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: io: [----------] 14 tests from NeoRadosIo (42494 ms total) 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: io: 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: io: [----------] Global test environment tear-down 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: io: [==========] 14 tests from 1 test suite ran. (42494 ms total) 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: io: [ PASSED ] 14 tests. 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: Running main() from gmock_main.cc 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-08-19T09:52:02.185 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [----------] Global test environment set-up. 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2756 ms) 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3070 ms) 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3021 ms) 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3596 ms) 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2632 ms) 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-08-19T09:52:02.186 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3632 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3055 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2292 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3021 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3001 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3028 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2999 ms) 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-08-19T09:52:02.187 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2997 ms) 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3350 ms) 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (42450 ms total) 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [----------] Global test environment tear-down 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (42450 ms total) 2024-08-19T09:52:02.188 INFO:tasks.workunit.client.0.smithi082.stdout: read_operations: [ PASSED ] 14 tests. 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] Global test environment set-up. 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: 2024-08-19T09:52:06.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: 2024-08-19T09:51:19.453+0000 7f04aef16780 -1 failed for service _ceph-mon._tcp 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: 2024-08-19T09:51:19.453+0000 7f04aef16780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (68 ms) 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5019 ms) 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5087 ms total) 2024-08-19T09:52:06.223 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: started 0x56064c609740 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: started 0x56064cf87e60 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: started 2 aios 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: waiting 0x56064c609740 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: waiting 0x56064cf87e60 2024-08-19T09:52:06.224 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: done. 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (6677 ms) 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (6677 ms total) 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.Exec (68 ms) 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (19 ms) 2024-08-19T09:52:06.225 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5244 ms) 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (27052 ms) 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] 8 tests from LibRadosMisc (32385 ms total) 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [----------] Global test environment tear-down 2024-08-19T09:52:06.226 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (46815 ms total) 2024-08-19T09:52:06.227 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc: [ PASSED ] 12 tests. 2024-08-19T09:52:15.808 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: Running main() from gmock_main.cc 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] Global test environment set-up. 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2067 ms) 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2042 ms) 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-08-19T09:52:15.809 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2030 ms) 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2087 ms) 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8226 ms total) 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4083 ms) 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4052 ms) 2024-08-19T09:52:15.810 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4330 ms) 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (12465 ms total) 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3035 ms) 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1961 ms) 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2028 ms) 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-08-19T09:52:15.811 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2008 ms) 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9032 ms total) 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (3608 ms) 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4435 ms) 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8043 ms total) 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [----------] Global test environment tear-down 2024-08-19T09:52:15.812 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (56381 ms total) 2024-08-19T09:52:15.813 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-08-19T09:52:22.844 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-08-19T09:52:22.844 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T09:52:22.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T09:52:22.848 INFO:tasks.ceph.mon.g:Started 2024-08-19T09:52:22.848 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T09:52:22.848 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T09:52:22.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T09:52:22.851 INFO:tasks.ceph.mon.a:Started 2024-08-19T09:52:22.851 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-08-19T09:52:22.851 INFO:tasks.ceph.mon.i:Restarting daemon 2024-08-19T09:52:22.852 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-08-19T09:52:22.855 INFO:tasks.ceph.mon.i:Started 2024-08-19T09:52:22.855 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-08-19T09:52:22.855 INFO:tasks.ceph.mon.b:Sent signal 19 2024-08-19T09:52:22.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:52:32.967 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: seed 25992 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-08-19T09:52:32.968 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (222 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (37 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (6 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (11 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (8 ms) 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-08-19T09:52:32.969 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1 ms) 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (33 ms) 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (6 ms) 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (18 ms) 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (6 ms) 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: osd_max_attr_size = 0 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-08-19T09:52:32.970 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2361 ms) 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (242 ms) 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: done waiting, doing copies 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: waiting for final deep scrubs... 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: done waiting 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60597 ms) 2024-08-19T09:52:32.971 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-08-19T09:52:32.974 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (6 ms) 2024-08-19T09:52:32.974 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-08-19T09:52:32.974 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-08-19T09:52:32.974 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (7907 ms) 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-08-19T09:52:32.975 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-08-19T09:52:35.992 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP snapshots: Running main() from gmock_main.cc 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [----------] Global test environment set-up. 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (5812 ms) 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5600 ms) 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3637 ms) 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5672 ms) 2024-08-19T09:52:35.993 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6342 ms) 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (27063 ms total) 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5003 ms) 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6190 ms) 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8211 ms) 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6005 ms) 2024-08-19T09:52:35.994 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3598 ms) 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi082-27136-11. 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: Waiting for snaps to purge. 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20179 ms) 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (49186 ms total) 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [----------] Global test environment tear-down 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (76249 ms total) 2024-08-19T09:52:35.995 INFO:tasks.workunit.client.0.smithi082.stdout: snapshots: [ PASSED ] 11 tests. 2024-08-19T09:52:37.857 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-08-19T09:52:37.857 INFO:tasks.ceph.mon.b:Sent signal 18 2024-08-19T09:52:37.858 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:52:37.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:52:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.981+0000 7ff0c966e640 10 monclient: get_monmap_and_config 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.981+0000 7ff0c966e640 10 monclient: build_initial_monmap 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.981+0000 7ff0c966e640 10 monclient: monmap: 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:37.982433+0000 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:37.982433+0000 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 1 Processor -- start 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 1 -- start start 2024-08-19T09:52:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient: init 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient: _add_conns ranks=[7,4,2,1,0,6,5,3,8] 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient(hunting): picked mon.noname-i con 0x7ff0c40a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c40a3b40 con 0x7ff0c413f5c0 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c40a8130 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c406ede0 con 0x7ff0c40a67c0 2024-08-19T09:52:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c966e640 10 monclient(hunting): authenticate will time out at 814.058732s 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c27fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff0c40a6ba0 0x7ff0c40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36428/0 (socket says 172.21.15.1:36428) 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.982+0000 7ff0c27fc640 1 -- 172.21.15.1:0/3730928141 learned_addr learned my addr 172.21.15.1:0/3730928141 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.984+0000 7ff0c27fc640 1 -- 172.21.15.1:0/3730928141 >> v1:172.21.15.1:6791/0 conn(0x7ff0c40a6ba0 legacy=0x7ff0c40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c37fe640 1 -- 172.21.15.1:0/3730928141 >> v1:172.21.15.1:6790/0 conn(0x7ff0c413f5c0 legacy=0x7ff0c41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c2ffd640 1 -- 172.21.15.1:0/3730928141 >> v1:172.21.15.82:6792/0 conn(0x7ff0c40a67c0 legacy=0x7ff0c40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2248994621 0 0) 0x7ff0c40a8130 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): my global_id is 26401 2024-08-19T09:52:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac003520 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1753687793 0 0) 0x7ff0c40a3b40 con 0x7ff0c413f5c0 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): my global_id is 26471 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0c40a8130 con 0x7ff0c413f5c0 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211548567 0 0) 0x7ff0c406ede0 con 0x7ff0c40a67c0 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient(hunting): my global_id is 134116 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0c40a3b40 con 0x7ff0c40a67c0 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3810377361 0 0) 0x7ff0ac003520 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 >> v1:172.21.15.82:6792/0 conn(0x7ff0c40a67c0 legacy=0x7ff0c40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 >> v1:172.21.15.1:6790/0 conn(0x7ff0c413f5c0 legacy=0x7ff0c41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 monclient: found mon.noname-c 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c40a92a0 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: _finish_auth 0 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0b8001ae0 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c966e640 5 monclient: authenticate success, global_id 26401 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c966e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:37.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: dump: 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:37.993 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c966e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0b80048a0 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 1 -- 172.21.15.1:0/3730928141 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0b8004e10 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c966e640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c966e640 4 set_mon_vals no callback set 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.985+0000 7ff0c1ffb640 10 monclient: dump: 2024-08-19T09:52:37.994 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:37.995 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 10 monclient: shutdown 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 -- 172.21.15.1:0/3730928141 >> v1:172.21.15.1:6791/0 conn(0x7ff0c40a6ba0 legacy=0x7ff0c40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 -- 172.21.15.1:0/3730928141 shutdown_connections 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 -- 172.21.15.1:0/3730928141 >> 172.21.15.1:0/3730928141 conn(0x7ff0c40ac4b0 msgr2=0x7ff0c40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 -- 172.21.15.1:0/3730928141 shutdown_connections 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 -- 172.21.15.1:0/3730928141 wait complete. 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 10 monclient: build_initial_monmap 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 10 monclient: monmap: 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:37.997 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 librados: starting msgr at 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.986+0000 7ff0c966e640 1 librados: starting objecter 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 Processor -- start 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 -- start start 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 librados: setting wanted keys 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 librados: calling monclient init 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient: init 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient: _add_conns ranks=[5,7,6,3,8,2,1,0,4] 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient(hunting): picked mon.noname-h con 0x7ff0c40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient(hunting): picked mon.noname-i con 0x7ff0c40a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient(hunting): picked mon.noname-d con 0x7ff0c413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c413a900 con 0x7ff0c40a67c0 2024-08-19T09:52:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c40a9cc0 con 0x7ff0c413f5c0 2024-08-19T09:52:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c40a1700 con 0x7ff0c40a6ba0 2024-08-19T09:52:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c966e640 10 monclient(hunting): authenticate will time out at 814.063836s 2024-08-19T09:52:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c37fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ff0c413f5c0 0x7ff0c40a0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35570/0 (socket says 172.21.15.1:35570) 2024-08-19T09:52:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.987+0000 7ff0c37fe640 1 -- 172.21.15.1:0/3340648579 learned_addr learned my addr 172.21.15.1:0/3340648579 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c37fe640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.1:6792/0 conn(0x7ff0c413f5c0 legacy=0x7ff0c40a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 client.?.objecter ms_handle_connect 0x7ff0c413f5c0 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 client.?.objecter ms_handle_connect 0x7ff0c40a67c0 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c2ffd640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.82:6791/0 conn(0x7ff0c40a67c0 legacy=0x7ff0c409fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 client.?.objecter ms_handle_connect 0x7ff0c40a6ba0 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c27fc640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.82:6792/0 conn(0x7ff0c40a6ba0 legacy=0x7ff0c40a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1349913036 0 0) 0x7ff0c40a9cc0 con 0x7ff0c413f5c0 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): my global_id is 26385 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b0004eb0 con 0x7ff0c413f5c0 2024-08-19T09:52:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243241668 0 0) 0x7ff0c413a900 con 0x7ff0c40a67c0 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): my global_id is 26060 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0c40a9cc0 con 0x7ff0c40a67c0 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889578649 0 0) 0x7ff0c40a1700 con 0x7ff0c40a6ba0 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): my global_id is 134125 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0c413a900 con 0x7ff0c40a6ba0 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1383765944 0 0) 0x7ff0b0004eb0 con 0x7ff0c413f5c0 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0c40a1700 con 0x7ff0c413f5c0 2024-08-19T09:52:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0bc003990 con 0x7ff0c413f5c0 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): dump: 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 837061597 0 0) 0x7ff0c413a900 con 0x7ff0c40a6ba0 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0b0004eb0 con 0x7ff0c40a6ba0 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0b8004d90 con 0x7ff0c40a6ba0 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): dump: 2024-08-19T09:52:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 803703640 0 0) 0x7ff0c40a9cc0 con 0x7ff0c40a67c0 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0c413a900 con 0x7ff0c40a67c0 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0b4003610 con 0x7ff0c40a67c0 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient(hunting): dump: 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1189096573 0 0) 0x7ff0c40a1700 con 0x7ff0c413f5c0 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.82:6792/0 conn(0x7ff0c40a6ba0 legacy=0x7ff0c40a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.82:6791/0 conn(0x7ff0c40a67c0 legacy=0x7ff0c409fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 monclient: found mon.d 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c40a2260 con 0x7ff0c413f5c0 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: _finish_auth 0 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 5 monclient: authenticate success, global_id 26385 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 10 monclient: _renew_subs 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c41d7260 con 0x7ff0c413f5c0 2024-08-19T09:52:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 10 client.26385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 10 monclient: _renew_subs 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0c41d7610 con 0x7ff0c413f5c0 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c966e640 1 librados: init done 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0bc003e50 con 0x7ff0c413f5c0 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff0bc004340 con 0x7ff0c413f5c0 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0c17fa640 4 set_mon_vals no callback set 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.990+0000 7ff0ab7fe640 10 monclient: dump: 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.991+0000 7ff0c966e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.991+0000 7ff0c966e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.991+0000 7ff0c966e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.991+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a4005080 con 0x7ff0c413f5c0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.991+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff0bc0044e0 con 0x7ff0c413f5c0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.992+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(92..92 src has 1..92) ==== 8396+0+0 (unknown 3682579078 0 0) 0x7ff0bc04f040 con 0x7ff0c413f5c0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.992+0000 7ff0ab7fe640 10 client.26385.objecter ms_dispatch 0x7ff0c4000b90 osd_map(92..92 src has 1..92) 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.992+0000 7ff0ab7fe640 3 client.26385.objecter handle_osd_map got epochs [92,92] > 0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.992+0000 7ff0ab7fe640 3 client.26385.objecter handle_osd_map decoding full epoch 92 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.992+0000 7ff0c2ffd640 10 monclient: get_auth_request con 0x7ff0b00419a0 auth_method 0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.992+0000 7ff0ab7fe640 20 client.26385.objecter dump_active .. 0 homeless 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.994+0000 7ff0ab7fe640 10 client.26385.objecter ms_handle_connect 0x7ff0b00419a0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.996+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff0bc0177f0 con 0x7ff0c413f5c0 2024-08-19T09:52:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.996+0000 7ff0ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:37.996+0000 7ff0ab7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.249+0000 7ff0c966e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:52:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.249+0000 7ff0c966e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:52:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.249+0000 7ff0c966e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.250+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0a4005370 con 0x7ff0c413f5c0 2024-08-19T09:52:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.250+0000 7ff0ab7fe640 1 -- 172.21.15.1:0/3340648579 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 129807002) 0x7ff0bc0170f0 con 0x7ff0c413f5c0 2024-08-19T09:52:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.250+0000 7ff0ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:52:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.250+0000 7ff0ab7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:38.252 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 10 librados: watch_flush enter 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 10 librados: watch_flush exit 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 20 client.26385.objecter shutdown clearing up homeless session... 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 10 client.26385.objecter successfully canceled tick 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff0b00419a0 legacy=0x7ff0b0043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 10 monclient: shutdown 2024-08-19T09:52:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 >> v1:172.21.15.1:6792/0 conn(0x7ff0c413f5c0 legacy=0x7ff0c40a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 shutdown_connections 2024-08-19T09:52:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 >> 172.21.15.1:0/3340648579 conn(0x7ff0c40ac4b0 msgr2=0x7ff0c4138810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 shutdown_connections 2024-08-19T09:52:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 -- 172.21.15.1:0/3340648579 wait complete. 2024-08-19T09:52:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.252+0000 7ff0c966e640 1 librados: shutdown 2024-08-19T09:52:38.263 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:52:38.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:52:38.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:52:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.362+0000 7f46daa07640 10 monclient: get_monmap_and_config 2024-08-19T09:52:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.362+0000 7f46daa07640 10 monclient: build_initial_monmap 2024-08-19T09:52:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.362+0000 7f46daa07640 10 monclient: monmap: 2024-08-19T09:52:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:38.363167+0000 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:38.363167+0000 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:38.365 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:38.365 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:38.365 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.362+0000 7f46daa07640 1 Processor -- start 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 1 -- start start 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient: init 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient: _add_conns ranks=[3,6,5,4,7,1,8,2,0] 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient(hunting): picked mon.noname-g con 0x7f46d40a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient(hunting): picked mon.noname-d con 0x7f46d40694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient(hunting): picked mon.noname-h con 0x7f46d40a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d40a3c70 con 0x7f46d40a6830 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d40a33d0 con 0x7f46d40a46a0 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d40a36b0 con 0x7f46d40694b0 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46daa07640 10 monclient(hunting): authenticate will time out at 814.439603s 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46d37fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f46d40694b0 0x7f46d40a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35576/0 (socket says 172.21.15.1:35576) 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.363+0000 7f46d37fe640 1 -- 172.21.15.1:0/277054540 learned_addr learned my addr 172.21.15.1:0/277054540 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.365+0000 7f46d37fe640 1 -- 172.21.15.1:0/277054540 >> v1:172.21.15.1:6792/0 conn(0x7f46d40694b0 legacy=0x7f46d40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d8fa4640 1 -- 172.21.15.1:0/277054540 >> v1:172.21.15.82:6791/0 conn(0x7f46d40a46a0 legacy=0x7f46d4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d3fff640 1 -- 172.21.15.1:0/277054540 >> v1:172.21.15.82:6790/0 conn(0x7f46d40a6830 legacy=0x7f46d40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173107079 0 0) 0x7f46d40a36b0 con 0x7f46d40694b0 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient(hunting): my global_id is 26394 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46bc003520 con 0x7f46d40694b0 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3831394136 0 0) 0x7f46bc003520 con 0x7f46d40694b0 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 >> v1:172.21.15.82:6791/0 conn(0x7f46d40a46a0 legacy=0x7f46d4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 >> v1:172.21.15.82:6790/0 conn(0x7f46d40a6830 legacy=0x7f46d40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 monclient: found mon.noname-d 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d406dbb0 con 0x7f46d40694b0 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46daa07640 5 monclient: authenticate success, global_id 26394 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f46c8001ae0 con 0x7f46d40694b0 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: dump: 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46daa07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46c80048a0 con 0x7f46d40694b0 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 1 -- 172.21.15.1:0/277054540 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f46c8004dd0 con 0x7f46d40694b0 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46d2ffd640 10 monclient: dump: 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.376 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46daa07640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.366+0000 7f46daa07640 4 set_mon_vals no callback set 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 10 monclient: shutdown 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 1 -- 172.21.15.1:0/277054540 >> v1:172.21.15.1:6792/0 conn(0x7f46d40694b0 legacy=0x7f46d40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 1 -- 172.21.15.1:0/277054540 shutdown_connections 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 1 -- 172.21.15.1:0/277054540 >> 172.21.15.1:0/277054540 conn(0x7f46d40a9020 msgr2=0x7f46d4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 1 -- 172.21.15.1:0/277054540 shutdown_connections 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 1 -- 172.21.15.1:0/277054540 wait complete. 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 10 monclient: build_initial_monmap 2024-08-19T09:52:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.367+0000 7f46daa07640 10 monclient: monmap: 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 librados: starting msgr at 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 librados: starting objecter 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 Processor -- start 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 -- start start 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 librados: setting wanted keys 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 librados: calling monclient init 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient: init 2024-08-19T09:52:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient: _add_conns ranks=[3,0,2,5,8,1,7,4,6] 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient(hunting): picked mon.noname-g con 0x7f46d40694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient(hunting): picked mon.noname-a con 0x7f46d40a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient(hunting): picked mon.noname-b con 0x7f46d40a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d4142d60 con 0x7f46d40a46a0 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d413a950 con 0x7f46d40a6830 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d41cf260 con 0x7f46d40694b0 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46daa07640 10 monclient(hunting): authenticate will time out at 814.444882s 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46d37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f46d40a46a0 0x7f46d41ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58904/0 (socket says 172.21.15.1:58904) 2024-08-19T09:52:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.368+0000 7f46d37fe640 1 -- 172.21.15.1:0/299418348 learned_addr learned my addr 172.21.15.1:0/299418348 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d37fe640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46d40a46a0 legacy=0x7f46d41ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 client.?.objecter ms_handle_connect 0x7f46d40a46a0 2024-08-19T09:52:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 client.?.objecter ms_handle_connect 0x7f46d40a6830 2024-08-19T09:52:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d8fa4640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6790/0 conn(0x7f46d40a6830 legacy=0x7f46d41d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1867098564 0 0) 0x7f46d406dbb0 con 0x7f46d40a46a0 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): my global_id is 134118 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46c0004eb0 con 0x7f46d40a46a0 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d3fff640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.82:6790/0 conn(0x7f46d40694b0 legacy=0x7f46d41ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 client.?.objecter ms_handle_connect 0x7f46d40694b0 2024-08-19T09:52:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1407304791 0 0) 0x7f46c0004eb0 con 0x7f46d40a46a0 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46d406dbb0 con 0x7f46d40a46a0 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f46c8001d30 con 0x7f46d40a46a0 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3345100736 0 0) 0x7f46d406dbb0 con 0x7f46d40a46a0 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.82:6790/0 conn(0x7f46d40694b0 legacy=0x7f46d41ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6790/0 conn(0x7f46d40a6830 legacy=0x7f46d41d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 monclient: found mon.a 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d4138f50 con 0x7f46d40a46a0 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: _finish_auth 0 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:52:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 5 monclient: authenticate success, global_id 134118 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 10 monclient: _renew_subs 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46d4139cc0 con 0x7f46d40a46a0 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 10 client.134118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 10 monclient: _renew_subs 2024-08-19T09:52:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46d41379a0 con 0x7f46d40a46a0 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46daa07640 1 librados: init done 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46c80048a0 con 0x7f46d40a46a0 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f46c8005380 con 0x7f46d40a46a0 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d27fc640 4 set_mon_vals no callback set 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.371+0000 7f46d0ff9640 10 monclient: dump: 2024-08-19T09:52:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46daa07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f46b40065c0 con 0x7f46b4002860 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f46c80117e0 con 0x7f46d40a46a0 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(92..92 src has 1..92) ==== 8396+0+0 (unknown 3682579078 0 0) 0x7f46c804eea0 con 0x7f46d40a46a0 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46d0ff9640 10 client.134118.objecter ms_dispatch 0x7f46d4000b90 osd_map(92..92 src has 1..92) 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46d0ff9640 3 client.134118.objecter handle_osd_map got epochs [92,92] > 0 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.372+0000 7f46d0ff9640 3 client.134118.objecter handle_osd_map decoding full epoch 92 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.373+0000 7f46d0ff9640 20 client.134118.objecter dump_active .. 0 homeless 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.373+0000 7f46d8fa4640 10 monclient: get_auth_request con 0x7f46c0041000 auth_method 0 2024-08-19T09:52:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d3fff640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46b4002860 legacy=0x7f46b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 10 client.134118.objecter ms_handle_connect 0x7f46b4002860 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 10 client.134118.objecter resend_mon_ops 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1754673804 0 0) 0x7f46b40065c0 con 0x7f46b4002860 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 10 monclient(hunting): my global_id is 134127 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46d406dbb0 con 0x7f46b4002860 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.374+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3219264465 0 0) 0x7f46d406dbb0 con 0x7f46b4002860 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46b40065c0 con 0x7f46b4002860 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f46c4001b10 con 0x7f46b4002860 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 10 monclient: dump: 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.397 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3741045849 0 0) 0x7f46b40065c0 con 0x7f46b4002860 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46b400a160 con 0x7f46b4002860 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 10 client.134118.objecter ms_handle_connect 0x7f46c0041000 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f46c40067f0 con 0x7f46b4002860 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46b4002860 legacy=0x7f46b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.375+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46b4002860 legacy=0x7f46b4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.381+0000 7f46daa07640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.381+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f46b4009e70 con 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.383+0000 7f46d3fff640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46b4009130 legacy=0x7f46b40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.383+0000 7f46d0ff9640 10 client.134118.objecter ms_handle_connect 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.383+0000 7f46d0ff9640 10 client.134118.objecter resend_mon_ops 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1948087847 0 0) 0x7f46b4009e70 con 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient(hunting): my global_id is 134136 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b40065c0 con 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2419980734 0 0) 0x7f46b40065c0 con 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46b4009e70 con 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f46c4003d40 con 0x7f46b4009130 2024-08-19T09:52:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient: dump: 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2444024569 0 0) 0x7f46b4009e70 con 0x7f46b4009130 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46b400c410 con 0x7f46b4009130 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3801540288) 0x7f46c4006ac0 con 0x7f46b4009130 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46b4009130 legacy=0x7f46b40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d0ff9640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46b4009130 legacy=0x7f46b40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15446, 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:38.423 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 10 librados: watch_flush enter 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 10 librados: watch_flush exit 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 20 client.134118.objecter shutdown clearing up homeless session... 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 10 client.134118.objecter successfully canceled tick 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.82:6812/2549104415 conn(0x7f46c0041000 legacy=0x7f46c00434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 10 monclient: shutdown 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 >> v1:172.21.15.1:6789/0 conn(0x7f46d40a46a0 legacy=0x7f46d41ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.384+0000 7f46d8fa4640 1 -- 172.21.15.1:0/299418348 reap_dead start 2024-08-19T09:52:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.385+0000 7f46d8fa4640 1 -- 172.21.15.1:0/299418348 reap_dead start 2024-08-19T09:52:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.385+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 shutdown_connections 2024-08-19T09:52:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.385+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 >> 172.21.15.1:0/299418348 conn(0x7f46d40a9020 msgr2=0x7f46d40a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.385+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 shutdown_connections 2024-08-19T09:52:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.385+0000 7f46daa07640 1 -- 172.21.15.1:0/299418348 wait complete. 2024-08-19T09:52:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.385+0000 7f46daa07640 1 librados: shutdown 2024-08-19T09:52:38.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:38.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:52:38.427 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.525+0000 7f35ca7a0640 10 monclient: get_monmap_and_config 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.525+0000 7f35ca7a0640 10 monclient: build_initial_monmap 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.525+0000 7f35ca7a0640 10 monclient: monmap: 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:38.525954+0000 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:38.525954+0000 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:38.527 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.525+0000 7f35ca7a0640 1 Processor -- start 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.525+0000 7f35ca7a0640 1 -- start start 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.525+0000 7f35ca7a0640 10 monclient: init 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient: _add_conns ranks=[2,8,4,6,0,7,5,3,1] 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient(hunting): picked mon.noname-b con 0x7f35c406d8b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient(hunting): picked mon.noname-e con 0x7f35c40a5740 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient(hunting): picked mon.noname-c con 0x7f35c40a5b60 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c4136750 con 0x7f35c406d8b0 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c409dbf0 con 0x7f35c40a5b60 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c409ded0 con 0x7f35c40a5740 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35ca7a0640 10 monclient(hunting): authenticate will time out at 814.602336s 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35c8f9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f35c40a5740 0x7f35c413e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43574/0 (socket says 172.21.15.1:43574) 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35c8f9d640 1 -- 172.21.15.1:0/1005930526 learned_addr learned my addr 172.21.15.1:0/1005930526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35c979e640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f35c406d8b0 0x7f35c4136040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40504/0 (socket says 172.21.15.1:40504) 2024-08-19T09:52:38.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.526+0000 7f35c9f9f640 1 --1- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6791/0 conn(0x7f35c40a5b60 0x7f35c4141890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36444/0 (socket says 172.21.15.1:36444) 2024-08-19T09:52:38.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.528+0000 7f35c9f9f640 1 -- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6791/0 conn(0x7f35c40a5b60 legacy=0x7f35c4141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.528+0000 7f35c8f9d640 1 -- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6793/0 conn(0x7f35c40a5740 legacy=0x7f35c413e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.528+0000 7f35c979e640 1 -- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6790/0 conn(0x7f35c406d8b0 legacy=0x7f35c4136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121818067 0 0) 0x7f35c4136750 con 0x7f35c406d8b0 2024-08-19T09:52:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): my global_id is 26480 2024-08-19T09:52:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35ac003520 con 0x7f35c406d8b0 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984023663 0 0) 0x7f35c409ded0 con 0x7f35c40a5740 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 10 monclient(hunting): my global_id is 26873 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.881+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c4136750 con 0x7f35c40a5740 2024-08-19T09:52:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3058689281 0 0) 0x7f35c4136750 con 0x7f35c40a5740 2024-08-19T09:52:38.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6791/0 conn(0x7f35c40a5b60 legacy=0x7f35c4141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6790/0 conn(0x7f35c406d8b0 legacy=0x7f35c4136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 1 monclient: found mon.noname-e 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c409e1b0 con 0x7f35c40a5740 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 10 monclient: _finish_auth 0 2024-08-19T09:52:38.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.882+0000 7f35c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.884+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f35b80019f0 con 0x7f35c40a5740 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.884+0000 7f35c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.884+0000 7f35c3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.884+0000 7f35c3fff640 10 monclient: dump: 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.888 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35ca7a0640 5 monclient: authenticate success, global_id 26873 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35ca7a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35b80048a0 con 0x7f35c40a5740 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35c3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35ca7a0640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35ca7a0640 4 set_mon_vals no callback set 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35c3fff640 1 -- 172.21.15.1:0/1005930526 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f35b8004d90 con 0x7f35c40a5740 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35c3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.885+0000 7f35c3fff640 10 monclient: dump: 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.887+0000 7f35ca7a0640 10 monclient: shutdown 2024-08-19T09:52:38.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.887+0000 7f35ca7a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:38.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.887+0000 7f35ca7a0640 1 -- 172.21.15.1:0/1005930526 >> v1:172.21.15.1:6793/0 conn(0x7f35c40a5740 legacy=0x7f35c413e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.894+0000 7f35ca7a0640 1 -- 172.21.15.1:0/1005930526 shutdown_connections 2024-08-19T09:52:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.894+0000 7f35ca7a0640 1 -- 172.21.15.1:0/1005930526 >> 172.21.15.1:0/1005930526 conn(0x7f35c40ac090 msgr2=0x7f35c40a8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.896+0000 7f35ca7a0640 1 -- 172.21.15.1:0/1005930526 shutdown_connections 2024-08-19T09:52:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.897+0000 7f35ca7a0640 1 -- 172.21.15.1:0/1005930526 wait complete. 2024-08-19T09:52:38.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.898+0000 7f35ca7a0640 10 monclient: build_initial_monmap 2024-08-19T09:52:38.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.898+0000 7f35ca7a0640 10 monclient: monmap: 2024-08-19T09:52:38.904 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.899+0000 7f35ca7a0640 1 librados: starting msgr at 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.899+0000 7f35ca7a0640 1 librados: starting objecter 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.899+0000 7f35ca7a0640 1 Processor -- start 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.900+0000 7f35ca7a0640 1 -- start start 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.900+0000 7f35ca7a0640 1 librados: setting wanted keys 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.900+0000 7f35ca7a0640 1 librados: calling monclient init 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.900+0000 7f35ca7a0640 10 monclient: init 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.901+0000 7f35ca7a0640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.902+0000 7f35ca7a0640 10 monclient: _add_conns ranks=[4,3,6,0,5,2,8,1,7] 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.902+0000 7f35ca7a0640 10 monclient(hunting): picked mon.noname-c con 0x7f35c406d8b0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.902+0000 7f35c979e640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f35c406d8b0 0x7f35c4141ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36446/0 (socket says 172.21.15.1:36446) 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.902+0000 7f35c979e640 1 -- 172.21.15.1:0/818577949 learned_addr learned my addr 172.21.15.1:0/818577949 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.902+0000 7f35ca7a0640 10 monclient(hunting): picked mon.noname-g con 0x7f35c40a5740 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.903+0000 7f35ca7a0640 10 monclient(hunting): picked mon.noname-d con 0x7f35c40a5b60 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.903+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c4142cc0 con 0x7f35c40a5740 2024-08-19T09:52:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.903+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c40aa560 con 0x7f35c406d8b0 2024-08-19T09:52:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.903+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c412e280 con 0x7f35c40a5b60 2024-08-19T09:52:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.903+0000 7f35ca7a0640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.903+0000 7f35ca7a0640 10 monclient(hunting): authenticate will time out at 814.979595s 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.905+0000 7f35c9f9f640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6792/0 conn(0x7f35c40a5b60 legacy=0x7f35c41d6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.905+0000 7f35c1ffb640 10 client.?.objecter ms_handle_connect 0x7f35c40a5b60 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.905+0000 7f35c1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.905+0000 7f35c979e640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6791/0 conn(0x7f35c406d8b0 legacy=0x7f35c4141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.905+0000 7f35c1ffb640 10 client.?.objecter ms_handle_connect 0x7f35c406d8b0 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.905+0000 7f35c1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c8f9d640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.82:6790/0 conn(0x7f35c40a5740 legacy=0x7f35c41425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 10 client.?.objecter ms_handle_connect 0x7f35c40a5740 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2754339888 0 0) 0x7f35c412e280 con 0x7f35c40a5b60 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 10 monclient(hunting): my global_id is 26403 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.906+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b0004eb0 con 0x7f35c40a5b60 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.907+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3525453170 0 0) 0x7f35b0004eb0 con 0x7f35c40a5b60 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.907+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35c412e280 con 0x7f35c40a5b60 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.907+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f35b4001f60 con 0x7f35c40a5b60 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.907+0000 7f35c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.907+0000 7f35c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.907+0000 7f35c1ffb640 10 monclient(hunting): dump: 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2656948592 0 0) 0x7f35c412e280 con 0x7f35c40a5b60 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6791/0 conn(0x7f35c406d8b0 legacy=0x7f35c4141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.82:6790/0 conn(0x7f35c40a5740 legacy=0x7f35c41425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 1 monclient: found mon.d 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c4130000 con 0x7f35c40a5b60 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 10 monclient: _finish_auth 0 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35ca7a0640 5 monclient: authenticate success, global_id 26403 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35ca7a0640 10 monclient: _renew_subs 2024-08-19T09:52:38.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35ca7a0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35c412d320 con 0x7f35c40a5b60 2024-08-19T09:52:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35b4004e10 con 0x7f35c40a5b60 2024-08-19T09:52:38.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35ca7a0640 10 client.26403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.908+0000 7f35c37fe640 4 set_mon_vals no callback set 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f35b4005300 con 0x7f35c40a5b60 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35c1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35c1ffb640 10 monclient: dump: 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35ca7a0640 10 monclient: _renew_subs 2024-08-19T09:52:38.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f35b40055a0 con 0x7f35c40a5b60 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35ca7a0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f35c412f490 con 0x7f35c40a5b60 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.909+0000 7f35ca7a0640 1 librados: init done 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(93..93 src has 1..93) ==== 8409+0+0 (unknown 1238932098 0 0) 0x7f35b404eff0 con 0x7f35c40a5b60 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35c1ffb640 10 client.26403.objecter ms_dispatch 0x7f35c4000b90 osd_map(93..93 src has 1..93) 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35c1ffb640 3 client.26403.objecter handle_osd_map got epochs [93,93] > 0 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35c1ffb640 3 client.26403.objecter handle_osd_map decoding full epoch 93 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35c979e640 10 monclient: get_auth_request con 0x7f35b0040fa0 auth_method 0 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35c1ffb640 20 client.26403.objecter dump_active .. 0 homeless 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.910+0000 7f35ca7a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.911+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35c412e450 con 0x7f35c402e340 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.912+0000 7f35c1ffb640 10 client.26403.objecter ms_handle_connect 0x7f35b0040fa0 2024-08-19T09:52:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c8f9d640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6790/0 conn(0x7f35c402e340 legacy=0x7f35c4063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 client.26403.objecter ms_handle_connect 0x7f35c402e340 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 client.26403.objecter resend_mon_ops 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453814199 0 0) 0x7f35c412e450 con 0x7f35c402e340 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient(hunting): my global_id is 26489 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c412e280 con 0x7f35c402e340 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 726074532 0 0) 0x7f35c412e280 con 0x7f35c402e340 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35c412e450 con 0x7f35c402e340 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f35b80019f0 con 0x7f35c402e340 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.913+0000 7f35c1ffb640 10 monclient: dump: 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3832275624 0 0) 0x7f35c412e450 con 0x7f35c402e340 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35c402e720 con 0x7f35c402e340 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f35b80047e0 con 0x7f35c402e340 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6790/0 conn(0x7f35c402e340 legacy=0x7f35c4063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.914+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6790/0 conn(0x7f35c402e340 legacy=0x7f35c4063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.921+0000 7f35ca7a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.921+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35c4069c70 con 0x7f35c41379c0 2024-08-19T09:52:38.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c8f9d640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6790/0 conn(0x7f35c41379c0 legacy=0x7f35c41dbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:38.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 client.26403.objecter ms_handle_connect 0x7f35c41379c0 2024-08-19T09:52:38.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 client.26403.objecter resend_mon_ops 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4211105155 0 0) 0x7f35c4069c70 con 0x7f35c41379c0 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient(hunting): my global_id is 26498 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c412e450 con 0x7f35c41379c0 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2421917184 0 0) 0x7f35c412e450 con 0x7f35c41379c0 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35c4069c70 con 0x7f35c41379c0 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f35b8005050 con 0x7f35c41379c0 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.923+0000 7f35c1ffb640 10 monclient: dump: 2024-08-19T09:52:38.926 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 53924370 0 0) 0x7f35c4069c70 con 0x7f35c41379c0 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35c4064190 con 0x7f35c41379c0 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1872039317) 0x7f35b8007e10 con 0x7f35c41379c0 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6790/0 conn(0x7f35c41379c0 legacy=0x7f35c41dbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c1ffb640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6790/0 conn(0x7f35c41379c0 legacy=0x7f35c41dbc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:38.928 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 165214, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:38.929 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:38.930 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:38.931 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:38.932 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.934 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:38.936 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:38.939 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.940 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.941 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:38.942 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.943 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.944 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:38.945 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:38.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:38.947 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:38.948 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 10 librados: watch_flush enter 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 10 librados: watch_flush exit 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 20 client.26403.objecter shutdown clearing up homeless session... 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 10 client.26403.objecter successfully canceled tick 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.82:6812/2549104415 conn(0x7f35b0040fa0 legacy=0x7f35b0043460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 10 monclient: shutdown 2024-08-19T09:52:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 >> v1:172.21.15.1:6792/0 conn(0x7f35c40a5b60 legacy=0x7f35c41d6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c9f9f640 1 -- 172.21.15.1:0/818577949 reap_dead start 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35c9f9f640 1 -- 172.21.15.1:0/818577949 reap_dead start 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 shutdown_connections 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 >> 172.21.15.1:0/818577949 conn(0x7f35c40ac090 msgr2=0x7f35c40ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 shutdown_connections 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.924+0000 7f35ca7a0640 1 -- 172.21.15.1:0/818577949 wait complete. 2024-08-19T09:52:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:38.925+0000 7f35ca7a0640 1 librados: shutdown 2024-08-19T09:52:38.951 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:38.951 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:52:38.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:52:39.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.050+0000 7feda6fc9640 10 monclient: get_monmap_and_config 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.050+0000 7feda6fc9640 10 monclient: build_initial_monmap 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.050+0000 7feda6fc9640 10 monclient: monmap: 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:39.051805+0000 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:39.051805+0000 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:39.052 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:39.053 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:39.053 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:39.053 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:39.053 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:39.053 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.051+0000 7feda6fc9640 1 Processor -- start 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.051+0000 7feda6fc9640 1 -- start start 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.051+0000 7feda6fc9640 10 monclient: init 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient: _add_conns ranks=[7,6,0,4,2,1,5,3,8] 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient(hunting): picked mon.noname-i con 0x7feda00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:39.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient(hunting): picked mon.noname-d con 0x7feda00a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient(hunting): picked mon.noname-a con 0x7feda013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda00a3b40 con 0x7feda013f5c0 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda00a8130 con 0x7feda00a6ba0 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda006ede0 con 0x7feda00a67c0 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda6fc9640 10 monclient(hunting): authenticate will time out at 815.128154s 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7fed9ffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7feda00a6ba0 0x7feda00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35594/0 (socket says 172.21.15.1:35594) 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7fed9ffff640 1 -- 172.21.15.1:0/1789602556 learned_addr learned my addr 172.21.15.1:0/1789602556 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:39.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.052+0000 7feda5566640 1 --1- 172.21.15.1:0/1789602556 >> v1:172.21.15.1:6789/0 conn(0x7feda013f5c0 0x7feda01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58944/0 (socket says 172.21.15.1:58944) 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7feda5566640 1 -- 172.21.15.1:0/1789602556 >> v1:172.21.15.1:6789/0 conn(0x7feda013f5c0 legacy=0x7feda01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9ffff640 1 -- 172.21.15.1:0/1789602556 >> v1:172.21.15.1:6792/0 conn(0x7feda00a6ba0 legacy=0x7feda00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7feda4d65640 1 -- 172.21.15.1:0/1789602556 >> v1:172.21.15.82:6792/0 conn(0x7feda00a67c0 legacy=0x7feda00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 52192355 0 0) 0x7feda00a3b40 con 0x7feda013f5c0 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): my global_id is 134145 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed90003520 con 0x7feda013f5c0 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612006025 0 0) 0x7feda00a8130 con 0x7feda00a6ba0 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): my global_id is 26412 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda00a3b40 con 0x7feda00a6ba0 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2013773842 0 0) 0x7feda006ede0 con 0x7feda00a67c0 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient(hunting): my global_id is 134134 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda00a8130 con 0x7feda00a67c0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 725680670 0 0) 0x7feda00a3b40 con 0x7feda00a6ba0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 >> v1:172.21.15.82:6792/0 conn(0x7feda00a67c0 legacy=0x7feda00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 >> v1:172.21.15.1:6789/0 conn(0x7feda013f5c0 legacy=0x7feda01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 monclient: found mon.noname-d 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda00a92a0 con 0x7feda00a6ba0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: _finish_auth 0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed8c001ac0 con 0x7feda00a6ba0 2024-08-19T09:52:39.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7fed9f7fe640 10 monclient: dump: 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7feda6fc9640 5 monclient: authenticate success, global_id 26412 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.054+0000 7feda6fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed8c0048a0 con 0x7feda00a6ba0 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7fed9f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7fed9f7fe640 1 -- 172.21.15.1:0/1789602556 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed8c004e10 con 0x7feda00a6ba0 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 4 set_mon_vals no callback set 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7fed9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:39.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7fed9f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7fed9f7fe640 10 monclient: dump: 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.064 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 10 monclient: shutdown 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 1 -- 172.21.15.1:0/1789602556 >> v1:172.21.15.1:6792/0 conn(0x7feda00a6ba0 legacy=0x7feda00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 1 -- 172.21.15.1:0/1789602556 shutdown_connections 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 1 -- 172.21.15.1:0/1789602556 >> 172.21.15.1:0/1789602556 conn(0x7feda00ac4b0 msgr2=0x7feda00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 1 -- 172.21.15.1:0/1789602556 shutdown_connections 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.055+0000 7feda6fc9640 1 -- 172.21.15.1:0/1789602556 wait complete. 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 10 monclient: build_initial_monmap 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 10 monclient: monmap: 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:39.065 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:39.066 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 1 librados: starting msgr at 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 1 librados: starting objecter 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 1 Processor -- start 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 1 -- start start 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 1 librados: setting wanted keys 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 1 librados: calling monclient init 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.056+0000 7feda6fc9640 10 monclient: init 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient: _add_conns ranks=[1,5,7,6,2,4,3,8,0] 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient(hunting): picked mon.noname-f con 0x7feda00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient(hunting): picked mon.noname-h con 0x7feda00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient(hunting): picked mon.noname-i con 0x7feda013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:39.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda013b720 con 0x7feda00a67c0 2024-08-19T09:52:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda013cae0 con 0x7feda00a6ba0 2024-08-19T09:52:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda013a7b0 con 0x7feda013f5c0 2024-08-19T09:52:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7feda6fc9640 10 monclient(hunting): authenticate will time out at 815.133110s 2024-08-19T09:52:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7fed9ffff640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7feda00a6ba0 0x7feda013a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:46032/0 (socket says 172.21.15.1:46032) 2024-08-19T09:52:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.057+0000 7fed9ffff640 1 -- 172.21.15.1:0/66511281 learned_addr learned my addr 172.21.15.1:0/66511281 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9ffff640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6791/0 conn(0x7feda00a6ba0 legacy=0x7feda013a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7feda5566640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6792/0 conn(0x7feda013f5c0 legacy=0x7feda01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 client.?.objecter ms_handle_connect 0x7feda00a6ba0 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 client.?.objecter ms_handle_connect 0x7feda013f5c0 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 client.?.objecter ms_handle_connect 0x7feda00a67c0 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7feda4d65640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6789/0 conn(0x7feda00a67c0 legacy=0x7feda0139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65440725 0 0) 0x7feda013cae0 con 0x7feda00a6ba0 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): my global_id is 26069 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed84004eb0 con 0x7feda00a6ba0 2024-08-19T09:52:39.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071216947 0 0) 0x7feda013a7b0 con 0x7feda013f5c0 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): my global_id is 134143 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda013cae0 con 0x7feda013f5c0 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573725125 0 0) 0x7feda013b720 con 0x7feda00a67c0 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 10 monclient(hunting): my global_id is 27838 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.059+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda013a7b0 con 0x7feda00a67c0 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3851676494 0 0) 0x7feda013cae0 con 0x7feda013f5c0 2024-08-19T09:52:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feda013b720 con 0x7feda013f5c0 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed980020b0 con 0x7feda013f5c0 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): dump: 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.080 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2891364341 0 0) 0x7fed84004eb0 con 0x7feda00a6ba0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feda013cae0 con 0x7feda00a6ba0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 833293792 0 0) 0x7feda013a7b0 con 0x7feda00a67c0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed84004eb0 con 0x7feda00a67c0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed8c004d90 con 0x7feda00a6ba0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:39.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): dump: 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.082 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed94003610 con 0x7feda00a67c0 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): dump: 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.083 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2530772446 0 0) 0x7feda013b720 con 0x7feda013f5c0 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6791/0 conn(0x7feda00a6ba0 legacy=0x7feda013a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6789/0 conn(0x7feda00a67c0 legacy=0x7feda0139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 monclient: found mon.i 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda00a2a60 con 0x7feda013f5c0 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: _finish_auth 0 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 5 monclient: authenticate success, global_id 134143 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 10 monclient: _renew_subs 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feda013c800 con 0x7feda013f5c0 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 10 client.134143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 10 monclient: _renew_subs 2024-08-19T09:52:39.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7feda013f2a0 con 0x7feda013f5c0 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda6fc9640 1 librados: init done 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda5566640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6792/0 conn(0x7feda013f5c0 legacy=0x7feda01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda5566640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6792/0 conn(0x7feda013f5c0 legacy=0x7feda01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda5566640 1 --1- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6792/0 conn(0x7feda013f5c0 0x7feda01d6250 :-1 s=OPENED pgs=84 cs=1 l=1).handle_message read tag failed 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7feda5566640 1 --1- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6792/0 conn(0x7feda013f5c0 0x7feda01d6250 :-1 s=OPENED pgs=84 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.82:6792/0 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6792/0 conn(0x7feda013f5c0 legacy=0x7feda01d6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient: _add_conns ranks=[2,1,5,8,3,0,4,7,6] 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): picked mon.b con 0x7fed84007db0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): picked mon.f con 0x7feda00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): picked mon.h con 0x7feda00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feda013b720 con 0x7feda00a67c0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feda013a7b0 con 0x7fed84007db0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fed84014f90 con 0x7feda00a6ba0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.060+0000 7fed9d7fa640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.061+0000 7feda6fc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.061+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feda00a9ca0 con 0x7feda0139990 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 client.134143.objecter ms_handle_connect 0x7fed84007db0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7feda4d65640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6790/0 conn(0x7fed84007db0 legacy=0x7fed8400a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 client.134143.objecter resend_mon_ops 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 client.134143.objecter ms_handle_connect 0x7feda00a6ba0 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7feda5566640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6791/0 conn(0x7feda00a6ba0 legacy=0x7fed840113f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 client.134143.objecter resend_mon_ops 2024-08-19T09:52:39.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 client.134143.objecter ms_handle_connect 0x7feda00a67c0 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 client.134143.objecter resend_mon_ops 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9ffff640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6789/0 conn(0x7feda00a67c0 legacy=0x7fed8400d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552804090 0 0) 0x7feda013a7b0 con 0x7fed84007db0 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fed84015160 con 0x7fed84007db0 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2901768896 0 0) 0x7feda013b720 con 0x7feda00a67c0 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7feda013a7b0 con 0x7feda00a67c0 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093158936 0 0) 0x7fed84014f90 con 0x7feda00a6ba0 2024-08-19T09:52:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.062+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7feda013b720 con 0x7feda00a6ba0 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1359935306 0 0) 0x7feda013a7b0 con 0x7feda00a67c0 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6791/0 conn(0x7feda00a6ba0 legacy=0x7fed840113f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6790/0 conn(0x7fed84007db0 legacy=0x7fed8400a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 monclient: found mon.f 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fed840163b0 con 0x7feda00a67c0 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: _finish_auth 0 2024-08-19T09:52:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed8c0048a0 con 0x7feda00a67c0 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: dump: 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed8c001d10 con 0x7feda00a67c0 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9effd640 4 set_mon_vals no callback set 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 client.134143.objecter ms_handle_connect 0x7feda0139990 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 client.134143.objecter resend_mon_ops 2024-08-19T09:52:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7feda4d65640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda0139990 legacy=0x7feda0139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4237938794 0 0) 0x7feda00a9ca0 con 0x7feda0139990 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 10 monclient(hunting): my global_id is 26410 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.063+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda013a7b0 con 0x7feda0139990 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3477535807 0 0) 0x7feda013a7b0 con 0x7feda0139990 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feda00a9ca0 con 0x7feda0139990 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed94004340 con 0x7feda0139990 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 10 monclient: dump: 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.093 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fed8c011750 con 0x7feda00a67c0 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.1 v1:172.21.15.82:6789/0 6 ==== osd_map(93..93 src has 1..93) ==== 8409+0+0 (unknown 1238932098 0 0) 0x7fed8c04ebe0 con 0x7feda00a67c0 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 10 client.134143.objecter ms_dispatch 0x7feda0000b90 osd_map(93..93 src has 1..93) 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 3 client.134143.objecter handle_osd_map got epochs [93,93] > 0 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 3 client.134143.objecter handle_osd_map decoding full epoch 93 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 20 client.134143.objecter dump_active .. 0 homeless 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3155891749 0 0) 0x7feda00a9ca0 con 0x7feda0139990 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feda0142bb0 con 0x7feda0139990 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7feda5566640 10 monclient: get_auth_request con 0x7fed8404d5e0 auth_method 0 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fed940059c0 con 0x7feda0139990 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda0139990 legacy=0x7feda0139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.064+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda0139990 legacy=0x7feda0139d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.066+0000 7fed9d7fa640 10 client.134143.objecter ms_handle_connect 0x7fed8404d5e0 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.071+0000 7feda6fc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.071+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feda0142390 con 0x7feda002d1c0 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7feda4d65640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda002d1c0 legacy=0x7feda002d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 client.134143.objecter ms_handle_connect 0x7feda002d1c0 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 client.134143.objecter resend_mon_ops 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593727999 0 0) 0x7fed84004eb0 con 0x7feda002d1c0 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient(hunting): my global_id is 26419 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda00a9ca0 con 0x7feda002d1c0 2024-08-19T09:52:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1926517192 0 0) 0x7feda00a9ca0 con 0x7feda002d1c0 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed84004eb0 con 0x7feda002d1c0 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fed940061f0 con 0x7feda002d1c0 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7fed9d7fa640 10 monclient: dump: 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.073+0000 7feda4d65640 1 --1- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda002d1c0 0x7feda002d5a0 :-1 s=OPENED pgs=3122 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fed84004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:39.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.876+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4083069157 0 0) 0x7fed84004eb0 con 0x7feda002d1c0 2024-08-19T09:52:39.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.876+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feda013c3b0 con 0x7feda002d1c0 2024-08-19T09:52:39.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3935938561) 0x7fed94008570 con 0x7feda002d1c0 2024-08-19T09:52:39.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7fed9d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:39.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7fed9d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:39.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda002d1c0 legacy=0x7feda002d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7fed9d7fa640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.1:6791/0 conn(0x7feda002d1c0 legacy=0x7feda002d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:39.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda5566640 1 -- 172.21.15.1:0/66511281 reap_dead start 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 166147, 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:39.879 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:39.880 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:39.881 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.882 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:39.883 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:39.884 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:39.886 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.888 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.890 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.893 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:39.896 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:39.898 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:39.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:39.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:39.899 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:39.899 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 10 librados: watch_flush enter 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 10 librados: watch_flush exit 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 20 client.134143.objecter shutdown clearing up homeless session... 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 10 client.134143.objecter successfully canceled tick 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6812/2549104415 conn(0x7fed8404d5e0 legacy=0x7fed8404faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 10 monclient: shutdown 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 >> v1:172.21.15.82:6789/0 conn(0x7feda00a67c0 legacy=0x7fed8400d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 shutdown_connections 2024-08-19T09:52:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 >> 172.21.15.1:0/66511281 conn(0x7feda00ac4b0 msgr2=0x7feda0141c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 shutdown_connections 2024-08-19T09:52:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.877+0000 7feda6fc9640 1 -- 172.21.15.1:0/66511281 wait complete. 2024-08-19T09:52:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:39.878+0000 7feda6fc9640 1 librados: shutdown 2024-08-19T09:52:39.910 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:39.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:52:39.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.023+0000 7ff137b08640 10 monclient: get_monmap_and_config 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.023+0000 7ff137b08640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.023+0000 7ff137b08640 10 monclient: monmap: 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:40.024814+0000 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:40.024814+0000 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.026 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.024+0000 7ff137b08640 1 Processor -- start 2024-08-19T09:52:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.024+0000 7ff137b08640 1 -- start start 2024-08-19T09:52:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.024+0000 7ff137b08640 10 monclient: init 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient: _add_conns ranks=[7,6,8,2,5,0,3,4,1] 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient(hunting): picked mon.noname-i con 0x7ff1300a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient(hunting): picked mon.noname-d con 0x7ff1300a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient(hunting): picked mon.noname-e con 0x7ff13013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1300a3b40 con 0x7ff1300a6ba0 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1300a8130 con 0x7ff1300a67c0 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13006ede0 con 0x7ff13013f5c0 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff137b08640 10 monclient(hunting): authenticate will time out at 816.101190s 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff1360a5640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff13013f5c0 0x7ff1301419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43580/0 (socket says 172.21.15.1:43580) 2024-08-19T09:52:40.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff1360a5640 1 -- 172.21.15.1:0/4023953245 learned_addr learned my addr 172.21.15.1:0/4023953245 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.025+0000 7ff1350a3640 1 --1- 172.21.15.1:0/4023953245 >> v1:172.21.15.1:6792/0 conn(0x7ff1300a6ba0 0x7ff1300a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35600/0 (socket says 172.21.15.1:35600) 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1360a5640 1 -- 172.21.15.1:0/4023953245 >> v1:172.21.15.1:6793/0 conn(0x7ff13013f5c0 legacy=0x7ff1301419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1350a3640 1 -- 172.21.15.1:0/4023953245 >> v1:172.21.15.1:6792/0 conn(0x7ff1300a6ba0 legacy=0x7ff1300a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1358a4640 1 -- 172.21.15.1:0/4023953245 >> v1:172.21.15.82:6792/0 conn(0x7ff1300a67c0 legacy=0x7ff1300a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 753060255 0 0) 0x7ff13006ede0 con 0x7ff13013f5c0 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): my global_id is 26891 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff11c003520 con 0x7ff13013f5c0 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253917075 0 0) 0x7ff1300a3b40 con 0x7ff1300a6ba0 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): my global_id is 26430 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff13006ede0 con 0x7ff1300a6ba0 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1745458877 0 0) 0x7ff1300a8130 con 0x7ff1300a67c0 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient(hunting): my global_id is 134179 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1300a3b40 con 0x7ff1300a67c0 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4094877885 0 0) 0x7ff1300a3b40 con 0x7ff1300a67c0 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 >> v1:172.21.15.1:6793/0 conn(0x7ff13013f5c0 legacy=0x7ff1301419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 >> v1:172.21.15.1:6792/0 conn(0x7ff1300a6ba0 legacy=0x7ff1300a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 monclient: found mon.noname-i 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1300a92a0 con 0x7ff1300a67c0 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff12c001980 con 0x7ff1300a67c0 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff137b08640 5 monclient: authenticate success, global_id 134179 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff1348a2640 10 monclient: dump: 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.036 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.037 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.037 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.037 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.027+0000 7ff137b08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff12c001bd0 con 0x7ff1300a67c0 2024-08-19T09:52:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff1348a2640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff1348a2640 1 -- 172.21.15.1:0/4023953245 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff12c003950 con 0x7ff1300a67c0 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 4 set_mon_vals no callback set 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff1348a2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff1348a2640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff1348a2640 10 monclient: dump: 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.039 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.040 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 10 monclient: shutdown 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 1 -- 172.21.15.1:0/4023953245 >> v1:172.21.15.82:6792/0 conn(0x7ff1300a67c0 legacy=0x7ff1300a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 1 -- 172.21.15.1:0/4023953245 shutdown_connections 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 1 -- 172.21.15.1:0/4023953245 >> 172.21.15.1:0/4023953245 conn(0x7ff1300ac4b0 msgr2=0x7ff1300a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 1 -- 172.21.15.1:0/4023953245 shutdown_connections 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.028+0000 7ff137b08640 1 -- 172.21.15.1:0/4023953245 wait complete. 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient: monmap: 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.042 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 librados: starting msgr at 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 librados: starting objecter 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 Processor -- start 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 -- start start 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 librados: setting wanted keys 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 librados: calling monclient init 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient: init 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient: _add_conns ranks=[0,3,8,4,5,7,2,6,1] 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient(hunting): picked mon.noname-a con 0x7ff1300a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient(hunting): picked mon.noname-g con 0x7ff1300a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient(hunting): picked mon.noname-e con 0x7ff13013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13013d150 con 0x7ff1300a67c0 2024-08-19T09:52:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13013d430 con 0x7ff1300a6ba0 2024-08-19T09:52:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1301cb0c0 con 0x7ff13013f5c0 2024-08-19T09:52:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.029+0000 7ff137b08640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.030+0000 7ff137b08640 10 monclient(hunting): authenticate will time out at 816.106011s 2024-08-19T09:52:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.030+0000 7ff1360a5640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff13013f5c0 0x7ff1301d5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43584/0 (socket says 172.21.15.1:43584) 2024-08-19T09:52:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.030+0000 7ff1360a5640 1 -- 172.21.15.1:0/4025062824 learned_addr learned my addr 172.21.15.1:0/4025062824 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1360a5640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6793/0 conn(0x7ff13013f5c0 legacy=0x7ff1301d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 client.?.objecter ms_handle_connect 0x7ff13013f5c0 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 client.?.objecter ms_handle_connect 0x7ff1300a67c0 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1358a4640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6789/0 conn(0x7ff1300a67c0 legacy=0x7ff1301ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1350a3640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.82:6790/0 conn(0x7ff1300a6ba0 legacy=0x7ff1301ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 client.?.objecter ms_handle_connect 0x7ff1300a6ba0 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2602295817 0 0) 0x7ff13013d150 con 0x7ff1300a67c0 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): my global_id is 134154 2024-08-19T09:52:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff114004eb0 con 0x7ff1300a67c0 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134164034 0 0) 0x7ff1301cb0c0 con 0x7ff13013f5c0 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): my global_id is 26900 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff13013d150 con 0x7ff13013f5c0 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215900884 0 0) 0x7ff13013d430 con 0x7ff1300a6ba0 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): my global_id is 134130 2024-08-19T09:52:40.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1301cb0c0 con 0x7ff1300a6ba0 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1610214606 0 0) 0x7ff114004eb0 con 0x7ff1300a67c0 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff13013d430 con 0x7ff1300a67c0 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff12c003950 con 0x7ff1300a67c0 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): dump: 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3923245821 0 0) 0x7ff13013d150 con 0x7ff13013f5c0 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff114004eb0 con 0x7ff13013f5c0 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff128004b60 con 0x7ff13013f5c0 2024-08-19T09:52:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient(hunting): dump: 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2006411278 0 0) 0x7ff13013d430 con 0x7ff1300a67c0 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6793/0 conn(0x7ff13013f5c0 legacy=0x7ff1301d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.82:6790/0 conn(0x7ff1300a6ba0 legacy=0x7ff1301ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 monclient: found mon.a 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13013c130 con 0x7ff1300a67c0 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff1267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff137b08640 5 monclient: authenticate success, global_id 134154 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff137b08640 10 monclient: _renew_subs 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff137b08640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.032+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1300a9fc0 con 0x7ff1300a67c0 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 10 client.134154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:40.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 10 monclient: _renew_subs 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1301d72a0 con 0x7ff1300a67c0 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 1 librados: init done 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff12c001bd0 con 0x7ff1300a67c0 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff12c004280 con 0x7ff1300a67c0 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff127fff640 4 set_mon_vals no callback set 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 10 monclient: dump: 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.058 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.059 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1180065c0 con 0x7ff118002860 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.033+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff12c011a80 con 0x7ff1300a67c0 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.034+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(94..94 src has 1..94) ==== 7687+0+0 (unknown 3826229206 0 0) 0x7ff12c04e950 con 0x7ff1300a67c0 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.034+0000 7ff1267fc640 10 client.134154.objecter ms_dispatch 0x7ff130000b90 osd_map(94..94 src has 1..94) 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.034+0000 7ff1267fc640 3 client.134154.objecter handle_osd_map got epochs [94,94] > 0 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.034+0000 7ff1267fc640 3 client.134154.objecter handle_osd_map decoding full epoch 94 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.034+0000 7ff1267fc640 20 client.134154.objecter dump_active .. 0 homeless 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.034+0000 7ff1360a5640 10 monclient: get_auth_request con 0x7ff114041450 auth_method 0 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.035+0000 7ff1267fc640 10 client.134154.objecter ms_handle_connect 0x7ff118002860 2024-08-19T09:52:40.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.035+0000 7ff1267fc640 10 client.134154.objecter resend_mon_ops 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.035+0000 7ff1350a3640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6792/0 conn(0x7ff118002860 legacy=0x7ff118004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 58253444 0 0) 0x7ff1180065c0 con 0x7ff118002860 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient(hunting): my global_id is 26439 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff13013d430 con 0x7ff118002860 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2533113427 0 0) 0x7ff13013d430 con 0x7ff118002860 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1180065c0 con 0x7ff118002860 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff120004f70 con 0x7ff118002860 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient: dump: 2024-08-19T09:52:40.061 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.062 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2775455901 0 0) 0x7ff1180065c0 con 0x7ff118002860 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff11800a160 con 0x7ff118002860 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff120007350 con 0x7ff118002860 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6792/0 conn(0x7ff118002860 legacy=0x7ff118004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6792/0 conn(0x7ff118002860 legacy=0x7ff118004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.036+0000 7ff1267fc640 10 client.134154.objecter ms_handle_connect 0x7ff114041450 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.042+0000 7ff137b08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.042+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff118009e50 con 0x7ff118009110 2024-08-19T09:52:40.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 client.134154.objecter ms_handle_connect 0x7ff118009110 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 client.134154.objecter resend_mon_ops 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1350a3640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6792/0 conn(0x7ff118009110 legacy=0x7ff1180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887853045 0 0) 0x7ff118009e50 con 0x7ff118009110 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient(hunting): my global_id is 26448 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1180065c0 con 0x7ff118009110 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1512929226 0 0) 0x7ff1180065c0 con 0x7ff118009110 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff118009e50 con 0x7ff118009110 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff1200059b0 con 0x7ff118009110 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:40.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient: dump: 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.065 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 504057996 0 0) 0x7ff118009e50 con 0x7ff118009110 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff11800c3f0 con 0x7ff118009110 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1658677205) 0x7ff120006e30 con 0x7ff118009110 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6792/0 conn(0x7ff118009110 legacy=0x7ff1180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.045+0000 7ff1267fc640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6792/0 conn(0x7ff118009110 legacy=0x7ff1180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.066 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 166301, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:40.067 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.068 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.069 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:40.070 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:40.071 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:40.075 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:40.076 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.077 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:40.078 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:40.079 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.081 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.087 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:40.088 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 10 librados: watch_flush enter 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 10 librados: watch_flush exit 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 20 client.134154.objecter shutdown clearing up homeless session... 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 10 client.134154.objecter successfully canceled tick 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff114041450 legacy=0x7ff114043910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 10 monclient: shutdown 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 >> v1:172.21.15.1:6789/0 conn(0x7ff1300a67c0 legacy=0x7ff1301ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff1360a5640 1 -- 172.21.15.1:0/4025062824 reap_dead start 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff1360a5640 1 -- 172.21.15.1:0/4025062824 reap_dead start 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 shutdown_connections 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 >> 172.21.15.1:0/4025062824 conn(0x7ff1300ac4b0 msgr2=0x7ff1301419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 shutdown_connections 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 -- 172.21.15.1:0/4025062824 wait complete. 2024-08-19T09:52:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.046+0000 7ff137b08640 1 librados: shutdown 2024-08-19T09:52:40.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:40.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:52:40.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.190+0000 7f5423db1640 10 monclient: get_monmap_and_config 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.190+0000 7f5423db1640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.190+0000 7f5423db1640 10 monclient: monmap: 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:40.191192+0000 2024-08-19T09:52:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:40.191192+0000 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.192 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.190+0000 7f5423db1640 1 Processor -- start 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 1 -- start start 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient: init 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient: _add_conns ranks=[4,1,8,3,7,2,0,5,6] 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient(hunting): picked mon.noname-c con 0x7f541c0a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient(hunting): picked mon.noname-f con 0x7f541c0694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient(hunting): picked mon.noname-e con 0x7f541c0a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c0a3c70 con 0x7f541c0694b0 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c0a33d0 con 0x7f541c0a6830 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c0a36b0 con 0x7f541c0a46a0 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5423db1640 10 monclient(hunting): authenticate will time out at 816.267652s 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f542234e640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f541c0a46a0 0x7f541c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43586/0 (socket says 172.21.15.1:43586) 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f542234e640 1 -- 172.21.15.1:0/3488727878 learned_addr learned my addr 172.21.15.1:0/3488727878 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.191+0000 7f5421b4d640 1 --1- 172.21.15.1:0/3488727878 >> v1:172.21.15.1:6791/0 conn(0x7f541c0a6830 0x7f541c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36484/0 (socket says 172.21.15.1:36484) 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.193+0000 7f542234e640 1 -- 172.21.15.1:0/3488727878 >> v1:172.21.15.1:6793/0 conn(0x7f541c0a46a0 legacy=0x7f541c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.193+0000 7f5421b4d640 1 -- 172.21.15.1:0/3488727878 >> v1:172.21.15.1:6791/0 conn(0x7f541c0a6830 legacy=0x7f541c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f542134c640 1 -- 172.21.15.1:0/3488727878 >> v1:172.21.15.82:6789/0 conn(0x7f541c0694b0 legacy=0x7f541c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865298271 0 0) 0x7f541c0a36b0 con 0x7f541c0a46a0 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): my global_id is 26909 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5408003520 con 0x7f541c0a46a0 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089041137 0 0) 0x7f541c0a33d0 con 0x7f541c0a6830 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): my global_id is 26446 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c0a36b0 con 0x7f541c0a6830 2024-08-19T09:52:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021169116 0 0) 0x7f541c0a3c70 con 0x7f541c0694b0 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient(hunting): my global_id is 27856 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c0a33d0 con 0x7f541c0694b0 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2924563745 0 0) 0x7f5408003520 con 0x7f541c0a46a0 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 >> v1:172.21.15.1:6791/0 conn(0x7f541c0a6830 legacy=0x7f541c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 >> v1:172.21.15.82:6789/0 conn(0x7f541c0694b0 legacy=0x7f541c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 monclient: found mon.noname-e 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c06dbb0 con 0x7f541c0a46a0 2024-08-19T09:52:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5414001e60 con 0x7f541c0a46a0 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: dump: 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 5 monclient: authenticate success, global_id 26909 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5414004c20 con 0x7f541c0a46a0 2024-08-19T09:52:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 1 -- 172.21.15.1:0/3488727878 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5414005190 con 0x7f541c0a46a0 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 4 set_mon_vals no callback set 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5420b4b640 10 monclient: dump: 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 10 monclient: shutdown 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.194+0000 7f5423db1640 1 -- 172.21.15.1:0/3488727878 >> v1:172.21.15.1:6793/0 conn(0x7f541c0a46a0 legacy=0x7f541c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 -- 172.21.15.1:0/3488727878 shutdown_connections 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 -- 172.21.15.1:0/3488727878 >> 172.21.15.1:0/3488727878 conn(0x7f541c0a9020 msgr2=0x7f541c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 -- 172.21.15.1:0/3488727878 shutdown_connections 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 -- 172.21.15.1:0/3488727878 wait complete. 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 10 monclient: monmap: 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 librados: starting msgr at 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 librados: starting objecter 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.195+0000 7f5423db1640 1 Processor -- start 2024-08-19T09:52:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 1 -- start start 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 1 librados: setting wanted keys 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 1 librados: calling monclient init 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient: init 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient: _add_conns ranks=[4,1,3,8,6,2,0,7,5] 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient(hunting): picked mon.noname-c con 0x7f541c0694b0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient(hunting): picked mon.noname-f con 0x7f541c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient(hunting): picked mon.noname-g con 0x7f541c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c135cb0 con 0x7f541c0a46a0 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c138240 con 0x7f541c0a6830 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c138520 con 0x7f541c0694b0 2024-08-19T09:52:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5423db1640 10 monclient(hunting): authenticate will time out at 816.272481s 2024-08-19T09:52:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5421b4d640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f541c0694b0 0x7f541c134e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36494/0 (socket says 172.21.15.1:36494) 2024-08-19T09:52:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.196+0000 7f5421b4d640 1 -- 172.21.15.1:0/38112083 learned_addr learned my addr 172.21.15.1:0/38112083 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f5421b4d640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6791/0 conn(0x7f541c0694b0 legacy=0x7f541c134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 client.?.objecter ms_handle_connect 0x7f541c0694b0 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 client.?.objecter ms_handle_connect 0x7f541c0a46a0 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f542134c640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.82:6789/0 conn(0x7f541c0a46a0 legacy=0x7f541c1355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f542234e640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.82:6790/0 conn(0x7f541c0a6830 legacy=0x7f541c1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 client.?.objecter ms_handle_connect 0x7f541c0a6830 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270657222 0 0) 0x7f541c138520 con 0x7f541c0694b0 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): my global_id is 26455 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5400004ee0 con 0x7f541c0694b0 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1804676432 0 0) 0x7f541c135cb0 con 0x7f541c0a46a0 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): my global_id is 27865 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c138520 con 0x7f541c0a46a0 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1839929122 0 0) 0x7f541c138240 con 0x7f541c0a6830 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 10 monclient(hunting): my global_id is 134139 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.198+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c135cb0 con 0x7f541c0a6830 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3424957974 0 0) 0x7f5400004ee0 con 0x7f541c0694b0 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f541c138240 con 0x7f541c0694b0 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5418001d30 con 0x7f541c0694b0 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): dump: 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1292385457 0 0) 0x7f541c138520 con 0x7f541c0a46a0 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5400004ee0 con 0x7f541c0a46a0 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f540c003610 con 0x7f541c0a46a0 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): dump: 2024-08-19T09:52:40.220 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3441209634 0 0) 0x7f541c135cb0 con 0x7f541c0a6830 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f541c138520 con 0x7f541c0a6830 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5414005110 con 0x7f541c0a6830 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient(hunting): dump: 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3050835218 0 0) 0x7f541c138240 con 0x7f541c0694b0 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.82:6790/0 conn(0x7f541c0a6830 legacy=0x7f541c1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.82:6789/0 conn(0x7f541c0a46a0 legacy=0x7f541c1355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 monclient: found mon.c 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c138800 con 0x7f541c0694b0 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 5 monclient: authenticate success, global_id 26455 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 10 monclient: _renew_subs 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f541c136880 con 0x7f541c0694b0 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 10 client.26455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 10 monclient: _renew_subs 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f541c139430 con 0x7f541c0694b0 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5423db1640 1 librados: init done 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5418004bc0 con 0x7f541c0694b0 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f54180050f0 con 0x7f541c0694b0 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f5413fff640 4 set_mon_vals no callback set 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.199+0000 7f54127fc640 10 monclient: dump: 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5418005290 con 0x7f541c0694b0 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f5423db1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f54040065c0 con 0x7f5404002860 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(94..94 src has 1..94) ==== 7687+0+0 (unknown 3826229206 0 0) 0x7f541804eac0 con 0x7f541c0694b0 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f54127fc640 10 client.26455.objecter ms_dispatch 0x7f541c000b90 osd_map(94..94 src has 1..94) 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f54127fc640 3 client.26455.objecter handle_osd_map got epochs [94,94] > 0 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f54127fc640 3 client.26455.objecter handle_osd_map decoding full epoch 94 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f54127fc640 20 client.26455.objecter dump_active .. 0 homeless 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.200+0000 7f542234e640 10 monclient: get_auth_request con 0x7f5400041ae0 auth_method 0 2024-08-19T09:52:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f542134c640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6793/0 conn(0x7f5404002860 legacy=0x7f5404004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 client.26455.objecter ms_handle_connect 0x7f5404002860 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 client.26455.objecter resend_mon_ops 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341577922 0 0) 0x7f54040065c0 con 0x7f5404002860 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient(hunting): my global_id is 26918 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c138240 con 0x7f5404002860 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4225676603 0 0) 0x7f541c138240 con 0x7f5404002860 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54040065c0 con 0x7f5404002860 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 client.26455.objecter ms_handle_connect 0x7f5400041ae0 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f540c004140 con 0x7f5404002860 2024-08-19T09:52:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.202+0000 7f54127fc640 10 monclient: dump: 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1288394145 0 0) 0x7f54040065c0 con 0x7f5404002860 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f540400a160 con 0x7f5404002860 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f540c0057c0 con 0x7f5404002860 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6793/0 conn(0x7f5404002860 legacy=0x7f5404004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.203+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6793/0 conn(0x7f5404002860 legacy=0x7f5404004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.209+0000 7f5423db1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.209+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5404009e50 con 0x7f5404009110 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.211+0000 7f542134c640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6793/0 conn(0x7f5404009110 legacy=0x7f54040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.211+0000 7f54127fc640 10 client.26455.objecter ms_handle_connect 0x7f5404009110 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.211+0000 7f54127fc640 10 client.26455.objecter resend_mon_ops 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236295354 0 0) 0x7f5404009e50 con 0x7f5404009110 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient(hunting): my global_id is 26927 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54040065c0 con 0x7f5404009110 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 946175990 0 0) 0x7f54040065c0 con 0x7f5404009110 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5404009e50 con 0x7f5404009110 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f540c005f80 con 0x7f5404009110 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient: dump: 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.232 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.233 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2930751812 0 0) 0x7f5404009e50 con 0x7f5404009110 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f540400c3a0 con 0x7f5404009110 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 468870679) 0x7f540c0074b0 con 0x7f5404009110 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6793/0 conn(0x7f5404009110 legacy=0x7f54040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.212+0000 7f54127fc640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6793/0 conn(0x7f5404009110 legacy=0x7f54040095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 166484, 2024-08-19T09:52:40.235 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.236 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.237 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.238 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:40.239 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:40.240 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.241 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.242 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:40.243 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.244 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:40.249 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.254 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.255 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.256 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:40.257 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:40.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 10 librados: watch_flush enter 2024-08-19T09:52:40.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 10 librados: watch_flush exit 2024-08-19T09:52:40.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 20 client.26455.objecter shutdown clearing up homeless session... 2024-08-19T09:52:40.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 10 client.26455.objecter successfully canceled tick 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5400041ae0 legacy=0x7f5400043fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 10 monclient: shutdown 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 >> v1:172.21.15.1:6791/0 conn(0x7f541c0694b0 legacy=0x7f541c134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f542234e640 1 -- 172.21.15.1:0/38112083 reap_dead start 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f542234e640 1 -- 172.21.15.1:0/38112083 reap_dead start 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 shutdown_connections 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 >> 172.21.15.1:0/38112083 conn(0x7f541c0a9020 msgr2=0x7f541c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 shutdown_connections 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 -- 172.21.15.1:0/38112083 wait complete. 2024-08-19T09:52:40.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.213+0000 7f5423db1640 1 librados: shutdown 2024-08-19T09:52:40.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:40.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:52:40.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:52:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.359+0000 7f75767f7640 10 monclient: get_monmap_and_config 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.359+0000 7f75767f7640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.359+0000 7f75767f7640 10 monclient: monmap: 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:40.360483+0000 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:40.360483+0000 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.363 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 1 Processor -- start 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 1 -- start start 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient: init 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient: _add_conns ranks=[7,5,4,0,6,1,8,2,3] 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient(hunting): picked mon.noname-i con 0x7f75700a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient(hunting): picked mon.noname-h con 0x7f75700a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient(hunting): picked mon.noname-c con 0x7f757013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75700a3b40 con 0x7f757013f5c0 2024-08-19T09:52:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75700a8130 con 0x7f75700a6ba0 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757006ede0 con 0x7f75700a67c0 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f75767f7640 10 monclient(hunting): authenticate will time out at 816.436863s 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f7574d94640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f757013f5c0 0x7f75701419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36498/0 (socket says 172.21.15.1:36498) 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.360+0000 7f7574d94640 1 -- 172.21.15.1:0/3199736660 learned_addr learned my addr 172.21.15.1:0/3199736660 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f7574d94640 1 -- 172.21.15.1:0/3199736660 >> v1:172.21.15.1:6791/0 conn(0x7f757013f5c0 legacy=0x7f75701419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756f7fe640 1 -- 172.21.15.1:0/3199736660 >> v1:172.21.15.82:6791/0 conn(0x7f75700a6ba0 legacy=0x7f75700a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756ffff640 1 -- 172.21.15.1:0/3199736660 >> v1:172.21.15.82:6792/0 conn(0x7f75700a67c0 legacy=0x7f75700a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906728389 0 0) 0x7f75700a3b40 con 0x7f757013f5c0 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): my global_id is 26464 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7558003520 con 0x7f757013f5c0 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82866387 0 0) 0x7f75700a8130 con 0x7f75700a6ba0 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): my global_id is 26078 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75700a3b40 con 0x7f75700a6ba0 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676092160 0 0) 0x7f757006ede0 con 0x7f75700a67c0 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient(hunting): my global_id is 134188 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75700a8130 con 0x7f75700a67c0 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2576921376 0 0) 0x7f7558003520 con 0x7f757013f5c0 2024-08-19T09:52:40.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 >> v1:172.21.15.82:6792/0 conn(0x7f75700a67c0 legacy=0x7f75700a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 >> v1:172.21.15.82:6791/0 conn(0x7f75700a6ba0 legacy=0x7f75700a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 monclient: found mon.noname-c 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75700a92a0 con 0x7f757013f5c0 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7568001e60 con 0x7f757013f5c0 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f75767f7640 5 monclient: authenticate success, global_id 26464 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f75767f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: dump: 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.383 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f75767f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75680020b0 con 0x7f757013f5c0 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 1 -- 172.21.15.1:0/3199736660 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7568004f60 con 0x7f757013f5c0 2024-08-19T09:52:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f75767f7640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f75767f7640 4 set_mon_vals no callback set 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.363+0000 7f756effd640 10 monclient: dump: 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 10 monclient: shutdown 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 1 -- 172.21.15.1:0/3199736660 >> v1:172.21.15.1:6791/0 conn(0x7f757013f5c0 legacy=0x7f75701419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 1 -- 172.21.15.1:0/3199736660 shutdown_connections 2024-08-19T09:52:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 1 -- 172.21.15.1:0/3199736660 >> 172.21.15.1:0/3199736660 conn(0x7f75700ac4b0 msgr2=0x7f75700a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 1 -- 172.21.15.1:0/3199736660 shutdown_connections 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 1 -- 172.21.15.1:0/3199736660 wait complete. 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.364+0000 7f75767f7640 10 monclient: monmap: 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 librados: starting msgr at 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 librados: starting objecter 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 Processor -- start 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 -- start start 2024-08-19T09:52:40.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 librados: setting wanted keys 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 librados: calling monclient init 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient: init 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient: _add_conns ranks=[3,7,1,8,6,5,0,4,2] 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient(hunting): picked mon.noname-g con 0x7f75700a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient(hunting): picked mon.noname-i con 0x7f75700a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient(hunting): picked mon.noname-f con 0x7f757013f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757013d310 con 0x7f757013f5c0 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75701cf020 con 0x7f75700a67c0 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757013ae60 con 0x7f75700a6ba0 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.365+0000 7f75767f7640 10 monclient(hunting): authenticate will time out at 816.441946s 2024-08-19T09:52:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f756ffff640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f75700a67c0 0x7f75701ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:39356/0 (socket says 172.21.15.1:39356) 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f756ffff640 1 -- 172.21.15.1:0/1255160505 learned_addr learned my addr 172.21.15.1:0/1255160505 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f7574d94640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f757013f5c0 legacy=0x7f75701d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f7574d94640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f757013f5c0 legacy=0x7f75701d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f7574d94640 1 --1- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f757013f5c0 0x7f75701d6460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f7574d94640 1 --1- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f757013f5c0 0x7f75701d6460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.366+0000 7f756cff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6789/0 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756ffff640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6790/0 conn(0x7f75700a67c0 legacy=0x7f75701ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756f7fe640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6792/0 conn(0x7f75700a6ba0 legacy=0x7f75701ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 client.?.objecter ms_handle_connect 0x7f75700a67c0 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 client.?.objecter ms_handle_connect 0x7f75700a6ba0 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1943891359 0 0) 0x7f757013ae60 con 0x7f75700a6ba0 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): my global_id is 134197 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756ffff640 1 --1- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6790/0 conn(0x7f75700a67c0 0x7f75701ce200 :-1 s=OPENED pgs=65 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f75701cf020 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f755c004eb0 con 0x7f75700a6ba0 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4287534360 0 0) 0x7f755c004eb0 con 0x7f75700a6ba0 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f757013ae60 con 0x7f75700a6ba0 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7564003610 con 0x7f75700a6ba0 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): dump: 2024-08-19T09:52:40.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1387334701 0 0) 0x7f757013ae60 con 0x7f75700a6ba0 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6790/0 conn(0x7f75700a67c0 legacy=0x7f75701ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f757013f5c0 legacy=0x7f75701d6460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 monclient: found mon.i 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7570139970 con 0x7f75700a6ba0 2024-08-19T09:52:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 5 monclient: authenticate success, global_id 134197 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345486476 0 0) 0x7f75701cf020 con 0x7f75700a67c0 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 10 monclient: _renew_subs 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f757013bcd0 con 0x7f75700a6ba0 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 10 client.134197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 10 monclient: _renew_subs 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f757013e540 con 0x7f75700a6ba0 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f75767f7640 1 librados: init done 2024-08-19T09:52:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.368+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7564003c00 con 0x7f75700a6ba0 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f75640040f0 con 0x7f75700a6ba0 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756e7fc640 4 set_mon_vals no callback set 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756cff9640 10 monclient: dump: 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f75767f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f75500065c0 con 0x7f7550002860 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.369+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7564011a80 con 0x7f75700a6ba0 2024-08-19T09:52:40.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.370+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(94..94 src has 1..94) ==== 7687+0+0 (unknown 3826229206 0 0) 0x7f756404ec30 con 0x7f75700a6ba0 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.370+0000 7f756cff9640 10 client.134197.objecter ms_dispatch 0x7f7570000b90 osd_map(94..94 src has 1..94) 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.370+0000 7f756cff9640 3 client.134197.objecter handle_osd_map got epochs [94,94] > 0 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.370+0000 7f756cff9640 3 client.134197.objecter handle_osd_map decoding full epoch 94 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.370+0000 7f7574d94640 10 monclient: get_auth_request con 0x7f755c040e40 auth_method 0 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.370+0000 7f756cff9640 20 client.134197.objecter dump_active .. 0 homeless 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 client.134197.objecter ms_handle_connect 0x7f7550002860 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 client.134197.objecter resend_mon_ops 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756ffff640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f7550002860 legacy=0x7f7550004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044011272 0 0) 0x7f75500065c0 con 0x7f7550002860 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient(hunting): my global_id is 27874 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75701cf020 con 0x7f7550002860 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2182533854 0 0) 0x7f75701cf020 con 0x7f7550002860 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75500065c0 con 0x7f7550002860 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 client.134197.objecter ms_handle_connect 0x7f755c040e40 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7560003610 con 0x7f7550002860 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient: dump: 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3551595303 0 0) 0x7f75500065c0 con 0x7f7550002860 2024-08-19T09:52:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f755000a160 con 0x7f7550002860 2024-08-19T09:52:40.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f75600057c0 con 0x7f7550002860 2024-08-19T09:52:40.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:40.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f7550002860 legacy=0x7f7550004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.372+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f7550002860 legacy=0x7f7550004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.379+0000 7f75767f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.379+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7550009e70 con 0x7f7550009130 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756ffff640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f7550009130 legacy=0x7f75500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 client.134197.objecter ms_handle_connect 0x7f7550009130 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 client.134197.objecter resend_mon_ops 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592777390 0 0) 0x7f7550009e70 con 0x7f7550009130 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 monclient(hunting): my global_id is 27883 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75500065c0 con 0x7f7550009130 2024-08-19T09:52:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1711278562 0 0) 0x7f75500065c0 con 0x7f7550009130 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7550009e70 con 0x7f7550009130 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7560003fb0 con 0x7f7550009130 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.381+0000 7f756cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 10 monclient: dump: 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 88318150 0 0) 0x7f7550009e70 con 0x7f7550009130 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f755000c410 con 0x7f7550009130 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 811527736) 0x7f7560004270 con 0x7f7550009130 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f7550009130 legacy=0x7f75500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f756cff9640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6789/0 conn(0x7f7550009130 legacy=0x7f75500095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 166642, 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:40.406 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:40.407 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.409 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:40.410 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.411 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:40.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.420 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.421 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.423 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.424 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:40.426 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 10 librados: watch_flush enter 2024-08-19T09:52:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 10 librados: watch_flush exit 2024-08-19T09:52:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 20 client.134197.objecter shutdown clearing up homeless session... 2024-08-19T09:52:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 10 client.134197.objecter successfully canceled tick 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6812/2549104415 conn(0x7f755c040e40 legacy=0x7f755c043300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 10 monclient: shutdown 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 >> v1:172.21.15.82:6792/0 conn(0x7f75700a6ba0 legacy=0x7f75701ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f7574d94640 1 -- 172.21.15.1:0/1255160505 reap_dead start 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f7574d94640 1 -- 172.21.15.1:0/1255160505 reap_dead start 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 shutdown_connections 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 >> 172.21.15.1:0/1255160505 conn(0x7f75700ac4b0 msgr2=0x7f757013edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 shutdown_connections 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 -- 172.21.15.1:0/1255160505 wait complete. 2024-08-19T09:52:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.382+0000 7f75767f7640 1 librados: shutdown 2024-08-19T09:52:40.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:40.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:52:40.430 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:52:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.529+0000 7f4cf7a56640 10 monclient: get_monmap_and_config 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.529+0000 7f4cf7a56640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.529+0000 7f4cf7a56640 10 monclient: monmap: 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:40.530347+0000 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:40.530347+0000 2024-08-19T09:52:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 1 Processor -- start 2024-08-19T09:52:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 1 -- start start 2024-08-19T09:52:40.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient: init 2024-08-19T09:52:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient: _add_conns ranks=[0,5,7,8,6,2,1,3,4] 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf00a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient(hunting): picked mon.noname-h con 0x7f4cf00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient(hunting): picked mon.noname-i con 0x7f4cf013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf00a3b40 con 0x7f4cf00a67c0 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf00a8130 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf006ede0 con 0x7f4cf013f5c0 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf7a56640 10 monclient(hunting): authenticate will time out at 816.606764s 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf57f2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4cf00a67c0 0x7f4cf00a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58952/0 (socket says 172.21.15.1:58952) 2024-08-19T09:52:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.530+0000 7f4cf57f2640 1 -- 172.21.15.1:0/495068447 learned_addr learned my addr 172.21.15.1:0/495068447 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf57f2640 1 -- 172.21.15.1:0/495068447 >> v1:172.21.15.1:6789/0 conn(0x7f4cf00a67c0 legacy=0x7f4cf00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 673464718 0 0) 0x7f4cf00a3b40 con 0x7f4cf00a67c0 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient(hunting): my global_id is 134163 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf4ff1640 1 -- 172.21.15.1:0/495068447 >> v1:172.21.15.82:6791/0 conn(0x7f4cf00a6ba0 legacy=0x7f4cf00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cdc003520 con 0x7f4cf00a67c0 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf5ff3640 1 -- 172.21.15.1:0/495068447 >> v1:172.21.15.82:6792/0 conn(0x7f4cf013f5c0 legacy=0x7f4cf01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2366996978 0 0) 0x7f4cdc003520 con 0x7f4cf00a67c0 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 >> v1:172.21.15.82:6792/0 conn(0x7f4cf013f5c0 legacy=0x7f4cf01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 >> v1:172.21.15.82:6791/0 conn(0x7f4cf00a6ba0 legacy=0x7f4cf00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 monclient: found mon.noname-a 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf00a92a0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4cec001ae0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf7a56640 5 monclient: authenticate success, global_id 134163 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf7a56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:52:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: dump: 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf7a56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4cec0048a0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 1 -- 172.21.15.1:0/495068447 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4cec004dd0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf7a56640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4cf7a56640 4 set_mon_vals no callback set 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.533+0000 7f4ce7fff640 10 monclient: dump: 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.554 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 10 monclient: shutdown 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 1 -- 172.21.15.1:0/495068447 >> v1:172.21.15.1:6789/0 conn(0x7f4cf00a67c0 legacy=0x7f4cf00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 1 -- 172.21.15.1:0/495068447 shutdown_connections 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 1 -- 172.21.15.1:0/495068447 >> 172.21.15.1:0/495068447 conn(0x7f4cf00ac4b0 msgr2=0x7f4cf00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 1 -- 172.21.15.1:0/495068447 shutdown_connections 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 1 -- 172.21.15.1:0/495068447 wait complete. 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.534+0000 7f4cf7a56640 10 monclient: monmap: 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:40.555 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.556 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 librados: starting msgr at 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 librados: starting objecter 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 Processor -- start 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 -- start start 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 librados: setting wanted keys 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 librados: calling monclient init 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient: init 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient: _add_conns ranks=[4,1,5,8,7,3,2,6,0] 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient(hunting): picked mon.noname-c con 0x7f4cf00a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient(hunting): picked mon.noname-f con 0x7f4cf00a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient(hunting): picked mon.noname-h con 0x7f4cf013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:40.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf01cf100 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf013e3d0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf0139b80 con 0x7f4cf013f5c0 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf7a56640 10 monclient(hunting): authenticate will time out at 816.611737s 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf57f2640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4cf00a67c0 0x7f4cf01ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36504/0 (socket says 172.21.15.1:36504) 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.535+0000 7f4cf57f2640 1 -- 172.21.15.1:0/1392480829 learned_addr learned my addr 172.21.15.1:0/1392480829 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.537+0000 7f4cf57f2640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.1:6791/0 conn(0x7f4cf00a67c0 legacy=0x7f4cf01ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.537+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cf00a67c0 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.537+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cf013f5c0 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf5ff3640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6791/0 conn(0x7f4cf013f5c0 legacy=0x7f4cf01d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cf00a6ba0 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf4ff1640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6789/0 conn(0x7f4cf00a6ba0 legacy=0x7f4cf01ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106367949 0 0) 0x7f4cf013e3d0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 26473 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cd4004eb0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727768641 0 0) 0x7f4cf0139b80 con 0x7f4cf013f5c0 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 26096 2024-08-19T09:52:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf013e3d0 con 0x7f4cf013f5c0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374092271 0 0) 0x7f4cf01cf100 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 27892 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf0139b80 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1513730165 0 0) 0x7f4cd4004eb0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cf01cf100 con 0x7f4cf00a67c0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4cec004d90 con 0x7f4cf00a67c0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): dump: 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.561 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1403734139 0 0) 0x7f4cf013e3d0 con 0x7f4cf013f5c0 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cd4004eb0 con 0x7f4cf013f5c0 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ce8001f80 con 0x7f4cf013f5c0 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): dump: 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.562 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1574753618 0 0) 0x7f4cf0139b80 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cf013e3d0 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ce0001cd0 con 0x7f4cf00a6ba0 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient(hunting): dump: 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 238956851 0 0) 0x7f4cf01cf100 con 0x7f4cf00a67c0 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6791/0 conn(0x7f4cf013f5c0 legacy=0x7f4cf01d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6789/0 conn(0x7f4cf00a6ba0 legacy=0x7f4cf01ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 monclient: found mon.c 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf013bee0 con 0x7f4cf00a67c0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 5 monclient: authenticate success, global_id 26473 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 10 monclient: _renew_subs 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf013b640 con 0x7f4cf00a67c0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4cec001d30 con 0x7f4cf00a67c0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4cec005700 con 0x7f4cf00a67c0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce5ffb640 10 monclient: dump: 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.567 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4ce77fe640 4 set_mon_vals no callback set 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 10 client.26473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 10 monclient: _renew_subs 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f4cf00a2890 con 0x7f4cf00a67c0 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.538+0000 7f4cf7a56640 1 librados: init done 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.539+0000 7f4cf7a56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.539+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4cf0143050 con 0x7f4cf009da50 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.539+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4cec011b60 con 0x7f4cf00a67c0 2024-08-19T09:52:40.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.539+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(94..94 src has 1..94) ==== 7687+0+0 (unknown 3826229206 0 0) 0x7f4cec04eb90 con 0x7f4cf00a67c0 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.540+0000 7f4ce5ffb640 10 client.26473.objecter ms_dispatch 0x7f4cf0000b90 osd_map(94..94 src has 1..94) 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.540+0000 7f4ce5ffb640 3 client.26473.objecter handle_osd_map got epochs [94,94] > 0 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.540+0000 7f4ce5ffb640 3 client.26473.objecter handle_osd_map decoding full epoch 94 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.540+0000 7f4ce5ffb640 20 client.26473.objecter dump_active .. 0 homeless 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.540+0000 7f4cf5ff3640 10 monclient: get_auth_request con 0x7f4cd4041960 auth_method 0 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.541+0000 7f4ce5ffb640 10 client.26473.objecter ms_handle_connect 0x7f4cf009da50 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.541+0000 7f4cf4ff1640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6790/0 conn(0x7f4cf009da50 legacy=0x7f4cf009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.541+0000 7f4ce5ffb640 10 client.26473.objecter resend_mon_ops 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462294505 0 0) 0x7f4cf0143050 con 0x7f4cf009da50 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 134157 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf01cf100 con 0x7f4cf009da50 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3239225410 0 0) 0x7f4cf01cf100 con 0x7f4cf009da50 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cf0143050 con 0x7f4cf009da50 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ce0003e10 con 0x7f4cf009da50 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient: dump: 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.570 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.571 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 client.26473.objecter ms_handle_connect 0x7f4cd4041960 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1307058090 0 0) 0x7f4cf0143050 con 0x7f4cf009da50 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cf013bb60 con 0x7f4cf009da50 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4ce00041c0 con 0x7f4cf009da50 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6790/0 conn(0x7f4cf009da50 legacy=0x7f4cf009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.542+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6790/0 conn(0x7f4cf009da50 legacy=0x7f4cf009de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.548+0000 7f4cf7a56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.549+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4cf0068f60 con 0x7f4cf009e540 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4cf4ff1640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6790/0 conn(0x7f4cf009e540 legacy=0x7f4cf01dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 10 client.26473.objecter ms_handle_connect 0x7f4cf009e540 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 10 client.26473.objecter resend_mon_ops 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2073795272 0 0) 0x7f4cf0068f60 con 0x7f4cf009e540 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 134166 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.551+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf0143050 con 0x7f4cf009e540 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3329871408 0 0) 0x7f4cf0143050 con 0x7f4cf009e540 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cf0068f60 con 0x7f4cf009e540 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ce0005b00 con 0x7f4cf009e540 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 10 monclient: dump: 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2831073051 0 0) 0x7f4cf0068f60 con 0x7f4cf009e540 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cf009e940 con 0x7f4cf009e540 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 2468832101) 0x7f4ce0006ff0 con 0x7f4cf009e540 2024-08-19T09:52:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6790/0 conn(0x7f4cf009e540 legacy=0x7f4cf01dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4ce5ffb640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6790/0 conn(0x7f4cf009e540 legacy=0x7f4cf01dbcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:40.578 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17613, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.579 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.580 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.581 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:40.582 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.583 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.584 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.589 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.598 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.598 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.599 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:40.601 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4cf7a56640 10 librados: watch_flush enter 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4cf7a56640 10 librados: watch_flush exit 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4cf7a56640 20 client.26473.objecter shutdown clearing up homeless session... 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.552+0000 7f4cf7a56640 10 client.26473.objecter successfully canceled tick 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4cd4041960 legacy=0x7f4cd4043e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 10 monclient: shutdown 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 >> v1:172.21.15.1:6791/0 conn(0x7f4cf00a67c0 legacy=0x7f4cf01ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf5ff3640 1 -- 172.21.15.1:0/1392480829 reap_dead start 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf5ff3640 1 -- 172.21.15.1:0/1392480829 reap_dead start 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 shutdown_connections 2024-08-19T09:52:40.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 >> 172.21.15.1:0/1392480829 conn(0x7f4cf00ac4b0 msgr2=0x7f4cf013eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 shutdown_connections 2024-08-19T09:52:40.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 -- 172.21.15.1:0/1392480829 wait complete. 2024-08-19T09:52:40.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.553+0000 7f4cf7a56640 1 librados: shutdown 2024-08-19T09:52:40.605 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:40.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:52:40.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:52:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.704+0000 7f5410afd640 10 monclient: get_monmap_and_config 2024-08-19T09:52:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.704+0000 7f5410afd640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.704+0000 7f5410afd640 10 monclient: monmap: 2024-08-19T09:52:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:40.705192+0000 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:40.705192+0000 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.704+0000 7f5410afd640 1 Processor -- start 2024-08-19T09:52:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.704+0000 7f5410afd640 1 -- start start 2024-08-19T09:52:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.704+0000 7f5410afd640 10 monclient: init 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient: _add_conns ranks=[3,5,2,7,1,0,8,4,6] 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient(hunting): picked mon.noname-g con 0x7f540c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient(hunting): picked mon.noname-h con 0x7f540c0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient(hunting): picked mon.noname-b con 0x7f540c0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c0a3c70 con 0x7f540c0a46a0 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c0a33d0 con 0x7f540c0a6830 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c0a36b0 con 0x7f540c0694b0 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f5410afd640 10 monclient(hunting): authenticate will time out at 816.781573s 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f540ad9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f540c0a46a0 0x7f540c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40536/0 (socket says 172.21.15.1:40536) 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.705+0000 7f540ad9d640 1 -- 172.21.15.1:0/384082534 learned_addr learned my addr 172.21.15.1:0/384082534 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.707+0000 7f540ad9d640 1 -- 172.21.15.1:0/384082534 >> v1:172.21.15.1:6790/0 conn(0x7f540c0a46a0 legacy=0x7f540c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.707+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142844107 0 0) 0x7f540c0a3c70 con 0x7f540c0a46a0 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.707+0000 7f540959a640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.707+0000 7f540959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.707+0000 7f540959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.707+0000 7f540959a640 10 monclient(hunting): my global_id is 26525 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f4003520 con 0x7f540c0a46a0 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5409d9b640 1 -- 172.21.15.1:0/384082534 >> v1:172.21.15.82:6791/0 conn(0x7f540c0694b0 legacy=0x7f540c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540a59c640 1 -- 172.21.15.1:0/384082534 >> v1:172.21.15.82:6790/0 conn(0x7f540c0a6830 legacy=0x7f540c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 64769092 0 0) 0x7f53f4003520 con 0x7f540c0a46a0 2024-08-19T09:52:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 >> v1:172.21.15.82:6791/0 conn(0x7f540c0694b0 legacy=0x7f540c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 >> v1:172.21.15.82:6790/0 conn(0x7f540c0a6830 legacy=0x7f540c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 monclient: found mon.noname-b 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c06dbb0 con 0x7f540c0a46a0 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5404001e60 con 0x7f540c0a46a0 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 5 monclient: authenticate success, global_id 26525 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:52:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: dump: 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5404004c20 con 0x7f540c0a46a0 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 1 -- 172.21.15.1:0/384082534 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5404005150 con 0x7f540c0a46a0 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 4 set_mon_vals no callback set 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f540959a640 10 monclient: dump: 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 10 monclient: shutdown 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 1 -- 172.21.15.1:0/384082534 >> v1:172.21.15.1:6790/0 conn(0x7f540c0a46a0 legacy=0x7f540c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 1 -- 172.21.15.1:0/384082534 shutdown_connections 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 1 -- 172.21.15.1:0/384082534 >> 172.21.15.1:0/384082534 conn(0x7f540c0a9020 msgr2=0x7f540c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.708+0000 7f5410afd640 1 -- 172.21.15.1:0/384082534 shutdown_connections 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.709+0000 7f5410afd640 1 -- 172.21.15.1:0/384082534 wait complete. 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.709+0000 7f5410afd640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.709+0000 7f5410afd640 10 monclient: monmap: 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.709+0000 7f5410afd640 1 librados: starting msgr at 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.709+0000 7f5410afd640 1 librados: starting objecter 2024-08-19T09:52:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 Processor -- start 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 -- start start 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 librados: setting wanted keys 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 librados: calling monclient init 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient: init 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient: _add_conns ranks=[1,7,4,2,5,0,3,6,8] 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient(hunting): picked mon.noname-f con 0x7f540c0694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient(hunting): picked mon.noname-i con 0x7f540c0a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient(hunting): picked mon.noname-c con 0x7f540c0a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c139780 con 0x7f540c0694b0 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c13bae0 con 0x7f540c0a6830 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c13b240 con 0x7f540c0a46a0 2024-08-19T09:52:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f5410afd640 10 monclient(hunting): authenticate will time out at 816.786612s 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f540ad9d640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f540c0a6830 0x7f540c138ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36512/0 (socket says 172.21.15.1:36512) 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.710+0000 7f540ad9d640 1 -- 172.21.15.1:0/2259382386 learned_addr learned my addr 172.21.15.1:0/2259382386 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f540ad9d640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.1:6791/0 conn(0x7f540c0a6830 legacy=0x7f540c138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 10 client.?.objecter ms_handle_connect 0x7f540c0a6830 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f5409d9b640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6792/0 conn(0x7f540c0a46a0 legacy=0x7f540c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 10 client.?.objecter ms_handle_connect 0x7f540c0a46a0 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 10 client.?.objecter ms_handle_connect 0x7f540c0694b0 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f540a59c640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6789/0 conn(0x7f540c0694b0 legacy=0x7f540c137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.712+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2537883294 0 0) 0x7f540c13bae0 con 0x7f540c0a6830 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): my global_id is 26482 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53ec004eb0 con 0x7f540c0a6830 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394791076 0 0) 0x7f540c13b240 con 0x7f540c0a46a0 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): my global_id is 134215 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f540c13bae0 con 0x7f540c0a46a0 2024-08-19T09:52:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997151445 0 0) 0x7f540c139780 con 0x7f540c0694b0 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): my global_id is 27901 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f540c13b240 con 0x7f540c0694b0 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1408790790 0 0) 0x7f53ec004eb0 con 0x7f540c0a6830 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f540c139780 con 0x7f540c0a6830 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5404005110 con 0x7f540c0a6830 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): dump: 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3395919344 0 0) 0x7f540c13bae0 con 0x7f540c0a46a0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53ec004eb0 con 0x7f540c0a46a0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4064726491 0 0) 0x7f540c13b240 con 0x7f540c0694b0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f540c13bae0 con 0x7f540c0694b0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5400001d10 con 0x7f540c0a46a0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): dump: 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f53fc001c70 con 0x7f540c0694b0 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient(hunting): dump: 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3134475443 0 0) 0x7f540c139780 con 0x7f540c0a6830 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6792/0 conn(0x7f540c0a46a0 legacy=0x7f540c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6789/0 conn(0x7f540c0694b0 legacy=0x7f540c137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 monclient: found mon.c 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c142320 con 0x7f540c0a6830 2024-08-19T09:52:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 5 monclient: authenticate success, global_id 26482 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 10 monclient: _renew_subs 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f540c1425d0 con 0x7f540c0a6830 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 10 client.26482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 10 monclient: _renew_subs 2024-08-19T09:52:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f540c1d75f0 con 0x7f540c0a6830 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5410afd640 1 librados: init done 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f54040020b0 con 0x7f540c0a6830 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5404004c20 con 0x7f540c0a6830 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f5408d99640 4 set_mon_vals no callback set 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.713+0000 7f53faffd640 10 monclient: dump: 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f5410afd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53f00065c0 con 0x7f53f0002860 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5404011d40 con 0x7f540c0a6830 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(94..94 src has 1..94) ==== 7687+0+0 (unknown 3826229206 0 0) 0x7f540404ee00 con 0x7f540c0a6830 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f53faffd640 10 client.26482.objecter ms_dispatch 0x7f540c000b90 osd_map(94..94 src has 1..94) 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f53faffd640 3 client.26482.objecter handle_osd_map got epochs [94,94] > 0 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f53faffd640 3 client.26482.objecter handle_osd_map decoding full epoch 94 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f5409d9b640 10 monclient: get_auth_request con 0x7f53ec041a70 auth_method 0 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.714+0000 7f53faffd640 20 client.26482.objecter dump_active .. 0 homeless 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 10 client.26482.objecter ms_handle_connect 0x7f53f0002860 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 10 client.26482.objecter resend_mon_ops 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f540a59c640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6791/0 conn(0x7f53f0002860 legacy=0x7f53f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136600288 0 0) 0x7f53f00065c0 con 0x7f53f0002860 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 10 monclient(hunting): my global_id is 26114 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.716+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f540c139780 con 0x7f53f0002860 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4191078024 0 0) 0x7f540c139780 con 0x7f53f0002860 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53f00065c0 con 0x7f53f0002860 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f53fc003c00 con 0x7f53f0002860 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 10 monclient: dump: 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 10 client.26482.objecter ms_handle_connect 0x7f53ec041a70 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2805181822 0 0) 0x7f53f00065c0 con 0x7f53f0002860 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53f000a160 con 0x7f53f0002860 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f53fc004010 con 0x7f53f0002860 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6791/0 conn(0x7f53f0002860 legacy=0x7f53f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.717+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6791/0 conn(0x7f53f0002860 legacy=0x7f53f0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.723+0000 7f5410afd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.723+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53f0009e50 con 0x7f53f0009110 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f540a59c640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6791/0 conn(0x7f53f0009110 legacy=0x7f53f00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 client.26482.objecter ms_handle_connect 0x7f53f0009110 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 client.26482.objecter resend_mon_ops 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270049732 0 0) 0x7f53f0009e50 con 0x7f53f0009110 2024-08-19T09:52:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient(hunting): my global_id is 26123 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f00065c0 con 0x7f53f0009110 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 478819606 0 0) 0x7f53f00065c0 con 0x7f53f0009110 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53f0009e50 con 0x7f53f0009110 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f53fc005b00 con 0x7f53f0009110 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 10 monclient: dump: 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 656146994 0 0) 0x7f53f0009e50 con 0x7f53f0009110 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.726+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53f000c3a0 con 0x7f53f0009110 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7851 (unknown 0 0 5478431) 0x7f53fc007ec0 con 0x7f53f0009110 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f53faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f53faffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6791/0 conn(0x7f53f0009110 legacy=0x7f53f00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f53faffd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6791/0 conn(0x7f53f0009110 legacy=0x7f53f00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:40.748 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 166992, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:40.749 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.750 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:40.751 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.752 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:40.753 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.754 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.755 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.756 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.757 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.764 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.765 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.766 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.767 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.768 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:40.769 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 10 librados: watch_flush enter 2024-08-19T09:52:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 10 librados: watch_flush exit 2024-08-19T09:52:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 20 client.26482.objecter shutdown clearing up homeless session... 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 10 client.26482.objecter successfully canceled tick 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.82:6812/2549104415 conn(0x7f53ec041a70 legacy=0x7f53ec043f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 10 monclient: shutdown 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 >> v1:172.21.15.1:6791/0 conn(0x7f540c0a6830 legacy=0x7f540c138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f540ad9d640 1 -- 172.21.15.1:0/2259382386 reap_dead start 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f540ad9d640 1 -- 172.21.15.1:0/2259382386 reap_dead start 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 shutdown_connections 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 >> 172.21.15.1:0/2259382386 conn(0x7f540c0a9020 msgr2=0x7f540c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 shutdown_connections 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 -- 172.21.15.1:0/2259382386 wait complete. 2024-08-19T09:52:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.727+0000 7f5410afd640 1 librados: shutdown 2024-08-19T09:52:40.773 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:40.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:52:40.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:52:40.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.872+0000 7f166b3f7640 10 monclient: get_monmap_and_config 2024-08-19T09:52:40.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.872+0000 7f166b3f7640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.872+0000 7f166b3f7640 10 monclient: monmap: 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:40.873152+0000 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:40.873152+0000 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.874 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.872+0000 7f166b3f7640 1 Processor -- start 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.872+0000 7f166b3f7640 1 -- start start 2024-08-19T09:52:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.872+0000 7f166b3f7640 10 monclient: init 2024-08-19T09:52:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient: _add_conns ranks=[6,5,7,4,1,0,2,8,3] 2024-08-19T09:52:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient(hunting): picked mon.noname-d con 0x7f16640a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient(hunting): picked mon.noname-h con 0x7f16640694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient(hunting): picked mon.noname-i con 0x7f16640a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16640a3c70 con 0x7f16640694b0 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16640a33d0 con 0x7f16640a6830 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16640a36b0 con 0x7f16640a46a0 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f166b3f7640 10 monclient(hunting): authenticate will time out at 816.949480s 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f1669193640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f16640a6830 0x7f16640a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35634/0 (socket says 172.21.15.1:35634) 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.873+0000 7f1669193640 1 -- 172.21.15.1:0/2760825906 learned_addr learned my addr 172.21.15.1:0/2760825906 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f1669193640 1 -- 172.21.15.1:0/2760825906 >> v1:172.21.15.1:6792/0 conn(0x7f16640a6830 legacy=0x7f16640a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f1669994640 1 -- 172.21.15.1:0/2760825906 >> v1:172.21.15.82:6792/0 conn(0x7f16640a46a0 legacy=0x7f1664141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f1668992640 1 -- 172.21.15.1:0/2760825906 >> v1:172.21.15.82:6791/0 conn(0x7f16640694b0 legacy=0x7f16640a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3627034207 0 0) 0x7f16640a33d0 con 0x7f16640a6830 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f165bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f165bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f165bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f165bfff640 10 monclient(hunting): my global_id is 26457 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.875+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1650003520 con 0x7f16640a6830 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485430763 0 0) 0x7f16640a36b0 con 0x7f16640a46a0 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): my global_id is 134224 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16640a33d0 con 0x7f16640a46a0 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371318970 0 0) 0x7f16640a3c70 con 0x7f16640694b0 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient(hunting): my global_id is 26132 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16640a36b0 con 0x7f16640694b0 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 21687552 0 0) 0x7f1650003520 con 0x7f16640a6830 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 >> v1:172.21.15.82:6792/0 conn(0x7f16640a46a0 legacy=0x7f1664141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 >> v1:172.21.15.82:6791/0 conn(0x7f16640694b0 legacy=0x7f16640a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 monclient: found mon.noname-d 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166406dbb0 con 0x7f16640a6830 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1660001ae0 con 0x7f16640a6830 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 5 monclient: authenticate success, global_id 26457 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: dump: 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16600048a0 con 0x7f16640a6830 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 1 -- 172.21.15.1:0/2760825906 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1660004e10 con 0x7f16640a6830 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 4 set_mon_vals no callback set 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f165bfff640 10 monclient: dump: 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 10 monclient: shutdown 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 1 -- 172.21.15.1:0/2760825906 >> v1:172.21.15.1:6792/0 conn(0x7f16640a6830 legacy=0x7f16640a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 1 -- 172.21.15.1:0/2760825906 shutdown_connections 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 1 -- 172.21.15.1:0/2760825906 >> 172.21.15.1:0/2760825906 conn(0x7f16640a9020 msgr2=0x7f1664068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.876+0000 7f166b3f7640 1 -- 172.21.15.1:0/2760825906 shutdown_connections 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 -- 172.21.15.1:0/2760825906 wait complete. 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 10 monclient: build_initial_monmap 2024-08-19T09:52:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 10 monclient: monmap: 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 librados: starting msgr at 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 librados: starting objecter 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 Processor -- start 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 -- start start 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 librados: setting wanted keys 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 1 librados: calling monclient init 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.877+0000 7f166b3f7640 10 monclient: init 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient: _add_conns ranks=[4,1,5,0,2,7,6,3,8] 2024-08-19T09:52:40.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient(hunting): picked mon.noname-c con 0x7f16640694b0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient(hunting): picked mon.noname-f con 0x7f16640a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient(hunting): picked mon.noname-h con 0x7f16640a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16640a9ec0 con 0x7f16640a46a0 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1664138bb0 con 0x7f16640694b0 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1664138e90 con 0x7f16640a6830 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f166b3f7640 10 monclient(hunting): authenticate will time out at 816.954438s 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f1669193640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f16640694b0 0x7f16641419f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36518/0 (socket says 172.21.15.1:36518) 2024-08-19T09:52:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.878+0000 7f1669193640 1 -- 172.21.15.1:0/1918657853 learned_addr learned my addr 172.21.15.1:0/1918657853 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:40.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1669193640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.1:6791/0 conn(0x7f16640694b0 legacy=0x7f16641419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 client.?.objecter ms_handle_connect 0x7f16640694b0 2024-08-19T09:52:40.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1669994640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6791/0 conn(0x7f16640a6830 legacy=0x7f16641d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 client.?.objecter ms_handle_connect 0x7f16640a6830 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 client.?.objecter ms_handle_connect 0x7f16640a46a0 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1668992640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6789/0 conn(0x7f16640a46a0 legacy=0x7f16641d3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274398724 0 0) 0x7f1664138bb0 con 0x7f16640694b0 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): my global_id is 26491 2024-08-19T09:52:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1648004eb0 con 0x7f16640694b0 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501121265 0 0) 0x7f1664138e90 con 0x7f16640a6830 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 10 monclient(hunting): my global_id is 26141 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.880+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1664138bb0 con 0x7f16640a6830 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1313066163 0 0) 0x7f16640a9ec0 con 0x7f16640a46a0 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): my global_id is 27910 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1664138e90 con 0x7f16640a46a0 2024-08-19T09:52:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2115837793 0 0) 0x7f1648004eb0 con 0x7f16640694b0 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16640a9ec0 con 0x7f16640694b0 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1660001d30 con 0x7f16640694b0 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): dump: 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.901 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3124385371 0 0) 0x7f1664138bb0 con 0x7f16640a6830 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1648004eb0 con 0x7f16640a6830 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f165c003990 con 0x7f16640a6830 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient(hunting): dump: 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.904 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1480393673 0 0) 0x7f16640a9ec0 con 0x7f16640694b0 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6791/0 conn(0x7f16640a6830 legacy=0x7f16641d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6789/0 conn(0x7f16640a46a0 legacy=0x7f16641d3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:52:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 monclient: found mon.c 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1664137c50 con 0x7f16640694b0 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: _finish_auth 0 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 5 monclient: authenticate success, global_id 26491 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 10 monclient: _renew_subs 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1664137130 con 0x7f16640694b0 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1660005040 con 0x7f16640694b0 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1660005530 con 0x7f16640694b0 2024-08-19T09:52:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f1659ffb640 10 monclient: dump: 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.909 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 10 client.26491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f165b7fe640 4 set_mon_vals no callback set 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 10 monclient: _renew_subs 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f16641422e0 con 0x7f16640694b0 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.881+0000 7f166b3f7640 1 librados: init done 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f166b3f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f16640a3c70 con 0x7f16641ce520 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1660011990 con 0x7f16640694b0 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(94..94 src has 1..94) ==== 7687+0+0 (unknown 3826229206 0 0) 0x7f166004ea30 con 0x7f16640694b0 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1659ffb640 10 client.26491.objecter ms_dispatch 0x7f1664000b90 osd_map(94..94 src has 1..94) 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1659ffb640 3 client.26491.objecter handle_osd_map got epochs [94,94] > 0 2024-08-19T09:52:40.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1659ffb640 3 client.26491.objecter handle_osd_map decoding full epoch 94 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1669994640 10 monclient: get_auth_request con 0x7f1648041710 auth_method 0 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.882+0000 7f1659ffb640 20 client.26491.objecter dump_active .. 0 homeless 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1668992640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6792/0 conn(0x7f16641ce520 legacy=0x7f166409e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 10 client.26491.objecter ms_handle_connect 0x7f16641ce520 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 10 client.26491.objecter resend_mon_ops 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330927609 0 0) 0x7f16640a36b0 con 0x7f16641ce520 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 10 monclient(hunting): my global_id is 134233 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.884+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16640a9ec0 con 0x7f16641ce520 2024-08-19T09:52:40.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1073399548 0 0) 0x7f16640a9ec0 con 0x7f16641ce520 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16640a36b0 con 0x7f16641ce520 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 10 client.26491.objecter ms_handle_connect 0x7f1648041710 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1654003ed0 con 0x7f16641ce520 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 10 monclient: dump: 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4228129127 0 0) 0x7f16640a36b0 con 0x7f16641ce520 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1664142bb0 con 0x7f16641ce520 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1654004170 con 0x7f16641ce520 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6792/0 conn(0x7f16641ce520 legacy=0x7f166409e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.885+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6792/0 conn(0x7f16641ce520 legacy=0x7f166409e880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.892+0000 7f166b3f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.892+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f16640aa720 con 0x7f166402e1d0 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1668992640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6792/0 conn(0x7f166402e1d0 legacy=0x7f166409ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 10 client.26491.objecter ms_handle_connect 0x7f166402e1d0 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 10 client.26491.objecter resend_mon_ops 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4057827367 0 0) 0x7f1654004420 con 0x7f166402e1d0 2024-08-19T09:52:40.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 10 monclient(hunting): my global_id is 134242 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.894+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16640a36b0 con 0x7f166402e1d0 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1428984051 0 0) 0x7f16640a36b0 con 0x7f166402e1d0 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1654004420 con 0x7f166402e1d0 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1654005d70 con 0x7f166402e1d0 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 10 monclient: dump: 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1507191241 0 0) 0x7f1654004420 con 0x7f166402e1d0 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16640a5d40 con 0x7f166402e1d0 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 4065860940) 0x7f1654008110 con 0x7f166402e1d0 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6792/0 conn(0x7f166402e1d0 legacy=0x7f166409ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1659ffb640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6792/0 conn(0x7f166402e1d0 legacy=0x7f166409ef90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17956, 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.919 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.920 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.921 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:40.922 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:40.923 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:40.924 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:40.927 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:40.932 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:40.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.938 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:40.939 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:40.940 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:40.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 10 librados: watch_flush enter 2024-08-19T09:52:40.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 10 librados: watch_flush exit 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 20 client.26491.objecter shutdown clearing up homeless session... 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 10 client.26491.objecter successfully canceled tick 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1648041710 legacy=0x7f1648043bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 10 monclient: shutdown 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 >> v1:172.21.15.1:6791/0 conn(0x7f16640694b0 legacy=0x7f16641419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1669994640 1 -- 172.21.15.1:0/1918657853 reap_dead start 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f1669994640 1 -- 172.21.15.1:0/1918657853 reap_dead start 2024-08-19T09:52:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 shutdown_connections 2024-08-19T09:52:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.895+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 >> 172.21.15.1:0/1918657853 conn(0x7f16640a9020 msgr2=0x7f16640a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.896+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 shutdown_connections 2024-08-19T09:52:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.896+0000 7f166b3f7640 1 -- 172.21.15.1:0/1918657853 wait complete. 2024-08-19T09:52:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:40.896+0000 7f166b3f7640 1 librados: shutdown 2024-08-19T09:52:40.944 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:40.944 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T09:52:40.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.043+0000 7ff5b2511640 10 monclient: get_monmap_and_config 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.043+0000 7ff5b2511640 10 monclient: build_initial_monmap 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.043+0000 7ff5b2511640 10 monclient: monmap: 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:41.044214+0000 2024-08-19T09:52:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:41.044214+0000 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:41.046 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:41.046 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:41.046 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.043+0000 7ff5b2511640 1 Processor -- start 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 1 -- start start 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient: init 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient: _add_conns ranks=[5,2,6,4,0,8,1,3,7] 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient(hunting): picked mon.noname-h con 0x7ff5ac0a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ac0694b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient(hunting): picked mon.noname-d con 0x7ff5ac0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac0a3c70 con 0x7ff5ac0694b0 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac0a33d0 con 0x7ff5ac0a6830 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac0a36b0 con 0x7ff5ac0a46a0 2024-08-19T09:52:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b2511640 10 monclient(hunting): authenticate will time out at 817.120570s 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b0aae640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ff5ac0a46a0 0x7ff5ac141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35648/0 (socket says 172.21.15.1:35648) 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.044+0000 7ff5b0aae640 1 -- 172.21.15.1:0/51879393 learned_addr learned my addr 172.21.15.1:0/51879393 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.046+0000 7ff5ab7fe640 1 -- 172.21.15.1:0/51879393 >> v1:172.21.15.1:6790/0 conn(0x7ff5ac0694b0 legacy=0x7ff5ac0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.046+0000 7ff5b0aae640 1 -- 172.21.15.1:0/51879393 >> v1:172.21.15.1:6792/0 conn(0x7ff5ac0a46a0 legacy=0x7ff5ac141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3469819222 0 0) 0x7ff5ac0a3c70 con 0x7ff5ac0694b0 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): my global_id is 26534 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff594003520 con 0x7ff5ac0694b0 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288601325 0 0) 0x7ff5ac0a36b0 con 0x7ff5ac0a46a0 2024-08-19T09:52:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient(hunting): my global_id is 26466 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5ac0a3c70 con 0x7ff5ac0a46a0 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5abfff640 1 -- 172.21.15.1:0/51879393 >> v1:172.21.15.82:6791/0 conn(0x7ff5ac0a6830 legacy=0x7ff5ac0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3400801647 0 0) 0x7ff594003520 con 0x7ff5ac0694b0 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 >> v1:172.21.15.1:6792/0 conn(0x7ff5ac0a46a0 legacy=0x7ff5ac141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 >> v1:172.21.15.82:6791/0 conn(0x7ff5ac0a6830 legacy=0x7ff5ac0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 monclient: found mon.noname-b 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ac06dbb0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff5a0001b30 con 0x7ff5ac0694b0 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 5 monclient: authenticate success, global_id 26534 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: dump: 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5a0003610 con 0x7ff5ac0694b0 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 1 -- 172.21.15.1:0/51879393 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff5a0003b80 con 0x7ff5ac0694b0 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 4 set_mon_vals no callback set 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5aaffd640 10 monclient: dump: 2024-08-19T09:52:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:41.064 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 10 monclient: shutdown 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 1 -- 172.21.15.1:0/51879393 >> v1:172.21.15.1:6790/0 conn(0x7ff5ac0694b0 legacy=0x7ff5ac0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 1 -- 172.21.15.1:0/51879393 shutdown_connections 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 1 -- 172.21.15.1:0/51879393 >> 172.21.15.1:0/51879393 conn(0x7ff5ac0a9020 msgr2=0x7ff5ac068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.047+0000 7ff5b2511640 1 -- 172.21.15.1:0/51879393 shutdown_connections 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.048+0000 7ff5b2511640 1 -- 172.21.15.1:0/51879393 wait complete. 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.048+0000 7ff5b2511640 10 monclient: build_initial_monmap 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.048+0000 7ff5b2511640 10 monclient: monmap: 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.048+0000 7ff5b2511640 1 librados: starting msgr at 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.048+0000 7ff5b2511640 1 librados: starting objecter 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.048+0000 7ff5b2511640 1 Processor -- start 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 1 -- start start 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 1 librados: setting wanted keys 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 1 librados: calling monclient init 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient: init 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient: _add_conns ranks=[8,4,1,0,6,5,2,7,3] 2024-08-19T09:52:41.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient(hunting): picked mon.noname-e con 0x7ff5ac0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ac0a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient(hunting): picked mon.noname-f con 0x7ff5ac0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac1cf1d0 con 0x7ff5ac0a6830 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac137f00 con 0x7ff5ac0a46a0 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac1384d0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5b2511640 10 monclient(hunting): authenticate will time out at 817.125497s 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5ab7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff5ac0a46a0 0x7ff5ac1ceac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36532/0 (socket says 172.21.15.1:36532) 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5ab7fe640 1 -- 172.21.15.1:0/2908687129 learned_addr learned my addr 172.21.15.1:0/2908687129 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.049+0000 7ff5abfff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff5ac0694b0 0x7ff5ac1ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43626/0 (socket says 172.21.15.1:43626) 2024-08-19T09:52:41.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5ab7fe640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.1:6791/0 conn(0x7ff5ac0a46a0 legacy=0x7ff5ac1ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 client.?.objecter ms_handle_connect 0x7ff5ac0a46a0 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5abfff640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.1:6793/0 conn(0x7ff5ac0694b0 legacy=0x7ff5ac1ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 client.?.objecter ms_handle_connect 0x7ff5ac0694b0 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5ab7fe640 1 --1- 172.21.15.1:0/2908687129 >> v1:172.21.15.1:6791/0 conn(0x7ff5ac0a46a0 0x7ff5ac1ceac0 :-1 s=OPENED pgs=3133 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7ff5ac137f00 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192216352 0 0) 0x7ff5ac1384d0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 monclient(hunting): my global_id is 26936 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff598004eb0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 client.?.objecter ms_handle_connect 0x7ff5ac0a6830 2024-08-19T09:52:41.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.051+0000 7ff5b0aae640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.82:6789/0 conn(0x7ff5ac0a6830 legacy=0x7ff5ac1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3631126078 0 0) 0x7ff598004eb0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5ac1384d0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff59c001cb0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:41.070 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214632224 0 0) 0x7ff5ac1cf1d0 con 0x7ff5ac0a6830 2024-08-19T09:52:41.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient(hunting): my global_id is 27919 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff598004eb0 con 0x7ff5ac0a6830 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3623995140 0 0) 0x7ff5ac1384d0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.1:6791/0 conn(0x7ff5ac0a46a0 legacy=0x7ff5ac1ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.82:6789/0 conn(0x7ff5ac0a6830 legacy=0x7ff5ac1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 monclient: found mon.e 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ac1387b0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: _finish_auth 0 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 5 monclient: authenticate success, global_id 26936 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393572513 0 0) 0x7ff5ac137f00 con 0x7ff5ac0a46a0 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 10 monclient: _renew_subs 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ac136ad0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 10 client.26936.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 10 monclient: _renew_subs 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5ac142830 con 0x7ff5ac0694b0 2024-08-19T09:52:41.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5b2511640 1 librados: init done 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff59c003a30 con 0x7ff5ac0694b0 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff59c003f60 con 0x7ff5ac0694b0 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5aa7fc640 4 set_mon_vals no callback set 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.052+0000 7ff5a8ff9640 10 monclient: dump: 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:41.074 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff59c004100 con 0x7ff5ac0694b0 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5b2511640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:41.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5b2511640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5b2511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff58c005080 con 0x7ff5ac0694b0 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(95..95 src has 1..95) ==== 8066+0+0 (unknown 3793247742 0 0) 0x7ff59c04ebb0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5a8ff9640 10 client.26936.objecter ms_dispatch 0x7ff5ac000b90 osd_map(95..95 src has 1..95) 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5a8ff9640 3 client.26936.objecter handle_osd_map got epochs [95,95] > 0 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5a8ff9640 3 client.26936.objecter handle_osd_map decoding full epoch 95 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5a8ff9640 20 client.26936.objecter dump_active .. 0 homeless 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.053+0000 7ff5ab7fe640 10 monclient: get_auth_request con 0x7ff598040de0 auth_method 0 2024-08-19T09:52:41.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.056+0000 7ff5a8ff9640 10 client.26936.objecter ms_handle_connect 0x7ff598040de0 2024-08-19T09:52:41.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.058+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff59c0174b0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.058+0000 7ff5a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:41.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.058+0000 7ff5a8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:41.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.309+0000 7ff5b2511640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T09:52:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.309+0000 7ff5b2511640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T09:52:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.309+0000 7ff5b2511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.309+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff58c005c40 con 0x7ff5ac0694b0 2024-08-19T09:52:41.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=3371296443,config=2586702516,health=2807885334,kv=1455257465,logm=2139814822}) 2024-08-19T09:52:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.311+0000 7ff5a8ff9640 1 -- 172.21.15.1:0/2908687129 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7ff59c016db0 con 0x7ff5ac0694b0 2024-08-19T09:52:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.311+0000 7ff5a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T09:52:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.311+0000 7ff5a8ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=10,kv=11,logm=57} crc {auth=486636539,config=2586702516,health=813971443,kv=2971667195,logm=3029922123}) 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=1629063032,config=2586702516,health=2807885334,kv=2401233588,logm=829832433}) 2024-08-19T09:52:41.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=486636539,config=2586702516,health=2807885334,kv=2971667195,logm=1163702984}) 2024-08-19T09:52:41.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=2425677008,config=2586702516,health=2807885334,kv=1455257465,logm=1036202866}) 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=486636539,config=2586702516,health=2807885334,kv=1487172480,logm=3101948471}) 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=20,config=2,health=12,kv=10,logm=56} crc {auth=486636539,config=2586702516,health=2807885334,kv=375198362,logm=3563939015}) 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=503365776,config=2586702516,health=2807885334,kv=1455257465,logm=1175378856}) 2024-08-19T09:52:41.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.311+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=486636539,config=2586702516,health=2807885334,kv=3949047626,logm=2662677155}) 2024-08-19T09:52:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 10 librados: watch_flush enter 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 10 librados: watch_flush exit 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 20 client.26936.objecter shutdown clearing up homeless session... 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 10 client.26936.objecter successfully canceled tick 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff598040de0 legacy=0x7ff5980432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 10 monclient: shutdown 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 >> v1:172.21.15.1:6793/0 conn(0x7ff5ac0694b0 legacy=0x7ff5ac1ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 shutdown_connections 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 >> 172.21.15.1:0/2908687129 conn(0x7ff5ac0a9020 msgr2=0x7ff5ac0a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 shutdown_connections 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 -- 172.21.15.1:0/2908687129 wait complete. 2024-08-19T09:52:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:41.314+0000 7ff5b2511640 1 librados: shutdown 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525797036}) 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501596347}) 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1678208017}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2678437189}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2492646215}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3662872438}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=538221422}) 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986794952}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.321+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3039318813}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1629976972}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241012074}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1151226552}) 2024-08-19T09:52:41.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1696932055}) 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2992576089}) 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2805868838}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2905725866}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105915677}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.322+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=605319186}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.323+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.323+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077939043}) 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958038265}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=670562465}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2129450569}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1669335561}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=414284065}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230629400}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58348611}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.324+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2389896108}) 2024-08-19T09:52:41.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3214593761}) 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=872376043}) 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=161556899}) 2024-08-19T09:52:41.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=37528879}) 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=137774950}) 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1186774186}) 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3132339287}) 2024-08-19T09:52:41.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599322030}) 2024-08-19T09:52:41.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.325+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2858453928}) 2024-08-19T09:52:41.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T09:52:41.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512433964}) 2024-08-19T09:52:41.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1062651623}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3856258952}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2973480009}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=727437077}) 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1700217704}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3737192543}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912428611}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.327+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=85357250}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960915419}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2105446620}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487810583}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1910390529}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2742208089}) 2024-08-19T09:52:41.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3683934143}) 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3329819459}) 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179944857}) 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4085847037}) 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2797738249}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236711702}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1421405707}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3846412405}) 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=483675859}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3234370642}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867649843}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597114521}) 2024-08-19T09:52:41.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=690405650}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3465761508}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260534343}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2443042065}) 2024-08-19T09:52:41.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=799687195}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2325084185}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3992583127}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3500745634}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348787600}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1998501253}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1706476268}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016405269}) 2024-08-19T09:52:41.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2769077261}) 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=203317786}) 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1770188670}) 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2429528990}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4071591236}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076587880}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1725752069}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3810601233}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3694881763}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3585892158}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3661273769}) 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=305763255}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3651345938}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3385975111}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329201323}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1698835544}) 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.382 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214203283}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599163499}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2422191988}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2374245496}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4049006076}) 2024-08-19T09:52:41.383 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1063927756}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1483490957}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417282038}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2809489260}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3361731687}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647235673}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3989495658}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2108184243}) 2024-08-19T09:52:41.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1682853086}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2994585666}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788277395}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048715751}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=466723014}) 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024110458}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4217932884}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1211241664}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1286834347}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=91952055}) 2024-08-19T09:52:41.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=47317404}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2942188905}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145837897}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1029375233}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047504214}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1268986580}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2538601019}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2497328161}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=565756168}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2606052986}) 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2862785242}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953297467}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.339+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1156724827}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076158170}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395704703}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4045902447}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4017381938}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1548033343}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=150039229}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2135054277}) 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642839462}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2781674163}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,osd_metadata=4,osd_snap=16,osdmap=8} crc {logm=2095786228,mdsmap=61829903,mgr=1503513097,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2000685225,mon_config_key=274864763,monmap=3096227638,osd_metadata=2600220349,osd_snap=919072624,osdmap=612903407}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=7} crc {logm=1436778498,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1793397674,mon_config_key=3373276865,monmap=3096227638,osd_metadata=1991917048,osd_pg_creating=3631593561,osd_snap=20715893,osdmap=1593711178}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=30,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=3} crc {logm=4099626493,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3493631983,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=3631593561,osd_snap=3547451919,osdmap=2835886545}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=15,osdmap=2} crc {logm=1720191403,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3493631983,mon_config_key=1905280744,monmap=3096227638,osd_metadata=1609305515,osd_pg_creating=3631593561,osd_snap=989030439,osdmap=3238098892}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=7} crc {logm=732041400,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3959475423,mon_config_key=2560225996,monmap=3096227638,osd_metadata=675158879,osd_pg_creating=3631593561,osd_snap=1367904004,osdmap=612874069}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=2} crc {logm=3780624559,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3493631983,mon_config_key=3373276865,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=3631593561,osd_snap=1448846793,osdmap=3238098892}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16} crc {logm=199235256,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3493631983,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=3631593561,osd_snap=4102662973}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16,osdmap=5} crc {logm=1971084644,mdsmap=61829903,mgr=1928440455,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1926539846,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2559605274,osd_pg_creating=3631593561,osd_snap=3070752784,osdmap=3340858880}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=4} crc {logm=3528977248,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2013674078,mon_config_key=1905280744,monmap=3580681835,osd_metadata=2120554823,osd_pg_creating=3631593561,osd_snap=4068608915,osdmap=3178863923}) 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2463387216}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3617350823}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2859621371}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2708487244}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1777372139}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1734604153}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3475508013}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3921177846}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2292198265}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=3392744405}) 2024-08-19T09:52:41.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=1860778654}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=44} crc {osdmap=346932040}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=45} crc {osdmap=2880020118}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=44} crc {osdmap=1920787936}) 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=44} crc {osdmap=476541680}) 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.345+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=43} crc {osdmap=463485332}) 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.346+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.346+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=3419202990}) 2024-08-19T09:52:41.397 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:52:41.346+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=44} crc {osdmap=1575606342}) 2024-08-19T09:52:42.343 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:52:42.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:52:42.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.449+0000 7f3901e60640 10 monclient: get_monmap_and_config 2024-08-19T09:52:42.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.449+0000 7f3901e60640 10 monclient: build_initial_monmap 2024-08-19T09:52:42.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.449+0000 7f3901e60640 10 monclient: monmap: 2024-08-19T09:52:42.450 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:42.450163+0000 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:42.450163+0000 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:42.451 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:42.452 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 1 Processor -- start 2024-08-19T09:52:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 1 -- start start 2024-08-19T09:52:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient: init 2024-08-19T09:52:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient: _add_conns ranks=[1,7,2,0,4,8,5,3,6] 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient(hunting): picked mon.noname-f con 0x7f38fc0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient(hunting): picked mon.noname-i con 0x7f38fc0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient(hunting): picked mon.noname-b con 0x7f38fc0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc0a3c70 con 0x7f38fc0a6830 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc0a33d0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc0a36b0 con 0x7f38fc0694b0 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f3901e60640 10 monclient(hunting): authenticate will time out at 818.526652s 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f38fbfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f38fc0a46a0 0x7f38fc141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40560/0 (socket says 172.21.15.1:40560) 2024-08-19T09:52:42.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.450+0000 7f38fbfff640 1 -- 172.21.15.1:0/1620591617 learned_addr learned my addr 172.21.15.1:0/1620591617 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38fbfff640 1 -- 172.21.15.1:0/1620591617 >> v1:172.21.15.1:6790/0 conn(0x7f38fc0a46a0 legacy=0x7f38fc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38fb7fe640 1 -- 172.21.15.1:0/1620591617 >> v1:172.21.15.82:6789/0 conn(0x7f38fc0a6830 legacy=0x7f38fc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38f2ffd640 1 -- 172.21.15.1:0/1620591617 >> v1:172.21.15.82:6792/0 conn(0x7f38fc0694b0 legacy=0x7f38fc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651277460 0 0) 0x7f38fc0a33d0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): my global_id is 26543 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38e8003520 con 0x7f38fc0a46a0 2024-08-19T09:52:42.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 251593335 0 0) 0x7f38fc0a3c70 con 0x7f38fc0a6830 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): my global_id is 27928 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38fc0a33d0 con 0x7f38fc0a6830 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863887293 0 0) 0x7f38fc0a36b0 con 0x7f38fc0694b0 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient(hunting): my global_id is 134251 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38fc0a3c70 con 0x7f38fc0694b0 2024-08-19T09:52:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3144685633 0 0) 0x7f38e8003520 con 0x7f38fc0a46a0 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 >> v1:172.21.15.82:6792/0 conn(0x7f38fc0694b0 legacy=0x7f38fc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 >> v1:172.21.15.82:6789/0 conn(0x7f38fc0a6830 legacy=0x7f38fc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 monclient: found mon.noname-b 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc06dbb0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f3901e60640 5 monclient: authenticate success, global_id 26543 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f3901e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38f4001e60 con 0x7f38fc0a46a0 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: dump: 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f3901e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f38f4004c20 con 0x7f38fc0a46a0 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 1 -- 172.21.15.1:0/1620591617 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38f4005190 con 0x7f38fc0a46a0 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f38faffd640 10 monclient: dump: 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f3901e60640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.453+0000 7f3901e60640 4 set_mon_vals no callback set 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 10 monclient: shutdown 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 1 -- 172.21.15.1:0/1620591617 >> v1:172.21.15.1:6790/0 conn(0x7f38fc0a46a0 legacy=0x7f38fc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 1 -- 172.21.15.1:0/1620591617 shutdown_connections 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 1 -- 172.21.15.1:0/1620591617 >> 172.21.15.1:0/1620591617 conn(0x7f38fc0a9020 msgr2=0x7f38fc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 1 -- 172.21.15.1:0/1620591617 shutdown_connections 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 1 -- 172.21.15.1:0/1620591617 wait complete. 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 10 monclient: build_initial_monmap 2024-08-19T09:52:42.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.454+0000 7f3901e60640 10 monclient: monmap: 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 librados: starting msgr at 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 librados: starting objecter 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 Processor -- start 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 -- start start 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 librados: setting wanted keys 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 librados: calling monclient init 2024-08-19T09:52:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 10 monclient: init 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 10 monclient: _add_conns ranks=[1,2,7,3,0,5,8,4,6] 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 10 monclient(hunting): picked mon.noname-f con 0x7f38fc0694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 10 monclient(hunting): picked mon.noname-b con 0x7f38fc0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 10 monclient(hunting): picked mon.noname-i con 0x7f38fc0a6830 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.455+0000 7f3901e60640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc1cf260 con 0x7f38fc0694b0 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.456+0000 7f3901e60640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc137fa0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.456+0000 7f3901e60640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc138570 con 0x7f38fc0a6830 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.456+0000 7f3901e60640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.456+0000 7f3901e60640 10 monclient(hunting): authenticate will time out at 818.532033s 2024-08-19T09:52:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.456+0000 7f38f2ffd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f38fc0a46a0 0x7f38fc1ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40574/0 (socket says 172.21.15.1:40574) 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.456+0000 7f38f2ffd640 1 -- 172.21.15.1:0/283099073 learned_addr learned my addr 172.21.15.1:0/283099073 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38f2ffd640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.1:6790/0 conn(0x7f38fc0a46a0 legacy=0x7f38fc1ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38f8ff9640 10 client.?.objecter ms_handle_connect 0x7f38fc0a46a0 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38f8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38f8ff9640 10 client.?.objecter ms_handle_connect 0x7f38fc0694b0 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38fb7fe640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.82:6789/0 conn(0x7f38fc0694b0 legacy=0x7f38fc1ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38f8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38fbfff640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.82:6792/0 conn(0x7f38fc0a6830 legacy=0x7f38fc1d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.458+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:52:42.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 client.?.objecter ms_handle_connect 0x7f38fc0a6830 2024-08-19T09:52:42.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:42.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039993925 0 0) 0x7f38fc137fa0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): my global_id is 26552 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38dc004eb0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2851609500 0 0) 0x7f38fc138570 con 0x7f38fc0a6830 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): my global_id is 134260 2024-08-19T09:52:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38fc137fa0 con 0x7f38fc0a6830 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909465184 0 0) 0x7f38fc1cf260 con 0x7f38fc0694b0 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): my global_id is 27937 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38fc138570 con 0x7f38fc0694b0 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3060620840 0 0) 0x7f38dc004eb0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38fc1cf260 con 0x7f38fc0a46a0 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38e4003610 con 0x7f38fc0a46a0 2024-08-19T09:52:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4185840859 0 0) 0x7f38fc137fa0 con 0x7f38fc0a6830 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38dc004eb0 con 0x7f38fc0a6830 2024-08-19T09:52:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38f4005110 con 0x7f38fc0a6830 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3829429108 0 0) 0x7f38fc138570 con 0x7f38fc0694b0 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38fc137fa0 con 0x7f38fc0694b0 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38ec003610 con 0x7f38fc0694b0 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2359985793 0 0) 0x7f38fc1cf260 con 0x7f38fc0a46a0 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.82:6792/0 conn(0x7f38fc0a6830 legacy=0x7f38fc1d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.82:6789/0 conn(0x7f38fc0694b0 legacy=0x7f38fc1ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 monclient: found mon.b 2024-08-19T09:52:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc1378b0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient: _finish_auth 0 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f38f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f3901e60640 5 monclient: authenticate success, global_id 26552 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f3901e60640 10 monclient: _renew_subs 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f3901e60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.460+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f38fc136af0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 10 client.26552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 10 monclient: _renew_subs 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f38fc1d7dc0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 1 librados: init done 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f38e4003b80 con 0x7f38fc0a46a0 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38e40040b0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38fa7fc640 4 set_mon_vals no callback set 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 10 monclient: dump: 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.486 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f38e4004300 con 0x7f38fc0a46a0 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.461+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e0005080 con 0x7f38fc0a46a0 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.462+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(96..96 src has 1..96) ==== 7785+0+0 (unknown 606465447 0 0) 0x7f38e404eba0 con 0x7f38fc0a46a0 2024-08-19T09:52:42.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.462+0000 7f38f8ff9640 10 client.26552.objecter ms_dispatch 0x7f38fc000b90 osd_map(96..96 src has 1..96) 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.462+0000 7f38f8ff9640 3 client.26552.objecter handle_osd_map got epochs [96,96] > 0 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.462+0000 7f38f8ff9640 3 client.26552.objecter handle_osd_map decoding full epoch 96 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.462+0000 7f38f8ff9640 20 client.26552.objecter dump_active .. 0 homeless 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.462+0000 7f38fb7fe640 10 monclient: get_auth_request con 0x7f38dc041830 auth_method 0 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.464+0000 7f38f8ff9640 10 client.26552.objecter ms_handle_connect 0x7f38dc041830 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.466+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f38e4017620 con 0x7f38fc0a46a0 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.466+0000 7f38f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.466+0000 7f38f8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f3901e60640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f3901e60640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f3901e60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f38e0005370 con 0x7f38fc0a46a0 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f38f8ff9640 1 -- 172.21.15.1:0/283099073 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 1959149415) 0x7f38e4016f20 con 0x7f38fc0a46a0 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f38f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:52:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.721+0000 7f38f8ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:42.724 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 10 librados: watch_flush enter 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 10 librados: watch_flush exit 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 20 client.26552.objecter shutdown clearing up homeless session... 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 10 client.26552.objecter successfully canceled tick 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.82:6812/2549104415 conn(0x7f38dc041830 legacy=0x7f38dc043cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 10 monclient: shutdown 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 >> v1:172.21.15.1:6790/0 conn(0x7f38fc0a46a0 legacy=0x7f38fc1ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 shutdown_connections 2024-08-19T09:52:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 >> 172.21.15.1:0/283099073 conn(0x7f38fc0a9020 msgr2=0x7f38fc0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:42.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 shutdown_connections 2024-08-19T09:52:42.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.723+0000 7f3901e60640 1 -- 172.21.15.1:0/283099073 wait complete. 2024-08-19T09:52:42.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.724+0000 7f3901e60640 1 librados: shutdown 2024-08-19T09:52:42.735 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:52:42.735 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T09:52:42.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:52:42.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.834+0000 7fe1abfb1640 10 monclient: get_monmap_and_config 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.834+0000 7fe1abfb1640 10 monclient: build_initial_monmap 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.834+0000 7fe1abfb1640 10 monclient: monmap: 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:42.834951+0000 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:42.834951+0000 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:42.835 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:42.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.834+0000 7fe1abfb1640 1 Processor -- start 2024-08-19T09:52:42.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.834+0000 7fe1abfb1640 1 -- start start 2024-08-19T09:52:42.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.834+0000 7fe1abfb1640 10 monclient: init 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient: _add_conns ranks=[1,8,7,2,5,6,0,3,4] 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient(hunting): picked mon.noname-f con 0x7fe1a40a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient(hunting): picked mon.noname-e con 0x7fe1a40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient(hunting): picked mon.noname-i con 0x7fe1a413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a40a3b40 con 0x7fe1a40a67c0 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a40a8130 con 0x7fe1a413f5c0 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a406ede0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1abfb1640 10 monclient(hunting): authenticate will time out at 818.911322s 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1a954c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe1a40a6ba0 0x7fe1a40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43632/0 (socket says 172.21.15.1:43632) 2024-08-19T09:52:42.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.835+0000 7fe1a954c640 1 -- 172.21.15.1:0/1433163553 learned_addr learned my addr 172.21.15.1:0/1433163553 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a954c640 1 -- 172.21.15.1:0/1433163553 >> v1:172.21.15.1:6793/0 conn(0x7fe1a40a6ba0 legacy=0x7fe1a40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187162048 0 0) 0x7fe1a406ede0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a8d4b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a8d4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a8d4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a8d4b640 10 monclient(hunting): my global_id is 26945 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe18c003520 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1aa54e640 1 -- 172.21.15.1:0/1433163553 >> v1:172.21.15.82:6792/0 conn(0x7fe1a413f5c0 legacy=0x7fe1a41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.837+0000 7fe1a9d4d640 1 -- 172.21.15.1:0/1433163553 >> v1:172.21.15.82:6789/0 conn(0x7fe1a40a67c0 legacy=0x7fe1a40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2590187400 0 0) 0x7fe18c003520 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 >> v1:172.21.15.82:6792/0 conn(0x7fe1a413f5c0 legacy=0x7fe1a41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 >> v1:172.21.15.82:6789/0 conn(0x7fe1a40a67c0 legacy=0x7fe1a40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: _finish_hunting 0 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 monclient: found mon.noname-e 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a40a92a0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: _finish_auth 0 2024-08-19T09:52:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: _check_auth_tickets 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe198001ad0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 5 monclient: authenticate success, global_id 26945 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: dump: 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:42.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1980048a0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 1 -- 172.21.15.1:0/1433163553 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe198004e10 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1a8d4b640 10 monclient: dump: 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 4 set_mon_vals no callback set 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 10 monclient: shutdown 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 1 -- 172.21.15.1:0/1433163553 >> v1:172.21.15.1:6793/0 conn(0x7fe1a40a6ba0 legacy=0x7fe1a40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 1 -- 172.21.15.1:0/1433163553 shutdown_connections 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 1 -- 172.21.15.1:0/1433163553 >> 172.21.15.1:0/1433163553 conn(0x7fe1a40ac4b0 msgr2=0x7fe1a40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 1 -- 172.21.15.1:0/1433163553 shutdown_connections 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.838+0000 7fe1abfb1640 1 -- 172.21.15.1:0/1433163553 wait complete. 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 10 monclient: build_initial_monmap 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 10 monclient: monmap: 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 1 librados: starting msgr at 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 1 librados: starting objecter 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 1 Processor -- start 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 1 -- start start 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 1 librados: setting wanted keys 2024-08-19T09:52:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 1 librados: calling monclient init 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.839+0000 7fe1abfb1640 10 monclient: init 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient: _add_conns ranks=[2,3,7,4,5,8,1,6,0] 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient(hunting): picked mon.noname-b con 0x7fe1a40a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient(hunting): picked mon.noname-g con 0x7fe1a40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient(hunting): picked mon.noname-i con 0x7fe1a413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a413adf0 con 0x7fe1a40a67c0 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a413b3c0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a4139900 con 0x7fe1a413f5c0 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1abfb1640 10 monclient(hunting): authenticate will time out at 818.916302s 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1a9d4d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fe1a40a67c0 0x7fe1a413c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40582/0 (socket says 172.21.15.1:40582) 2024-08-19T09:52:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.840+0000 7fe1a9d4d640 1 -- 172.21.15.1:0/2004689578 learned_addr learned my addr 172.21.15.1:0/2004689578 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a9d4d640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6790/0 conn(0x7fe1a40a67c0 legacy=0x7fe1a413c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 client.?.objecter ms_handle_connect 0x7fe1a40a67c0 2024-08-19T09:52:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364175471 0 0) 0x7fe1a413adf0 con 0x7fe1a40a67c0 2024-08-19T09:52:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): my global_id is 26561 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1aa54e640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.82:6792/0 conn(0x7fe1a413f5c0 legacy=0x7fe1a41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe190004eb0 con 0x7fe1a40a67c0 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a954c640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.82:6790/0 conn(0x7fe1a40a6ba0 legacy=0x7fe1a413c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 client.?.objecter ms_handle_connect 0x7fe1a413f5c0 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 client.?.objecter ms_handle_connect 0x7fe1a40a6ba0 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3853838074 0 0) 0x7fe190004eb0 con 0x7fe1a40a67c0 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1a413adf0 con 0x7fe1a40a67c0 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe19c003990 con 0x7fe1a40a67c0 2024-08-19T09:52:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): dump: 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985054567 0 0) 0x7fe1a40a92a0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): my global_id is 134184 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe190004eb0 con 0x7fe1a40a6ba0 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661918214 0 0) 0x7fe1a4139900 con 0x7fe1a413f5c0 2024-08-19T09:52:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 10 monclient(hunting): my global_id is 134278 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.842+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a40a92a0 con 0x7fe1a413f5c0 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1999428108 0 0) 0x7fe1a413adf0 con 0x7fe1a40a67c0 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.82:6792/0 conn(0x7fe1a413f5c0 legacy=0x7fe1a41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.82:6790/0 conn(0x7fe1a40a6ba0 legacy=0x7fe1a413c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: _finish_hunting 0 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 monclient: found mon.b 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a413bc60 con 0x7fe1a40a67c0 2024-08-19T09:52:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: _finish_auth 0 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: _check_auth_tickets 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 5 monclient: authenticate success, global_id 26561 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 10 monclient: _renew_subs 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a413a700 con 0x7fe1a40a67c0 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 10 client.26561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 10 monclient: _renew_subs 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1a413cf80 con 0x7fe1a40a67c0 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 1 librados: init done 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe19c003f50 con 0x7fe1a40a67c0 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe19c004480 con 0x7fe1a40a67c0 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a3fff640 4 set_mon_vals no callback set 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1a27fc640 10 monclient: dump: 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.843+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1840065c0 con 0x7fe184002860 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe19c004620 con 0x7fe1a40a67c0 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(96..96 src has 1..96) ==== 7785+0+0 (unknown 606465447 0 0) 0x7fe19c04ef80 con 0x7fe1a40a67c0 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1a27fc640 10 client.26561.objecter ms_dispatch 0x7fe1a4000b90 osd_map(96..96 src has 1..96) 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1a27fc640 3 client.26561.objecter handle_osd_map got epochs [96,96] > 0 2024-08-19T09:52:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1a27fc640 3 client.26561.objecter handle_osd_map decoding full epoch 96 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1a27fc640 20 client.26561.objecter dump_active .. 0 homeless 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.844+0000 7fe1aa54e640 10 monclient: get_auth_request con 0x7fe1900411f0 auth_method 0 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a954c640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6789/0 conn(0x7fe184002860 legacy=0x7fe184004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 client.26561.objecter ms_handle_connect 0x7fe184002860 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 client.26561.objecter resend_mon_ops 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 323081479 0 0) 0x7fe1840065c0 con 0x7fe184002860 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient(hunting): my global_id is 134172 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a413adf0 con 0x7fe184002860 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3893709119 0 0) 0x7fe1a413adf0 con 0x7fe184002860 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1840065c0 con 0x7fe184002860 2024-08-19T09:52:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe198001d20 con 0x7fe184002860 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 10 monclient: dump: 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 444715331 0 0) 0x7fe1840065c0 con 0x7fe184002860 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.846+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe18400a160 con 0x7fe184002860 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.847+0000 7fe1a27fc640 10 client.26561.objecter ms_handle_connect 0x7fe1900411f0 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.847+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe198004a00 con 0x7fe184002860 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.847+0000 7fe1a27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.847+0000 7fe1a27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.847+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6789/0 conn(0x7fe184002860 legacy=0x7fe184004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.847+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6789/0 conn(0x7fe184002860 legacy=0x7fe184004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:42.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.853+0000 7fe1abfb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:42.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.853+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe184009e70 con 0x7fe184009130 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a954c640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6789/0 conn(0x7fe184009130 legacy=0x7fe1840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 client.26561.objecter ms_handle_connect 0x7fe184009130 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 client.26561.objecter resend_mon_ops 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961371908 0 0) 0x7fe184009e70 con 0x7fe184009130 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:42.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient(hunting): my global_id is 134181 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1840065c0 con 0x7fe184009130 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3273218939 0 0) 0x7fe1840065c0 con 0x7fe184009130 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe184009e70 con 0x7fe184009130 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe198005490 con 0x7fe184009130 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 10 monclient: dump: 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2251499146 0 0) 0x7fe184009e70 con 0x7fe184009130 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.855+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe18400c410 con 0x7fe184009130 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 2555539283) 0x7fe198008600 con 0x7fe184009130 2024-08-19T09:52:42.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1a27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1a27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6789/0 conn(0x7fe184009130 legacy=0x7fe1840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1a27fc640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6789/0 conn(0x7fe184009130 legacy=0x7fe1840095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:42.870 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19917, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:42.871 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:42.872 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:42.873 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:42.874 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:42.875 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:42.876 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.877 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.880 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.881 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.882 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.884 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:42.889 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:42.890 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 10 librados: watch_flush enter 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 10 librados: watch_flush exit 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 20 client.26561.objecter shutdown clearing up homeless session... 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 10 client.26561.objecter successfully canceled tick 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe1900411f0 legacy=0x7fe1900436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 10 monclient: shutdown 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 >> v1:172.21.15.1:6790/0 conn(0x7fe1a40a67c0 legacy=0x7fe1a413c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1aa54e640 1 -- 172.21.15.1:0/2004689578 reap_dead start 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1aa54e640 1 -- 172.21.15.1:0/2004689578 reap_dead start 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 shutdown_connections 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 >> 172.21.15.1:0/2004689578 conn(0x7fe1a40ac4b0 msgr2=0x7fe1a413ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 shutdown_connections 2024-08-19T09:52:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 -- 172.21.15.1:0/2004689578 wait complete. 2024-08-19T09:52:42.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:42.856+0000 7fe1abfb1640 1 librados: shutdown 2024-08-19T09:52:42.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:42.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:52:42.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: seed 26120 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2050 ms) 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-08-19T09:52:42.907 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2063 ms) 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2019 ms) 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2093 ms) 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4129 ms) 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (12354 ms total) 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4128 ms) 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-08-19T09:52:42.908 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3253 ms) 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6029 ms) 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3970 ms) 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2073 ms) 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5847-g79766939/rpm/el9/BUILD/ceph-19.0.0-5847-g79766939/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi082-26120-7 2024-08-19T09:52:42.909 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: waiting for snaps to purge 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18307 ms) 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (37762 ms total) 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (12 ms) 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (6451 ms) 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (6464 ms total) 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-08-19T09:52:42.910 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-08-19T09:52:42.911 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (10046 ms) 2024-08-19T09:52:42.911 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-08-19T09:52:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.021+0000 7fb44e5c9640 10 monclient: get_monmap_and_config 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.021+0000 7fb44e5c9640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.021+0000 7fb44e5c9640 10 monclient: monmap: 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:43.022713+0000 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:43.022713+0000 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.024 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.022+0000 7fb44e5c9640 1 Processor -- start 2024-08-19T09:52:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.022+0000 7fb44e5c9640 1 -- start start 2024-08-19T09:52:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.022+0000 7fb44e5c9640 10 monclient: init 2024-08-19T09:52:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.022+0000 7fb44e5c9640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.022+0000 7fb44e5c9640 10 monclient: _add_conns ranks=[8,1,3,7,6,4,0,2,5] 2024-08-19T09:52:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 10 monclient(hunting): picked mon.noname-e con 0x7fb4480a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 10 monclient(hunting): picked mon.noname-f con 0x7fb4480a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 10 monclient(hunting): picked mon.noname-g con 0x7fb44813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a3b40 con 0x7fb4480a6ba0 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a8130 con 0x7fb44813f5c0 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44806ede0 con 0x7fb4480a67c0 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb44e5c9640 10 monclient(hunting): authenticate will time out at 819.099077s 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb447fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fb4480a67c0 0x7fb4480a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43634/0 (socket says 172.21.15.1:43634) 2024-08-19T09:52:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.023+0000 7fb447fff640 1 -- 172.21.15.1:0/4204874905 learned_addr learned my addr 172.21.15.1:0/4204874905 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb447fff640 1 -- 172.21.15.1:0/4204874905 >> v1:172.21.15.1:6793/0 conn(0x7fb4480a67c0 legacy=0x7fb4480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb44cb66640 1 -- 172.21.15.1:0/4204874905 >> v1:172.21.15.82:6790/0 conn(0x7fb44813f5c0 legacy=0x7fb4481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb4477fe640 1 -- 172.21.15.1:0/4204874905 >> v1:172.21.15.82:6789/0 conn(0x7fb4480a6ba0 legacy=0x7fb4480a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570247925 0 0) 0x7fb44806ede0 con 0x7fb4480a67c0 2024-08-19T09:52:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): my global_id is 26954 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb430003520 con 0x7fb4480a67c0 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2694768895 0 0) 0x7fb4480a3b40 con 0x7fb4480a6ba0 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): my global_id is 27955 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb44806ede0 con 0x7fb4480a6ba0 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793264031 0 0) 0x7fb4480a8130 con 0x7fb44813f5c0 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient(hunting): my global_id is 134193 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4480a3b40 con 0x7fb44813f5c0 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3951324634 0 0) 0x7fb430003520 con 0x7fb4480a67c0 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 >> v1:172.21.15.82:6790/0 conn(0x7fb44813f5c0 legacy=0x7fb4481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 >> v1:172.21.15.82:6789/0 conn(0x7fb4480a6ba0 legacy=0x7fb4480a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 monclient: found mon.noname-e 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4480a92a0 con 0x7fb4480a67c0 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb438001ad0 con 0x7fb4480a67c0 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: dump: 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb44e5c9640 5 monclient: authenticate success, global_id 26954 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb44e5c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4380048a0 con 0x7fb4480a67c0 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 1 -- 172.21.15.1:0/4204874905 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb438004e10 con 0x7fb4480a67c0 2024-08-19T09:52:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb44e5c9640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb44e5c9640 4 set_mon_vals no callback set 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.025+0000 7fb446ffd640 10 monclient: dump: 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 10 monclient: shutdown 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 1 -- 172.21.15.1:0/4204874905 >> v1:172.21.15.1:6793/0 conn(0x7fb4480a67c0 legacy=0x7fb4480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 1 -- 172.21.15.1:0/4204874905 shutdown_connections 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 1 -- 172.21.15.1:0/4204874905 >> 172.21.15.1:0/4204874905 conn(0x7fb4480ac4b0 msgr2=0x7fb4480a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 1 -- 172.21.15.1:0/4204874905 shutdown_connections 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 1 -- 172.21.15.1:0/4204874905 wait complete. 2024-08-19T09:52:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.026+0000 7fb44e5c9640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient: monmap: 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 librados: starting msgr at 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 librados: starting objecter 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 Processor -- start 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 -- start start 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 librados: setting wanted keys 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 librados: calling monclient init 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient: init 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient: _add_conns ranks=[5,1,0,3,7,2,6,8,4] 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient(hunting): picked mon.noname-h con 0x7fb4480a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient(hunting): picked mon.noname-f con 0x7fb4480a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient(hunting): picked mon.noname-a con 0x7fb44813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb448139a70 con 0x7fb44813f5c0 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a9cc0 con 0x7fb4480a6ba0 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44813bdb0 con 0x7fb4480a67c0 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.027+0000 7fb44e5c9640 10 monclient(hunting): authenticate will time out at 819.103959s 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.028+0000 7fb44cb66640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb44813f5c0 0x7fb4481d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58990/0 (socket says 172.21.15.1:58990) 2024-08-19T09:52:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.028+0000 7fb44cb66640 1 -- 172.21.15.1:0/3474967911 learned_addr learned my addr 172.21.15.1:0/3474967911 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb447fff640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.82:6791/0 conn(0x7fb4480a67c0 legacy=0x7fb4481ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 client.?.objecter ms_handle_connect 0x7fb4480a67c0 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44cb66640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6789/0 conn(0x7fb44813f5c0 legacy=0x7fb4481d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 client.?.objecter ms_handle_connect 0x7fb44813f5c0 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 client.?.objecter ms_handle_connect 0x7fb4480a6ba0 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb4477fe640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.82:6789/0 conn(0x7fb4480a6ba0 legacy=0x7fb4481ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130333882 0 0) 0x7fb44813bdb0 con 0x7fb4480a67c0 2024-08-19T09:52:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): my global_id is 26168 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb434004eb0 con 0x7fb4480a67c0 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 356255922 0 0) 0x7fb448139a70 con 0x7fb44813f5c0 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): my global_id is 134217 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb44813bdb0 con 0x7fb44813f5c0 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105398239 0 0) 0x7fb44806ede0 con 0x7fb4480a6ba0 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): my global_id is 27964 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb448139a70 con 0x7fb4480a6ba0 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2037070252 0 0) 0x7fb448139a70 con 0x7fb4480a6ba0 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb44806ede0 con 0x7fb4480a6ba0 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb43c001d00 con 0x7fb4480a6ba0 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1950860435 0 0) 0x7fb44813bdb0 con 0x7fb44813f5c0 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb448139a70 con 0x7fb44813f5c0 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3005156494 0 0) 0x7fb434004eb0 con 0x7fb4480a67c0 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb44813bdb0 con 0x7fb4480a67c0 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb440003990 con 0x7fb44813f5c0 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb438004d90 con 0x7fb4480a67c0 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient(hunting): dump: 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2253855533 0 0) 0x7fb44806ede0 con 0x7fb4480a6ba0 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.82:6791/0 conn(0x7fb4480a67c0 legacy=0x7fb4481ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6789/0 conn(0x7fb44813f5c0 legacy=0x7fb4481d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 monclient: found mon.f 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4481cf020 con 0x7fb4480a6ba0 2024-08-19T09:52:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb444ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 5 monclient: authenticate success, global_id 27964 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 10 monclient: _renew_subs 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb44813af40 con 0x7fb4480a6ba0 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 10 client.27964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 10 monclient: _renew_subs 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb44813a880 con 0x7fb4480a6ba0 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.030+0000 7fb44e5c9640 1 librados: init done 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb43c003980 con 0x7fb4480a6ba0 2024-08-19T09:52:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb43c003e70 con 0x7fb4480a6ba0 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb4467fc640 4 set_mon_vals no callback set 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 10 monclient: dump: 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb44e5c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb4280065c0 con 0x7fb428002860 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.031+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb43c004090 con 0x7fb4480a6ba0 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.032+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(97..97 src has 1..97) ==== 6672+0+0 (unknown 3727910313 0 0) 0x7fb43c04e580 con 0x7fb4480a6ba0 2024-08-19T09:52:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.032+0000 7fb444ff9640 10 client.27964.objecter ms_dispatch 0x7fb448000b90 osd_map(97..97 src has 1..97) 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.032+0000 7fb444ff9640 3 client.27964.objecter handle_osd_map got epochs [97,97] > 0 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.032+0000 7fb444ff9640 3 client.27964.objecter handle_osd_map decoding full epoch 97 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.032+0000 7fb444ff9640 20 client.27964.objecter dump_active .. 0 homeless 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.032+0000 7fb44cb66640 10 monclient: get_auth_request con 0x7fb4340419e0 auth_method 0 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.033+0000 7fb447fff640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6790/0 conn(0x7fb428002860 legacy=0x7fb428004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.033+0000 7fb444ff9640 10 client.27964.objecter ms_handle_connect 0x7fb428002860 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.033+0000 7fb444ff9640 10 client.27964.objecter resend_mon_ops 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828801994 0 0) 0x7fb4280065c0 con 0x7fb428002860 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient(hunting): my global_id is 26579 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb44806ede0 con 0x7fb428002860 2024-08-19T09:52:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4236144565 0 0) 0x7fb44806ede0 con 0x7fb428002860 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4280065c0 con 0x7fb428002860 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb4380048a0 con 0x7fb428002860 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient: dump: 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2712780656 0 0) 0x7fb4280065c0 con 0x7fb428002860 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb42800a160 con 0x7fb428002860 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 client.27964.objecter ms_handle_connect 0x7fb4340419e0 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb438005490 con 0x7fb428002860 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6790/0 conn(0x7fb428002860 legacy=0x7fb428004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.034+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6790/0 conn(0x7fb428002860 legacy=0x7fb428004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.040+0000 7fb44e5c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.040+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb428009e70 con 0x7fb428009130 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb447fff640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6790/0 conn(0x7fb428009130 legacy=0x7fb4280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 client.27964.objecter ms_handle_connect 0x7fb428009130 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 client.27964.objecter resend_mon_ops 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479536943 0 0) 0x7fb428009e70 con 0x7fb428009130 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient(hunting): my global_id is 26588 2024-08-19T09:52:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4280065c0 con 0x7fb428009130 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2711294966 0 0) 0x7fb4280065c0 con 0x7fb428009130 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb428009e70 con 0x7fb428009130 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb438005ca0 con 0x7fb428009130 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient: dump: 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.061 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3482282309 0 0) 0x7fb428009e70 con 0x7fb428009130 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb42800c410 con 0x7fb428009130 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2933710057) 0x7fb438006120 con 0x7fb428009130 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:43.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6790/0 conn(0x7fb428009130 legacy=0x7fb4280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.043+0000 7fb444ff9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.1:6790/0 conn(0x7fb428009130 legacy=0x7fb4280095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 169333, 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:43.063 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.066 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:43.067 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.068 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.071 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.072 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.074 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:43.083 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 10 librados: watch_flush enter 2024-08-19T09:52:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 10 librados: watch_flush exit 2024-08-19T09:52:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 20 client.27964.objecter shutdown clearing up homeless session... 2024-08-19T09:52:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 10 client.27964.objecter successfully canceled tick 2024-08-19T09:52:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb4340419e0 legacy=0x7fb434043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 10 monclient: shutdown 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 >> v1:172.21.15.82:6789/0 conn(0x7fb4480a6ba0 legacy=0x7fb4481ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44cb66640 1 -- 172.21.15.1:0/3474967911 reap_dead start 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44cb66640 1 -- 172.21.15.1:0/3474967911 reap_dead start 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 shutdown_connections 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 >> 172.21.15.1:0/3474967911 conn(0x7fb4480ac4b0 msgr2=0x7fb4481421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 shutdown_connections 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 -- 172.21.15.1:0/3474967911 wait complete. 2024-08-19T09:52:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.044+0000 7fb44e5c9640 1 librados: shutdown 2024-08-19T09:52:43.086 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:43.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:52:43.087 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:52:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.185+0000 7f3ca7e20640 10 monclient: get_monmap_and_config 2024-08-19T09:52:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.185+0000 7f3ca7e20640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.185+0000 7f3ca7e20640 10 monclient: monmap: 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:43.186560+0000 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:43.186560+0000 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.188 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 1 Processor -- start 2024-08-19T09:52:43.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 1 -- start start 2024-08-19T09:52:43.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient: init 2024-08-19T09:52:43.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient: _add_conns ranks=[3,5,8,0,7,6,4,1,2] 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient(hunting): picked mon.noname-g con 0x7f3ca00a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient(hunting): picked mon.noname-h con 0x7f3ca00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient(hunting): picked mon.noname-e con 0x7f3ca013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca00a3b40 con 0x7f3ca00a67c0 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca00a8130 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca006ede0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.186+0000 7f3ca7e20640 10 monclient(hunting): authenticate will time out at 819.262954s 2024-08-19T09:52:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.187+0000 7f3ca63bd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f3ca013f5c0 0x7f3ca01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43650/0 (socket says 172.21.15.1:43650) 2024-08-19T09:52:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.187+0000 7f3ca63bd640 1 -- 172.21.15.1:0/2999114520 learned_addr learned my addr 172.21.15.1:0/2999114520 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca63bd640 1 -- 172.21.15.1:0/2999114520 >> v1:172.21.15.1:6793/0 conn(0x7f3ca013f5c0 legacy=0x7f3ca01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca53bb640 1 -- 172.21.15.1:0/2999114520 >> v1:172.21.15.82:6791/0 conn(0x7f3ca00a6ba0 legacy=0x7f3ca00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca5bbc640 1 -- 172.21.15.1:0/2999114520 >> v1:172.21.15.82:6790/0 conn(0x7f3ca00a67c0 legacy=0x7f3ca00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341615740 0 0) 0x7f3ca006ede0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): my global_id is 26963 2024-08-19T09:52:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c8c003520 con 0x7f3ca013f5c0 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915013339 0 0) 0x7f3ca00a8130 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient(hunting): my global_id is 26177 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ca006ede0 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 291001824 0 0) 0x7f3c8c003520 con 0x7f3ca013f5c0 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 >> v1:172.21.15.82:6791/0 conn(0x7f3ca00a6ba0 legacy=0x7f3ca00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 >> v1:172.21.15.82:6790/0 conn(0x7f3ca00a67c0 legacy=0x7f3ca00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 monclient: found mon.noname-e 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca00a92a0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c98001e50 con 0x7f3ca013f5c0 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca7e20640 5 monclient: authenticate success, global_id 26963 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca7e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: dump: 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca7e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c98004c20 con 0x7f3ca013f5c0 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 1 -- 172.21.15.1:0/2999114520 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c98005190 con 0x7f3ca013f5c0 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca7e20640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca7e20640 4 set_mon_vals no callback set 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.189+0000 7f3ca4bba640 10 monclient: dump: 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 10 monclient: shutdown 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 1 -- 172.21.15.1:0/2999114520 >> v1:172.21.15.1:6793/0 conn(0x7f3ca013f5c0 legacy=0x7f3ca01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 1 -- 172.21.15.1:0/2999114520 shutdown_connections 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 1 -- 172.21.15.1:0/2999114520 >> 172.21.15.1:0/2999114520 conn(0x7f3ca00ac4b0 msgr2=0x7f3ca00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 1 -- 172.21.15.1:0/2999114520 shutdown_connections 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 1 -- 172.21.15.1:0/2999114520 wait complete. 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.190+0000 7f3ca7e20640 10 monclient: monmap: 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 librados: starting msgr at 2024-08-19T09:52:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 librados: starting objecter 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 Processor -- start 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 -- start start 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 librados: setting wanted keys 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 librados: calling monclient init 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient: init 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient: _add_conns ranks=[1,0,7,8,2,3,5,6,4] 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient(hunting): picked mon.noname-f con 0x7f3ca00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca00a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient(hunting): picked mon.noname-i con 0x7f3ca013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca013a800 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca00a1c00 con 0x7f3ca00a67c0 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca00a1dd0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca7e20640 10 monclient(hunting): authenticate will time out at 819.267784s 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca53bb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ca00a6ba0 0x7f3ca00a01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59002/0 (socket says 172.21.15.1:59002) 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.191+0000 7f3ca53bb640 1 -- 172.21.15.1:0/1301220299 learned_addr learned my addr 172.21.15.1:0/1301220299 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca53bb640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6789/0 conn(0x7f3ca00a6ba0 legacy=0x7f3ca00a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 client.?.objecter ms_handle_connect 0x7f3ca00a6ba0 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 client.?.objecter ms_handle_connect 0x7f3ca013f5c0 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca63bd640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.82:6792/0 conn(0x7f3ca013f5c0 legacy=0x7f3ca00a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 client.?.objecter ms_handle_connect 0x7f3ca00a67c0 2024-08-19T09:52:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca5bbc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.82:6789/0 conn(0x7f3ca00a67c0 legacy=0x7f3ca009fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61452614 0 0) 0x7f3ca013a800 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): my global_id is 134226 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c84004eb0 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2537021370 0 0) 0x7f3ca00a1dd0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): my global_id is 134287 2024-08-19T09:52:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ca013a800 con 0x7f3ca013f5c0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366285408 0 0) 0x7f3ca00a1c00 con 0x7f3ca00a67c0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): my global_id is 27973 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ca00a1dd0 con 0x7f3ca00a67c0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 719945120 0 0) 0x7f3c84004eb0 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ca00a1c00 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c90003610 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): dump: 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.212 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1598332924 0 0) 0x7f3ca013a800 con 0x7f3ca013f5c0 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c84004eb0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c980020a0 con 0x7f3ca013f5c0 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient(hunting): dump: 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3908299729 0 0) 0x7f3ca00a1dd0 con 0x7f3ca00a67c0 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ca013a800 con 0x7f3ca00a67c0 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1303993321 0 0) 0x7f3ca00a1c00 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.82:6792/0 conn(0x7f3ca013f5c0 legacy=0x7f3ca00a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.82:6789/0 conn(0x7f3ca00a67c0 legacy=0x7f3ca009fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 monclient: found mon.a 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca01da290 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 5 monclient: authenticate success, global_id 134226 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 10 monclient: _renew_subs 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca01d7260 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 10 client.134226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 10 monclient: _renew_subs 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ca01d77c0 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3ca7e20640 1 librados: init done 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c90003bf0 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c900040e0 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c97fff640 4 set_mon_vals no callback set 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.194+0000 7f3c967fc640 10 monclient: dump: 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.218 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.195+0000 7f3ca7e20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.195+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3c880065c0 con 0x7f3c88002860 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.195+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3c90004280 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.196+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(97..97 src has 1..97) ==== 6672+0+0 (unknown 3727910313 0 0) 0x7f3c9004e700 con 0x7f3ca00a6ba0 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.196+0000 7f3c967fc640 10 client.134226.objecter ms_dispatch 0x7f3ca0000b90 osd_map(97..97 src has 1..97) 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.196+0000 7f3c967fc640 3 client.134226.objecter handle_osd_map got epochs [97,97] > 0 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.196+0000 7f3c967fc640 3 client.134226.objecter handle_osd_map decoding full epoch 97 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.196+0000 7f3c967fc640 20 client.134226.objecter dump_active .. 0 homeless 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.196+0000 7f3ca63bd640 10 monclient: get_auth_request con 0x7f3c84041850 auth_method 0 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.197+0000 7f3c967fc640 10 client.134226.objecter ms_handle_connect 0x7f3c88002860 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.197+0000 7f3c967fc640 10 client.134226.objecter resend_mon_ops 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.197+0000 7f3ca5bbc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6791/0 conn(0x7f3c88002860 legacy=0x7f3c88004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839850663 0 0) 0x7f3c880065c0 con 0x7f3c88002860 2024-08-19T09:52:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient(hunting): my global_id is 26509 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ca00a1c00 con 0x7f3c88002860 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1314336004 0 0) 0x7f3ca00a1c00 con 0x7f3c88002860 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c880065c0 con 0x7f3c88002860 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c9c003e10 con 0x7f3c88002860 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient: dump: 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 460953424 0 0) 0x7f3c880065c0 con 0x7f3c88002860 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c8800a160 con 0x7f3c88002860 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 client.134226.objecter ms_handle_connect 0x7f3c84041850 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3c9c004070 con 0x7f3c88002860 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6791/0 conn(0x7f3c88002860 legacy=0x7f3c88004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.198+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6791/0 conn(0x7f3c88002860 legacy=0x7f3c88004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.204+0000 7f3ca7e20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.204+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3c88009e50 con 0x7f3c88009110 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3ca5bbc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6791/0 conn(0x7f3c88009110 legacy=0x7f3c880095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 client.134226.objecter ms_handle_connect 0x7f3c88009110 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 client.134226.objecter resend_mon_ops 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4250760269 0 0) 0x7f3c88009e50 con 0x7f3c88009110 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient(hunting): my global_id is 26518 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c880065c0 con 0x7f3c88009110 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3255892081 0 0) 0x7f3c880065c0 con 0x7f3c88009110 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c88009e50 con 0x7f3c88009110 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3c9c005b00 con 0x7f3c88009110 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient: dump: 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1654504767 0 0) 0x7f3c88009e50 con 0x7f3c88009110 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c8800c3a0 con 0x7f3c88009110 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3222495638) 0x7f3c9c006fb0 con 0x7f3c88009110 2024-08-19T09:52:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6791/0 conn(0x7f3c88009110 legacy=0x7f3c880095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.207+0000 7f3c967fc640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6791/0 conn(0x7f3c88009110 legacy=0x7f3c880095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:43.227 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:43.228 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 169478, 2024-08-19T09:52:43.228 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:43.228 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:43.229 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.230 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:43.232 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.233 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.234 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.235 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.236 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.237 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.238 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.239 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.240 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.241 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.247 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:43.248 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 10 librados: watch_flush enter 2024-08-19T09:52:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 10 librados: watch_flush exit 2024-08-19T09:52:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 20 client.134226.objecter shutdown clearing up homeless session... 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 10 client.134226.objecter successfully canceled tick 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3c84041850 legacy=0x7f3c84043d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 10 monclient: shutdown 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 >> v1:172.21.15.1:6789/0 conn(0x7f3ca00a6ba0 legacy=0x7f3ca00a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca63bd640 1 -- 172.21.15.1:0/1301220299 reap_dead start 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca63bd640 1 -- 172.21.15.1:0/1301220299 reap_dead start 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 shutdown_connections 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 >> 172.21.15.1:0/1301220299 conn(0x7f3ca00ac4b0 msgr2=0x7f3ca0138910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 shutdown_connections 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 -- 172.21.15.1:0/1301220299 wait complete. 2024-08-19T09:52:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.208+0000 7f3ca7e20640 1 librados: shutdown 2024-08-19T09:52:43.252 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:43.252 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:52:43.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:52:43.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.350+0000 7fb60be60640 10 monclient: get_monmap_and_config 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.350+0000 7fb60be60640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.350+0000 7fb60be60640 10 monclient: monmap: 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:43.351451+0000 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:43.351451+0000 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.352 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 1 Processor -- start 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 1 -- start start 2024-08-19T09:52:43.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient: init 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient: _add_conns ranks=[2,1,3,7,8,4,6,0,5] 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient(hunting): picked mon.noname-b con 0x7fb6040a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient(hunting): picked mon.noname-f con 0x7fb6040a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient(hunting): picked mon.noname-g con 0x7fb60413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6040a3b40 con 0x7fb6040a6ba0 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6040a8130 con 0x7fb6040a67c0 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60406ede0 con 0x7fb60413f5c0 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb60be60640 10 monclient(hunting): authenticate will time out at 819.427837s 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb609bfc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb6040a67c0 0x7fb6040a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40620/0 (socket says 172.21.15.1:40620) 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.351+0000 7fb609bfc640 1 -- 172.21.15.1:0/3890215936 learned_addr learned my addr 172.21.15.1:0/3890215936 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb609bfc640 1 -- 172.21.15.1:0/3890215936 >> v1:172.21.15.1:6790/0 conn(0x7fb6040a67c0 legacy=0x7fb6040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 661449990 0 0) 0x7fb6040a8130 con 0x7fb6040a67c0 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient(hunting): my global_id is 26597 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5f0003520 con 0x7fb6040a67c0 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb60a3fd640 1 -- 172.21.15.1:0/3890215936 >> v1:172.21.15.82:6790/0 conn(0x7fb60413f5c0 legacy=0x7fb6041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb6093fb640 1 -- 172.21.15.1:0/3890215936 >> v1:172.21.15.82:6789/0 conn(0x7fb6040a6ba0 legacy=0x7fb6040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 743468355 0 0) 0x7fb5f0003520 con 0x7fb6040a67c0 2024-08-19T09:52:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 >> v1:172.21.15.82:6790/0 conn(0x7fb60413f5c0 legacy=0x7fb6041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 >> v1:172.21.15.82:6789/0 conn(0x7fb6040a6ba0 legacy=0x7fb6040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 monclient: found mon.noname-b 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6040a92a0 con 0x7fb6040a67c0 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb600001ae0 con 0x7fb6040a67c0 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: dump: 2024-08-19T09:52:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb60be60640 5 monclient: authenticate success, global_id 26597 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb60be60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6000048a0 con 0x7fb6040a67c0 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 1 -- 172.21.15.1:0/3890215936 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb600004dd0 con 0x7fb6040a67c0 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb60be60640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb60be60640 4 set_mon_vals no callback set 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.354+0000 7fb608bfa640 10 monclient: dump: 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.362 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.363 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 10 monclient: shutdown 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 -- 172.21.15.1:0/3890215936 >> v1:172.21.15.1:6790/0 conn(0x7fb6040a67c0 legacy=0x7fb6040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 -- 172.21.15.1:0/3890215936 shutdown_connections 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 -- 172.21.15.1:0/3890215936 >> 172.21.15.1:0/3890215936 conn(0x7fb6040ac4b0 msgr2=0x7fb6040a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 -- 172.21.15.1:0/3890215936 shutdown_connections 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 -- 172.21.15.1:0/3890215936 wait complete. 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 10 monclient: monmap: 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 librados: starting msgr at 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.355+0000 7fb60be60640 1 librados: starting objecter 2024-08-19T09:52:43.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 Processor -- start 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 -- start start 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 librados: setting wanted keys 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 librados: calling monclient init 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient: init 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient: _add_conns ranks=[5,7,4,6,1,0,2,8,3] 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient(hunting): picked mon.noname-h con 0x7fb6040a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient(hunting): picked mon.noname-i con 0x7fb6040a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient(hunting): picked mon.noname-c con 0x7fb60413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb604139a70 con 0x7fb60413f5c0 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60413bdb0 con 0x7fb6040a67c0 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6041cf020 con 0x7fb6040a6ba0 2024-08-19T09:52:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60be60640 10 monclient(hunting): authenticate will time out at 819.432582s 2024-08-19T09:52:43.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60a3fd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb60413f5c0 0x7fb6041d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:36546/0 (socket says 172.21.15.1:36546) 2024-08-19T09:52:43.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.356+0000 7fb60a3fd640 1 -- 172.21.15.1:0/3569695282 learned_addr learned my addr 172.21.15.1:0/3569695282 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb60a3fd640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6791/0 conn(0x7fb60413f5c0 legacy=0x7fb6041d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 client.?.objecter ms_handle_connect 0x7fb60413f5c0 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696710361 0 0) 0x7fb604139a70 con 0x7fb60413f5c0 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 monclient(hunting): my global_id is 26527 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb609bfc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.82:6791/0 conn(0x7fb6040a67c0 legacy=0x7fb6041ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb6093fb640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.82:6792/0 conn(0x7fb6040a6ba0 legacy=0x7fb6041ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5e8004eb0 con 0x7fb60413f5c0 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 client.?.objecter ms_handle_connect 0x7fb6040a67c0 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 client.?.objecter ms_handle_connect 0x7fb6040a6ba0 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.358+0000 7fb5fa7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb6093fb640 1 --1- 172.21.15.1:0/3569695282 >> v1:172.21.15.82:6792/0 conn(0x7fb6040a6ba0 0x7fb6041ce910 :-1 s=OPENED pgs=101 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fb6041cf020 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1045124473 0 0) 0x7fb60413bdb0 con 0x7fb6040a67c0 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): my global_id is 26186 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb604139a70 con 0x7fb6040a67c0 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3383542935 0 0) 0x7fb5e8004eb0 con 0x7fb60413f5c0 2024-08-19T09:52:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb60413bdb0 con 0x7fb60413f5c0 2024-08-19T09:52:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb5fc003990 con 0x7fb60413f5c0 2024-08-19T09:52:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:52:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): dump: 2024-08-19T09:52:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.377 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2430409203 0 0) 0x7fb60413bdb0 con 0x7fb60413f5c0 2024-08-19T09:52:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.82:6792/0 conn(0x7fb6040a6ba0 legacy=0x7fb6041ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.82:6791/0 conn(0x7fb6040a67c0 legacy=0x7fb6041ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 monclient: found mon.c 2024-08-19T09:52:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60413af40 con 0x7fb60413f5c0 2024-08-19T09:52:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 5 monclient: authenticate success, global_id 26527 2024-08-19T09:52:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 10 monclient: _renew_subs 2024-08-19T09:52:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418745321 0 0) 0x7fb6041cf020 con 0x7fb6040a6ba0 2024-08-19T09:52:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb60413bad0 con 0x7fb60413f5c0 2024-08-19T09:52:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:52:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 10 client.26527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 10 monclient: _renew_subs 2024-08-19T09:52:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:52:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fb60413d1e0 con 0x7fb60413f5c0 2024-08-19T09:52:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb60be60640 1 librados: init done 2024-08-19T09:52:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb5fc003e90 con 0x7fb60413f5c0 2024-08-19T09:52:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb5fc0043c0 con 0x7fb60413f5c0 2024-08-19T09:52:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fa7fc640 10 monclient: dump: 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.359+0000 7fb5fbfff640 4 set_mon_vals no callback set 2024-08-19T09:52:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb60be60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb6040a9cc0 con 0x7fb60402e340 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb5fc004560 con 0x7fb60413f5c0 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(97..97 src has 1..97) ==== 6672+0+0 (unknown 3727910313 0 0) 0x7fb5fc04e970 con 0x7fb60413f5c0 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb5fa7fc640 10 client.26527.objecter ms_dispatch 0x7fb604000b90 osd_map(97..97 src has 1..97) 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb5fa7fc640 3 client.26527.objecter handle_osd_map got epochs [97,97] > 0 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb5fa7fc640 3 client.26527.objecter handle_osd_map decoding full epoch 97 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb5fa7fc640 20 client.26527.objecter dump_active .. 0 homeless 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.360+0000 7fb6093fb640 10 monclient: get_auth_request con 0x7fb5e8041010 auth_method 0 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 client.26527.objecter ms_handle_connect 0x7fb60402e340 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 client.26527.objecter resend_mon_ops 2024-08-19T09:52:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb609bfc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6792/0 conn(0x7fb60402e340 legacy=0x7fb604063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565441386 0 0) 0x7fb6040a9cc0 con 0x7fb60402e340 2024-08-19T09:52:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient(hunting): my global_id is 26484 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6041cf020 con 0x7fb60402e340 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1375626090 0 0) 0x7fb6041cf020 con 0x7fb60402e340 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6040a9cc0 con 0x7fb60402e340 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb6000048a0 con 0x7fb60402e340 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 10 monclient: dump: 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.387 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2162559172 0 0) 0x7fb6040a9cc0 con 0x7fb60402e340 2024-08-19T09:52:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.362+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6040a29a0 con 0x7fb60402e340 2024-08-19T09:52:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.363+0000 7fb5fa7fc640 10 client.26527.objecter ms_handle_connect 0x7fb5e8041010 2024-08-19T09:52:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.363+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb600004b80 con 0x7fb60402e340 2024-08-19T09:52:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.363+0000 7fb5fa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.363+0000 7fb5fa7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.363+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6792/0 conn(0x7fb60402e340 legacy=0x7fb604063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.363+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6792/0 conn(0x7fb60402e340 legacy=0x7fb604063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.369+0000 7fb60be60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.369+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb604068f60 con 0x7fb60409ea60 2024-08-19T09:52:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb609bfc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6792/0 conn(0x7fb60409ea60 legacy=0x7fb6041dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 10 client.26527.objecter ms_handle_connect 0x7fb60409ea60 2024-08-19T09:52:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 10 client.26527.objecter resend_mon_ops 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363859012 0 0) 0x7fb604068f60 con 0x7fb60409ea60 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 10 monclient(hunting): my global_id is 26493 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.371+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6040a9cc0 con 0x7fb60409ea60 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1155587587 0 0) 0x7fb6040a9cc0 con 0x7fb60409ea60 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb604068f60 con 0x7fb60409ea60 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb600005450 con 0x7fb60409ea60 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 10 monclient: dump: 2024-08-19T09:52:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.395 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 84464641 0 0) 0x7fb604068f60 con 0x7fb60409ea60 2024-08-19T09:52:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb60402e720 con 0x7fb60409ea60 2024-08-19T09:52:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1941594543) 0x7fb6000085c0 con 0x7fb60409ea60 2024-08-19T09:52:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6792/0 conn(0x7fb60409ea60 legacy=0x7fb6041dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb5fa7fc640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6792/0 conn(0x7fb60409ea60 legacy=0x7fb6041dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.397 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:43.397 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 169628, 2024-08-19T09:52:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:43.399 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:43.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:43.399 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:43.399 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:43.399 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:43.400 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:43.400 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:43.400 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:43.400 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.400 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.401 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.402 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.403 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.404 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.405 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.405 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:43.405 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:43.406 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:43.406 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:43.406 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:43.406 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:43.407 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.407 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.407 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.408 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:43.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.410 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.412 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:43.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.416 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.416 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:43.418 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:43.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:43.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.424 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.425 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.426 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:43.427 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 10 librados: watch_flush enter 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 10 librados: watch_flush exit 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 20 client.26527.objecter shutdown clearing up homeless session... 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 10 client.26527.objecter successfully canceled tick 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb5e8041010 legacy=0x7fb5e80434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 10 monclient: shutdown 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 >> v1:172.21.15.1:6791/0 conn(0x7fb60413f5c0 legacy=0x7fb6041d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60a3fd640 1 -- 172.21.15.1:0/3569695282 reap_dead start 2024-08-19T09:52:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60a3fd640 1 -- 172.21.15.1:0/3569695282 reap_dead start 2024-08-19T09:52:43.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 shutdown_connections 2024-08-19T09:52:43.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 >> 172.21.15.1:0/3569695282 conn(0x7fb6040ac4b0 msgr2=0x7fb6041421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 shutdown_connections 2024-08-19T09:52:43.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.372+0000 7fb60be60640 1 -- 172.21.15.1:0/3569695282 wait complete. 2024-08-19T09:52:43.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.373+0000 7fb60be60640 1 librados: shutdown 2024-08-19T09:52:43.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:43.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:52:43.430 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.529+0000 7fdfac856640 10 monclient: get_monmap_and_config 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.529+0000 7fdfac856640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.529+0000 7fdfac856640 10 monclient: monmap: 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:43.530175+0000 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:43.530175+0000 2024-08-19T09:52:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.531 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.529+0000 7fdfac856640 1 Processor -- start 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.529+0000 7fdfac856640 1 -- start start 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.529+0000 7fdfac856640 10 monclient: init 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient: _add_conns ranks=[7,6,2,1,0,5,8,3,4] 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient(hunting): picked mon.noname-i con 0x7fdfa80a6830 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient(hunting): picked mon.noname-d con 0x7fdfa80694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa80a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa80a3c70 con 0x7fdfa80a46a0 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa80a33d0 con 0x7fdfa80694b0 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa80a36b0 con 0x7fdfa80a6830 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfac856640 10 monclient(hunting): authenticate will time out at 819.606636s 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfa6d9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fdfa80a46a0 0x7fdfa8141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40630/0 (socket says 172.21.15.1:40630) 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfa6d9d640 1 -- 172.21.15.1:0/2285601163 learned_addr learned my addr 172.21.15.1:0/2285601163 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.530+0000 7fdfa5d9b640 1 --1- 172.21.15.1:0/2285601163 >> v1:172.21.15.1:6792/0 conn(0x7fdfa80694b0 0x7fdfa80a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35682/0 (socket says 172.21.15.1:35682) 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.532+0000 7fdfa5d9b640 1 -- 172.21.15.1:0/2285601163 >> v1:172.21.15.1:6792/0 conn(0x7fdfa80694b0 legacy=0x7fdfa80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.532+0000 7fdfa6d9d640 1 -- 172.21.15.1:0/2285601163 >> v1:172.21.15.1:6790/0 conn(0x7fdfa80a46a0 legacy=0x7fdfa8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3382963454 0 0) 0x7fdfa80a33d0 con 0x7fdfa80694b0 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa659c640 1 -- 172.21.15.1:0/2285601163 >> v1:172.21.15.82:6792/0 conn(0x7fdfa80a6830 legacy=0x7fdfa80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): my global_id is 26502 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf90003520 con 0x7fdfa80694b0 2024-08-19T09:52:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 356312425 0 0) 0x7fdfa80a3c70 con 0x7fdfa80a46a0 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient(hunting): my global_id is 26606 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa80a33d0 con 0x7fdfa80a46a0 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3213982366 0 0) 0x7fdf90003520 con 0x7fdfa80694b0 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 >> v1:172.21.15.82:6792/0 conn(0x7fdfa80a6830 legacy=0x7fdfa80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 >> v1:172.21.15.1:6790/0 conn(0x7fdfa80a46a0 legacy=0x7fdfa8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 monclient: found mon.noname-d 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa806dbb0 con 0x7fdfa80694b0 2024-08-19T09:52:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf9c001ae0 con 0x7fdfa80694b0 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 5 monclient: authenticate success, global_id 26502 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: dump: 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf9c0047e0 con 0x7fdfa80694b0 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 1 -- 172.21.15.1:0/2285601163 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf9c004d50 con 0x7fdfa80694b0 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfa559a640 10 monclient: dump: 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 4 set_mon_vals no callback set 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 10 monclient: shutdown 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.533+0000 7fdfac856640 1 -- 172.21.15.1:0/2285601163 >> v1:172.21.15.1:6792/0 conn(0x7fdfa80694b0 legacy=0x7fdfa80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 1 -- 172.21.15.1:0/2285601163 shutdown_connections 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 1 -- 172.21.15.1:0/2285601163 >> 172.21.15.1:0/2285601163 conn(0x7fdfa80a9020 msgr2=0x7fdfa8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 1 -- 172.21.15.1:0/2285601163 shutdown_connections 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 1 -- 172.21.15.1:0/2285601163 wait complete. 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 10 monclient: monmap: 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 1 librados: starting msgr at 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.534+0000 7fdfac856640 1 librados: starting objecter 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 Processor -- start 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 -- start start 2024-08-19T09:52:43.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 librados: setting wanted keys 2024-08-19T09:52:43.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 librados: calling monclient init 2024-08-19T09:52:43.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient: init 2024-08-19T09:52:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient: _add_conns ranks=[6,5,8,4,0,2,7,1,3] 2024-08-19T09:52:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient(hunting): picked mon.noname-d con 0x7fdfa80694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient(hunting): picked mon.noname-h con 0x7fdfa80a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient(hunting): picked mon.noname-e con 0x7fdfa80a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa8139580 con 0x7fdfa80a46a0 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa813a9d0 con 0x7fdfa80694b0 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa80a9f30 con 0x7fdfa80a6830 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfac856640 10 monclient(hunting): authenticate will time out at 819.611638s 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfa659c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdfa80694b0 0x7fdfa8141f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35698/0 (socket says 172.21.15.1:35698) 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.535+0000 7fdfa659c640 1 -- 172.21.15.1:0/1860363193 learned_addr learned my addr 172.21.15.1:0/1860363193 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdfa659c640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6792/0 conn(0x7fdfa80694b0 legacy=0x7fdfa8141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 client.?.objecter ms_handle_connect 0x7fdfa80694b0 2024-08-19T09:52:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 client.?.objecter ms_handle_connect 0x7fdfa80a6830 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdfa6d9d640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdfa80a6830 legacy=0x7fdfa81d6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830678672 0 0) 0x7fdfa813a9d0 con 0x7fdfa80694b0 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 26511 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf88004eb0 con 0x7fdfa80694b0 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1489703845 0 0) 0x7fdfa80a9f30 con 0x7fdfa80a6830 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.537+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 26972 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa813a9d0 con 0x7fdfa80a6830 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 client.?.objecter ms_handle_connect 0x7fdfa80a46a0 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfa5d9b640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.82:6791/0 conn(0x7fdfa80a46a0 legacy=0x7fdfa81d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 663204971 0 0) 0x7fdf88004eb0 con 0x7fdfa80694b0 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfa80a9f30 con 0x7fdfa80694b0 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf98003610 con 0x7fdfa80694b0 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): dump: 2024-08-19T09:52:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 866363542 0 0) 0x7fdfa813a9d0 con 0x7fdfa80a6830 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf88004eb0 con 0x7fdfa80a6830 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113422316 0 0) 0x7fdfa8139580 con 0x7fdfa80a46a0 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 26195 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa813a9d0 con 0x7fdfa80a46a0 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdfa0004b60 con 0x7fdfa80a6830 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient(hunting): dump: 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2911362447 0 0) 0x7fdfa80a9f30 con 0x7fdfa80694b0 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdfa80a6830 legacy=0x7fdfa81d6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.82:6791/0 conn(0x7fdfa80a46a0 legacy=0x7fdfa81d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 monclient: found mon.d 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa8139350 con 0x7fdfa80694b0 2024-08-19T09:52:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 5 monclient: authenticate success, global_id 26511 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 10 monclient: _renew_subs 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa8068a70 con 0x7fdfa80694b0 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 10 client.26511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 10 monclient: _renew_subs 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fdfa813a6f0 con 0x7fdfa80694b0 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfac856640 1 librados: init done 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf98003b90 con 0x7fdfa80694b0 2024-08-19T09:52:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf980040c0 con 0x7fdfa80694b0 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdfa4d99640 4 set_mon_vals no callback set 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.538+0000 7fdf96ffd640 10 monclient: dump: 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdfac856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf8c0065c0 con 0x7fdf8c002860 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fdf98004260 con 0x7fdfa80694b0 2024-08-19T09:52:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(97..97 src has 1..97) ==== 6672+0+0 (unknown 3727910313 0 0) 0x7fdf9804e7c0 con 0x7fdfa80694b0 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdf96ffd640 10 client.26511.objecter ms_dispatch 0x7fdfa8000b90 osd_map(97..97 src has 1..97) 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdf96ffd640 3 client.26511.objecter handle_osd_map got epochs [97,97] > 0 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdf96ffd640 3 client.26511.objecter handle_osd_map decoding full epoch 97 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdf96ffd640 20 client.26511.objecter dump_active .. 0 homeless 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.539+0000 7fdfa6d9d640 10 monclient: get_auth_request con 0x7fdf88041700 auth_method 0 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 10 client.26511.objecter ms_handle_connect 0x7fdf8c002860 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdfa5d9b640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdf8c002860 legacy=0x7fdf8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 10 client.26511.objecter resend_mon_ops 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290922623 0 0) 0x7fdf8c0065c0 con 0x7fdf8c002860 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 26981 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa80a9f30 con 0x7fdf8c002860 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 375560091 0 0) 0x7fdfa80a9f30 con 0x7fdf8c002860 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.541+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf8c0065c0 con 0x7fdf8c002860 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf9c001320 con 0x7fdf8c002860 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 10 monclient: dump: 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 455494893 0 0) 0x7fdf8c0065c0 con 0x7fdf8c002860 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf8c00a160 con 0x7fdf8c002860 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 10 client.26511.objecter ms_handle_connect 0x7fdf88041700 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdf9c004b20 con 0x7fdf8c002860 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdf8c002860 legacy=0x7fdf8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.542+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdf8c002860 legacy=0x7fdf8c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.548+0000 7fdfac856640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.548+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf8c009e70 con 0x7fdf8c009130 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.550+0000 7fdfa5d9b640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdf8c009130 legacy=0x7fdf8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.550+0000 7fdf96ffd640 10 client.26511.objecter ms_handle_connect 0x7fdf8c009130 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.550+0000 7fdf96ffd640 10 client.26511.objecter resend_mon_ops 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136366675 0 0) 0x7fdf8c009e70 con 0x7fdf8c009130 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient(hunting): my global_id is 26990 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf8c0065c0 con 0x7fdf8c009130 2024-08-19T09:52:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1717499451 0 0) 0x7fdf8c0065c0 con 0x7fdf8c009130 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf8c009e70 con 0x7fdf8c009130 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf9c005290 con 0x7fdf8c009130 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient: dump: 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1777324810 0 0) 0x7fdf8c009e70 con 0x7fdf8c009130 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf8c00c3f0 con 0x7fdf8c009130 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 4002887124) 0x7fdf9c0075f0 con 0x7fdf8c009130 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdf8c009130 legacy=0x7fdf8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdf96ffd640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6793/0 conn(0x7fdf8c009130 legacy=0x7fdf8c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.571 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 169823, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:43.572 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.573 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.574 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:43.575 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:43.576 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:43.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.579 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.583 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.584 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:43.588 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.590 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:43.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:43.592 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 10 librados: watch_flush enter 2024-08-19T09:52:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 10 librados: watch_flush exit 2024-08-19T09:52:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 20 client.26511.objecter shutdown clearing up homeless session... 2024-08-19T09:52:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 10 client.26511.objecter successfully canceled tick 2024-08-19T09:52:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.82:6812/2549104415 conn(0x7fdf88041700 legacy=0x7fdf88043bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 10 monclient: shutdown 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 >> v1:172.21.15.1:6792/0 conn(0x7fdfa80694b0 legacy=0x7fdfa8141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.551+0000 7fdfa6d9d640 1 -- 172.21.15.1:0/1860363193 reap_dead start 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.552+0000 7fdfa6d9d640 1 -- 172.21.15.1:0/1860363193 reap_dead start 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.552+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 shutdown_connections 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.552+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 >> 172.21.15.1:0/1860363193 conn(0x7fdfa80a9020 msgr2=0x7fdfa8133600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.552+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 shutdown_connections 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.552+0000 7fdfac856640 1 -- 172.21.15.1:0/1860363193 wait complete. 2024-08-19T09:52:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.552+0000 7fdfac856640 1 librados: shutdown 2024-08-19T09:52:43.595 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:43.596 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:52:43.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.694+0000 7f7244bc3640 10 monclient: get_monmap_and_config 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.694+0000 7f7244bc3640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.694+0000 7f7244bc3640 10 monclient: monmap: 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:43.695576+0000 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:43.695576+0000 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 1 Processor -- start 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 1 -- start start 2024-08-19T09:52:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient: init 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient: _add_conns ranks=[8,1,7,5,0,6,3,2,4] 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient(hunting): picked mon.noname-e con 0x7f72400a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient(hunting): picked mon.noname-f con 0x7f72400a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient(hunting): picked mon.noname-i con 0x7f724013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72400a3b40 con 0x7f72400a6ba0 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72400a8130 con 0x7f724013f5c0 2024-08-19T09:52:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f724006ede0 con 0x7f72400a67c0 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f7244bc3640 10 monclient(hunting): authenticate will time out at 819.771786s 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f723e59c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f72400a67c0 0x7f72400a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43686/0 (socket says 172.21.15.1:43686) 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.695+0000 7f723e59c640 1 -- 172.21.15.1:0/4245568186 learned_addr learned my addr 172.21.15.1:0/4245568186 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723e59c640 1 -- 172.21.15.1:0/4245568186 >> v1:172.21.15.1:6793/0 conn(0x7f72400a67c0 legacy=0x7f72400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723ed9d640 1 -- 172.21.15.1:0/4245568186 >> v1:172.21.15.82:6792/0 conn(0x7f724013f5c0 legacy=0x7f72401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f7235d9b640 1 -- 172.21.15.1:0/4245568186 >> v1:172.21.15.82:6789/0 conn(0x7f72400a6ba0 legacy=0x7f72400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2043874947 0 0) 0x7f724006ede0 con 0x7f72400a67c0 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): my global_id is 26999 2024-08-19T09:52:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f722c003520 con 0x7f72400a67c0 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510612097 0 0) 0x7f72400a8130 con 0x7f724013f5c0 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): my global_id is 134314 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f724006ede0 con 0x7f724013f5c0 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 839248040 0 0) 0x7f72400a3b40 con 0x7f72400a6ba0 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient(hunting): my global_id is 27982 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72400a8130 con 0x7f72400a6ba0 2024-08-19T09:52:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 491572568 0 0) 0x7f722c003520 con 0x7f72400a67c0 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 >> v1:172.21.15.82:6792/0 conn(0x7f724013f5c0 legacy=0x7f72401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 >> v1:172.21.15.82:6789/0 conn(0x7f72400a6ba0 legacy=0x7f72400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 monclient: found mon.noname-e 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72400a92a0 con 0x7f72400a67c0 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f7244bc3640 5 monclient: authenticate success, global_id 26999 2024-08-19T09:52:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7230001ad0 con 0x7f72400a67c0 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f7244bc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: dump: 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f7244bc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72300048a0 con 0x7f72400a67c0 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 1 -- 172.21.15.1:0/4245568186 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7230004e10 con 0x7f72400a67c0 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f7244bc3640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f7244bc3640 4 set_mon_vals no callback set 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.698+0000 7f723dd9b640 10 monclient: dump: 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 10 monclient: shutdown 2024-08-19T09:52:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 1 -- 172.21.15.1:0/4245568186 >> v1:172.21.15.1:6793/0 conn(0x7f72400a67c0 legacy=0x7f72400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 1 -- 172.21.15.1:0/4245568186 shutdown_connections 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 1 -- 172.21.15.1:0/4245568186 >> 172.21.15.1:0/4245568186 conn(0x7f72400ac4b0 msgr2=0x7f72400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 1 -- 172.21.15.1:0/4245568186 shutdown_connections 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 1 -- 172.21.15.1:0/4245568186 wait complete. 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.699+0000 7f7244bc3640 10 monclient: monmap: 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.711 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 librados: starting msgr at 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 librados: starting objecter 2024-08-19T09:52:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 Processor -- start 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 -- start start 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 librados: setting wanted keys 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 librados: calling monclient init 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient: init 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient: _add_conns ranks=[5,0,2,1,3,7,8,6,4] 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient(hunting): picked mon.noname-h con 0x7f72400a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient(hunting): picked mon.noname-a con 0x7f72400a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient(hunting): picked mon.noname-b con 0x7f724013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f724013ad70 con 0x7f72400a6ba0 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7240139880 con 0x7f724013f5c0 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f724013bbe0 con 0x7f72400a67c0 2024-08-19T09:52:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.700+0000 7f7244bc3640 10 monclient(hunting): authenticate will time out at 819.776933s 2024-08-19T09:52:43.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.701+0000 7f7235d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f72400a6ba0 0x7f724013c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59008/0 (socket says 172.21.15.1:59008) 2024-08-19T09:52:43.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.701+0000 7f7235d9b640 1 -- 172.21.15.1:0/218856287 learned_addr learned my addr 172.21.15.1:0/218856287 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7235d9b640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.1:6789/0 conn(0x7f72400a6ba0 legacy=0x7f724013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 client.?.objecter ms_handle_connect 0x7f72400a6ba0 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f723ed9d640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.1:6790/0 conn(0x7f724013f5c0 legacy=0x7f72401d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 client.?.objecter ms_handle_connect 0x7f724013f5c0 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f723e59c640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6791/0 conn(0x7f72400a67c0 legacy=0x7f724013c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 client.?.objecter ms_handle_connect 0x7f72400a67c0 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408802494 0 0) 0x7f724013ad70 con 0x7f72400a6ba0 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): my global_id is 134235 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7220004ee0 con 0x7f72400a6ba0 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241603749 0 0) 0x7f7240139880 con 0x7f724013f5c0 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): my global_id is 26615 2024-08-19T09:52:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f724013ad70 con 0x7f724013f5c0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822051142 0 0) 0x7f724013bbe0 con 0x7f72400a67c0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): my global_id is 26204 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7240139880 con 0x7f72400a67c0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2131264037 0 0) 0x7f7220004ee0 con 0x7f72400a6ba0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f724013bbe0 con 0x7f72400a6ba0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7228003950 con 0x7f72400a6ba0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): dump: 2024-08-19T09:52:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 637459131 0 0) 0x7f724013ad70 con 0x7f724013f5c0 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7220004ee0 con 0x7f724013f5c0 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7238002070 con 0x7f724013f5c0 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): dump: 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2193487319 0 0) 0x7f7240139880 con 0x7f72400a67c0 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f724013ad70 con 0x7f72400a67c0 2024-08-19T09:52:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7230004d90 con 0x7f72400a67c0 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient(hunting): dump: 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 599551315 0 0) 0x7f724013bbe0 con 0x7f72400a6ba0 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6791/0 conn(0x7f72400a67c0 legacy=0x7f724013c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.1:6790/0 conn(0x7f724013f5c0 legacy=0x7f72401d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 monclient: found mon.a 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724013b340 con 0x7f72400a6ba0 2024-08-19T09:52:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f72377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 5 monclient: authenticate success, global_id 134235 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 10 monclient: _renew_subs 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f724013a680 con 0x7f72400a6ba0 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 10 client.134235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 10 monclient: _renew_subs 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.703+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f724013cf00 con 0x7f72400a6ba0 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f7244bc3640 1 librados: init done 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7228001ce0 con 0x7f72400a6ba0 2024-08-19T09:52:43.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7228004160 con 0x7f72400a6ba0 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f723d59a640 4 set_mon_vals no callback set 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 10 monclient: dump: 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.730 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f7244bc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f72240065c0 con 0x7f7224002860 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.704+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7228011a80 con 0x7f72400a6ba0 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.705+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(97..97 src has 1..97) ==== 6672+0+0 (unknown 3727910313 0 0) 0x7f722804e880 con 0x7f72400a6ba0 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.705+0000 7f72377fe640 10 client.134235.objecter ms_dispatch 0x7f7240000b90 osd_map(97..97 src has 1..97) 2024-08-19T09:52:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.705+0000 7f72377fe640 3 client.134235.objecter handle_osd_map got epochs [97,97] > 0 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.705+0000 7f72377fe640 3 client.134235.objecter handle_osd_map decoding full epoch 97 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.705+0000 7f72377fe640 20 client.134235.objecter dump_active .. 0 homeless 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.705+0000 7f723ed9d640 10 monclient: get_auth_request con 0x7f72200418e0 auth_method 0 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f723e59c640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6789/0 conn(0x7f7224002860 legacy=0x7f7224004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 client.134235.objecter ms_handle_connect 0x7f7224002860 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 client.134235.objecter resend_mon_ops 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475469258 0 0) 0x7f72240065c0 con 0x7f7224002860 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient(hunting): my global_id is 27991 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f724013bbe0 con 0x7f7224002860 2024-08-19T09:52:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2621586447 0 0) 0x7f724013bbe0 con 0x7f7224002860 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72240065c0 con 0x7f7224002860 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f72300048a0 con 0x7f7224002860 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 monclient: dump: 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 10 client.134235.objecter ms_handle_connect 0x7f72200418e0 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 617588206 0 0) 0x7f72240065c0 con 0x7f7224002860 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.707+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f722400a160 con 0x7f7224002860 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.708+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7230005490 con 0x7f7224002860 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.708+0000 7f72377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.708+0000 7f72377fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.708+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6789/0 conn(0x7f7224002860 legacy=0x7f7224004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.708+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6789/0 conn(0x7f7224002860 legacy=0x7f7224004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.714+0000 7f7244bc3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.714+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7224009e70 con 0x7f7224009130 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f723e59c640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6789/0 conn(0x7f7224009130 legacy=0x7f72240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 10 client.134235.objecter ms_handle_connect 0x7f7224009130 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 10 client.134235.objecter resend_mon_ops 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130490145 0 0) 0x7f7224009e70 con 0x7f7224009130 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 10 monclient(hunting): my global_id is 28000 2024-08-19T09:52:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.716+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72240065c0 con 0x7f7224009130 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2146297163 0 0) 0x7f72240065c0 con 0x7f7224009130 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7224009e70 con 0x7f7224009130 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7230005d10 con 0x7f7224009130 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 10 monclient: dump: 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3789392790 0 0) 0x7f7224009e70 con 0x7f7224009130 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f722400c410 con 0x7f7224009130 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 773726000) 0x7f7230008010 con 0x7f7224009130 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6789/0 conn(0x7f7224009130 legacy=0x7f72240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f72377fe640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6789/0 conn(0x7f7224009130 legacy=0x7f72240095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 169977, 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:43.759 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 10 librados: watch_flush enter 2024-08-19T09:52:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 10 librados: watch_flush exit 2024-08-19T09:52:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 20 client.134235.objecter shutdown clearing up homeless session... 2024-08-19T09:52:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 10 client.134235.objecter successfully canceled tick 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.82:6812/2549104415 conn(0x7f72200418e0 legacy=0x7f7220043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 10 monclient: shutdown 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 >> v1:172.21.15.1:6789/0 conn(0x7f72400a6ba0 legacy=0x7f724013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f723ed9d640 1 -- 172.21.15.1:0/218856287 reap_dead start 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f723ed9d640 1 -- 172.21.15.1:0/218856287 reap_dead start 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 shutdown_connections 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.717+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 >> 172.21.15.1:0/218856287 conn(0x7f72400ac4b0 msgr2=0x7f724013ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.718+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 shutdown_connections 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.718+0000 7f7244bc3640 1 -- 172.21.15.1:0/218856287 wait complete. 2024-08-19T09:52:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.718+0000 7f7244bc3640 1 librados: shutdown 2024-08-19T09:52:43.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:43.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:52:43.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:52:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.861+0000 7f489869f640 10 monclient: get_monmap_and_config 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.861+0000 7f489869f640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.862+0000 7f489869f640 10 monclient: monmap: 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:43.862935+0000 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:43.862935+0000 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.864 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.862+0000 7f489869f640 1 Processor -- start 2024-08-19T09:52:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.862+0000 7f489869f640 1 -- start start 2024-08-19T09:52:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.862+0000 7f489869f640 10 monclient: init 2024-08-19T09:52:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient: _add_conns ranks=[2,6,5,8,1,7,4,3,0] 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient(hunting): picked mon.noname-b con 0x7f48900a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient(hunting): picked mon.noname-d con 0x7f48900694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient(hunting): picked mon.noname-h con 0x7f48900a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48900a3c70 con 0x7f48900a6830 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48900a33d0 con 0x7f48900a46a0 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48900a36b0 con 0x7f48900694b0 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489869f640 10 monclient(hunting): authenticate will time out at 819.939345s 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489643b640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f48900a6830 0x7f48900a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40652/0 (socket says 172.21.15.1:40652) 2024-08-19T09:52:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.863+0000 7f489643b640 1 -- 172.21.15.1:0/272724433 learned_addr learned my addr 172.21.15.1:0/272724433 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f489643b640 1 -- 172.21.15.1:0/272724433 >> v1:172.21.15.1:6790/0 conn(0x7f48900a6830 legacy=0x7f48900a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895c3a640 1 -- 172.21.15.1:0/272724433 >> v1:172.21.15.1:6792/0 conn(0x7f48900694b0 legacy=0x7f48900a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4896c3c640 1 -- 172.21.15.1:0/272724433 >> v1:172.21.15.82:6791/0 conn(0x7f48900a46a0 legacy=0x7f4890141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2179264434 0 0) 0x7f48900a36b0 con 0x7f48900694b0 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): my global_id is 26520 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4878003520 con 0x7f48900694b0 2024-08-19T09:52:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097966243 0 0) 0x7f48900a3c70 con 0x7f48900a6830 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): my global_id is 26624 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48900a36b0 con 0x7f48900a6830 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 544533892 0 0) 0x7f48900a33d0 con 0x7f48900a46a0 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 10 monclient(hunting): my global_id is 26213 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48900a3c70 con 0x7f48900a46a0 2024-08-19T09:52:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.865+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2991557823 0 0) 0x7f4878003520 con 0x7f48900694b0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 >> v1:172.21.15.82:6791/0 conn(0x7f48900a46a0 legacy=0x7f4890141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 >> v1:172.21.15.1:6790/0 conn(0x7f48900a6830 legacy=0x7f48900a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 monclient: found mon.noname-d 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489006dbb0 con 0x7f48900694b0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4884001a60 con 0x7f48900694b0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: dump: 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 5 monclient: authenticate success, global_id 26520 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4884001d30 con 0x7f48900694b0 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 1 -- 172.21.15.1:0/272724433 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4884003950 con 0x7f48900694b0 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f4895439640 10 monclient: dump: 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 4 set_mon_vals no callback set 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 10 monclient: shutdown 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 1 -- 172.21.15.1:0/272724433 >> v1:172.21.15.1:6792/0 conn(0x7f48900694b0 legacy=0x7f48900a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 1 -- 172.21.15.1:0/272724433 shutdown_connections 2024-08-19T09:52:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 1 -- 172.21.15.1:0/272724433 >> 172.21.15.1:0/272724433 conn(0x7f48900a9020 msgr2=0x7f4890068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 1 -- 172.21.15.1:0/272724433 shutdown_connections 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.866+0000 7f489869f640 1 -- 172.21.15.1:0/272724433 wait complete. 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 10 monclient: build_initial_monmap 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 10 monclient: monmap: 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 1 librados: starting msgr at 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 1 librados: starting objecter 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 1 Processor -- start 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 1 -- start start 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 1 librados: setting wanted keys 2024-08-19T09:52:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 1 librados: calling monclient init 2024-08-19T09:52:43.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.867+0000 7f489869f640 10 monclient: init 2024-08-19T09:52:43.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:43.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient: _add_conns ranks=[1,3,7,4,2,6,0,8,5] 2024-08-19T09:52:43.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient(hunting): picked mon.noname-f con 0x7f48900694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient(hunting): picked mon.noname-g con 0x7f48900a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient(hunting): picked mon.noname-i con 0x7f48900a6830 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4890137c10 con 0x7f48900694b0 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48901381e0 con 0x7f48900a46a0 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4890136720 con 0x7f48900a6830 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:43.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489869f640 10 monclient(hunting): authenticate will time out at 819.944265s 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489643b640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f48900694b0 0x7f4890134c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:54804/0 (socket says 172.21.15.1:54804) 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f489643b640 1 -- 172.21.15.1:0/2672658436 learned_addr learned my addr 172.21.15.1:0/2672658436 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.868+0000 7f4895c3a640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f48900a46a0 0x7f4890135340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:39444/0 (socket says 172.21.15.1:39444) 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f489643b640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6789/0 conn(0x7f48900694b0 legacy=0x7f4890134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f4895c3a640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f48900a46a0 legacy=0x7f4890135340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 client.?.objecter ms_handle_connect 0x7f48900694b0 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 client.?.objecter ms_handle_connect 0x7f48900a46a0 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f4896c3c640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6792/0 conn(0x7f48900a6830 legacy=0x7f48901d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 client.?.objecter ms_handle_connect 0x7f48900a6830 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:43.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2275234918 0 0) 0x7f4890137c10 con 0x7f48900694b0 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): my global_id is 28009 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f487c004eb0 con 0x7f48900694b0 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313720265 0 0) 0x7f489006dbb0 con 0x7f48900a46a0 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): my global_id is 134220 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4890137c10 con 0x7f48900a46a0 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971632072 0 0) 0x7f4890136720 con 0x7f48900a6830 2024-08-19T09:52:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 10 monclient(hunting): my global_id is 134323 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.870+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f489006dbb0 con 0x7f48900a6830 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3164331670 0 0) 0x7f487c004eb0 con 0x7f48900694b0 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4890136720 con 0x7f48900694b0 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4880004b60 con 0x7f48900694b0 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): dump: 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.890 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.891 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 580798663 0 0) 0x7f4890137c10 con 0x7f48900a46a0 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f487c004eb0 con 0x7f48900a46a0 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4884003950 con 0x7f48900a46a0 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): dump: 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.892 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4095256069 0 0) 0x7f489006dbb0 con 0x7f48900a6830 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4890137c10 con 0x7f48900a6830 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4888003610 con 0x7f48900a6830 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient(hunting): dump: 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 850775255 0 0) 0x7f4890136720 con 0x7f48900694b0 2024-08-19T09:52:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6792/0 conn(0x7f48900a6830 legacy=0x7f48901d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f48900a46a0 legacy=0x7f4890135340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 monclient: found mon.f 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4890135a50 con 0x7f48900694b0 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: _finish_auth 0 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 5 monclient: authenticate success, global_id 28009 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 10 monclient: _renew_subs 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4890138a80 con 0x7f48900694b0 2024-08-19T09:52:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 10 client.28009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 10 monclient: _renew_subs 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4890142540 con 0x7f48900694b0 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f489869f640 1 librados: init done 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4880005030 con 0x7f48900694b0 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4880005520 con 0x7f48900694b0 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f4894c38640 4 set_mon_vals no callback set 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.871+0000 7f488effd640 10 monclient: dump: 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.898 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f489869f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48700065c0 con 0x7f4870002860 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f48800057c0 con 0x7f48900694b0 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(97..97 src has 1..97) ==== 6672+0+0 (unknown 3727910313 0 0) 0x7f488004ead0 con 0x7f48900694b0 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f488effd640 10 client.28009.objecter ms_dispatch 0x7f4890000b90 osd_map(97..97 src has 1..97) 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f488effd640 3 client.28009.objecter handle_osd_map got epochs [97,97] > 0 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.872+0000 7f488effd640 3 client.28009.objecter handle_osd_map decoding full epoch 97 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.873+0000 7f4896c3c640 10 monclient: get_auth_request con 0x7f487c041870 auth_method 0 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.873+0000 7f488effd640 20 client.28009.objecter dump_active .. 0 homeless 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f4895c3a640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f4870002860 legacy=0x7f4870004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 10 client.28009.objecter ms_handle_connect 0x7f4870002860 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 10 client.28009.objecter resend_mon_ops 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2358801741 0 0) 0x7f48700065c0 con 0x7f4870002860 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 10 monclient(hunting): my global_id is 134229 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.874+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4890136720 con 0x7f4870002860 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1060318680 0 0) 0x7f4890136720 con 0x7f4870002860 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48700065c0 con 0x7f4870002860 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4884003e30 con 0x7f4870002860 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:52:43.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 10 monclient: dump: 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 155076315 0 0) 0x7f48700065c0 con 0x7f4870002860 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f487000a160 con 0x7f4870002860 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 10 client.28009.objecter ms_handle_connect 0x7f487c041870 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4884004090 con 0x7f4870002860 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f4870002860 legacy=0x7f4870004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.875+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f4870002860 legacy=0x7f4870004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.881+0000 7f489869f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.881+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4870009e70 con 0x7f4870009130 2024-08-19T09:52:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f4895c3a640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f4870009130 legacy=0x7f48700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 client.28009.objecter ms_handle_connect 0x7f4870009130 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 client.28009.objecter resend_mon_ops 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726379356 0 0) 0x7f4870009e70 con 0x7f4870009130 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient(hunting): my global_id is 134238 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48700065c0 con 0x7f4870009130 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2063988289 0 0) 0x7f48700065c0 con 0x7f4870009130 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4870009e70 con 0x7f4870009130 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4884005b00 con 0x7f4870009130 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.884+0000 7f488effd640 10 monclient: dump: 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4258994210 0 0) 0x7f4870009e70 con 0x7f4870009130 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f487000c410 con 0x7f4870009130 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3471897605) 0x7f4884005e10 con 0x7f4870009130 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f4870009130 legacy=0x7f48700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f488effd640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6790/0 conn(0x7f4870009130 legacy=0x7f48700095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 20946, 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:43.908 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:43.909 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.910 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.911 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.911 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.912 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:43.913 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:43.914 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:43.915 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.917 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:43.918 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:43.919 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.920 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.921 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.929 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:43.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:43.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:43.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:43.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:43.932 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:43.932 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 10 librados: watch_flush enter 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 10 librados: watch_flush exit 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 20 client.28009.objecter shutdown clearing up homeless session... 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 10 client.28009.objecter successfully canceled tick 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6812/2549104415 conn(0x7f487c041870 legacy=0x7f487c043d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 10 monclient: shutdown 2024-08-19T09:52:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 >> v1:172.21.15.82:6789/0 conn(0x7f48900694b0 legacy=0x7f4890134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f4896c3c640 1 -- 172.21.15.1:0/2672658436 reap_dead start 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f4896c3c640 1 -- 172.21.15.1:0/2672658436 reap_dead start 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.885+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 shutdown_connections 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.886+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 >> 172.21.15.1:0/2672658436 conn(0x7f48900a9020 msgr2=0x7f48900a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.886+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 shutdown_connections 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.886+0000 7f489869f640 1 -- 172.21.15.1:0/2672658436 wait complete. 2024-08-19T09:52:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:43.886+0000 7f489869f640 1 librados: shutdown 2024-08-19T09:52:43.935 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:43.935 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:52:43.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:52:44.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.044+0000 7ff069ce5640 10 monclient: get_monmap_and_config 2024-08-19T09:52:44.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.044+0000 7ff069ce5640 10 monclient: build_initial_monmap 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.044+0000 7ff069ce5640 10 monclient: monmap: 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:44.045566+0000 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:44.045566+0000 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:44.047 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:44.047 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:44.047 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:44.047 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:44.047 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 1 Processor -- start 2024-08-19T09:52:44.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 1 -- start start 2024-08-19T09:52:44.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient: init 2024-08-19T09:52:44.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:44.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient: _add_conns ranks=[1,5,8,6,2,4,3,0,7] 2024-08-19T09:52:44.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient(hunting): picked mon.noname-f con 0x7ff0640a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient(hunting): picked mon.noname-h con 0x7ff0640a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient(hunting): picked mon.noname-e con 0x7ff06413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0640a3b40 con 0x7ff0640a67c0 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0640a8130 con 0x7ff0640a6ba0 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06406ede0 con 0x7ff06413f5c0 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.045+0000 7ff069ce5640 10 monclient(hunting): authenticate will time out at 820.121974s 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.046+0000 7ff063fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff06413f5c0 0x7ff0641419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43702/0 (socket says 172.21.15.1:43702) 2024-08-19T09:52:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.046+0000 7ff063fff640 1 -- 172.21.15.1:0/3589072695 learned_addr learned my addr 172.21.15.1:0/3589072695 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff063fff640 1 -- 172.21.15.1:0/3589072695 >> v1:172.21.15.1:6793/0 conn(0x7ff06413f5c0 legacy=0x7ff0641419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042218813 0 0) 0x7ff06406ede0 con 0x7ff06413f5c0 2024-08-19T09:52:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff062ffd640 1 -- 172.21.15.1:0/3589072695 >> v1:172.21.15.82:6791/0 conn(0x7ff0640a6ba0 legacy=0x7ff0640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): my global_id is 27026 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff04c003520 con 0x7ff06413f5c0 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0637fe640 1 -- 172.21.15.1:0/3589072695 >> v1:172.21.15.82:6789/0 conn(0x7ff0640a67c0 legacy=0x7ff0640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847281036 0 0) 0x7ff0640a8130 con 0x7ff0640a6ba0 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient(hunting): my global_id is 26222 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff06406ede0 con 0x7ff0640a6ba0 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1891906763 0 0) 0x7ff04c003520 con 0x7ff06413f5c0 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 >> v1:172.21.15.82:6791/0 conn(0x7ff0640a6ba0 legacy=0x7ff0640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 >> v1:172.21.15.82:6789/0 conn(0x7ff0640a67c0 legacy=0x7ff0640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: _finish_hunting 0 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 monclient: found mon.noname-e 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0640a92a0 con 0x7ff06413f5c0 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: _finish_auth 0 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: _check_auth_tickets 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff05c001e50 con 0x7ff06413f5c0 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff069ce5640 5 monclient: authenticate success, global_id 27026 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff069ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: dump: 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff069ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff05c004c20 con 0x7ff06413f5c0 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 1 -- 172.21.15.1:0/3589072695 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff05c0051d0 con 0x7ff06413f5c0 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff069ce5640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff069ce5640 4 set_mon_vals no callback set 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.048+0000 7ff0627fc640 10 monclient: dump: 2024-08-19T09:52:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 10 monclient: shutdown 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 1 -- 172.21.15.1:0/3589072695 >> v1:172.21.15.1:6793/0 conn(0x7ff06413f5c0 legacy=0x7ff0641419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 1 -- 172.21.15.1:0/3589072695 shutdown_connections 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 1 -- 172.21.15.1:0/3589072695 >> 172.21.15.1:0/3589072695 conn(0x7ff0640ac4b0 msgr2=0x7ff0640a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 1 -- 172.21.15.1:0/3589072695 shutdown_connections 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 1 -- 172.21.15.1:0/3589072695 wait complete. 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 10 monclient: build_initial_monmap 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.049+0000 7ff069ce5640 10 monclient: monmap: 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 librados: starting msgr at 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 librados: starting objecter 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 Processor -- start 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 -- start start 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 librados: setting wanted keys 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 librados: calling monclient init 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient: init 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient: _add_conns ranks=[7,8,6,5,1,0,4,3,2] 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient(hunting): picked mon.noname-i con 0x7ff0640a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient(hunting): picked mon.noname-e con 0x7ff0640a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient(hunting): picked mon.noname-d con 0x7ff06413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06413a4d0 con 0x7ff06413f5c0 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06413a7b0 con 0x7ff0640a67c0 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06413a1f0 con 0x7ff0640a6ba0 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff069ce5640 10 monclient(hunting): authenticate will time out at 820.126911s 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff062ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff0640a6ba0 0x7ff0640a04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43718/0 (socket says 172.21.15.1:43718) 2024-08-19T09:52:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.050+0000 7ff062ffd640 1 -- 172.21.15.1:0/3125727019 learned_addr learned my addr 172.21.15.1:0/3125727019 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:44.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff063fff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.1:6792/0 conn(0x7ff06413f5c0 legacy=0x7ff0641d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff062ffd640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.1:6793/0 conn(0x7ff0640a6ba0 legacy=0x7ff0640a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 client.?.objecter ms_handle_connect 0x7ff06413f5c0 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 client.?.objecter ms_handle_connect 0x7ff0640a6ba0 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff0637fe640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6792/0 conn(0x7ff0640a67c0 legacy=0x7ff06409fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 client.?.objecter ms_handle_connect 0x7ff0640a67c0 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383686781 0 0) 0x7ff06413a4d0 con 0x7ff06413f5c0 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): my global_id is 26538 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff050004eb0 con 0x7ff06413f5c0 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2321593295 0 0) 0x7ff06413a1f0 con 0x7ff0640a6ba0 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): my global_id is 27035 2024-08-19T09:52:44.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff06413a4d0 con 0x7ff0640a6ba0 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2959130399 0 0) 0x7ff06413a7b0 con 0x7ff0640a67c0 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): my global_id is 134332 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff06413a1f0 con 0x7ff0640a67c0 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 772472989 0 0) 0x7ff06413a4d0 con 0x7ff0640a6ba0 2024-08-19T09:52:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff06413a7b0 con 0x7ff0640a6ba0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1649105995 0 0) 0x7ff050004eb0 con 0x7ff06413f5c0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff06413a4d0 con 0x7ff06413f5c0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3179357650 0 0) 0x7ff06413a1f0 con 0x7ff0640a67c0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff050004eb0 con 0x7ff0640a67c0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff058003610 con 0x7ff0640a6ba0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): dump: 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff05c0020a0 con 0x7ff06413f5c0 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): dump: 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff054003610 con 0x7ff0640a67c0 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient(hunting): dump: 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3283549874 0 0) 0x7ff06413a4d0 con 0x7ff06413f5c0 2024-08-19T09:52:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.1:6793/0 conn(0x7ff0640a6ba0 legacy=0x7ff0640a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6792/0 conn(0x7ff0640a67c0 legacy=0x7ff06409fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 monclient: found mon.d 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0640a0be0 con 0x7ff06413f5c0 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient: _finish_auth 0 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 5 monclient: authenticate success, global_id 26538 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 10 monclient: _renew_subs 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0640a0e10 con 0x7ff06413f5c0 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 10 client.26538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 10 monclient: _renew_subs 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0641d7b50 con 0x7ff06413f5c0 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff069ce5640 1 librados: init done 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.053+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff05c004c20 con 0x7ff06413f5c0 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff04bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff05c005700 con 0x7ff06413f5c0 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff04bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff061ffb640 4 set_mon_vals no callback set 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff04bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:52:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff04bfff640 10 monclient: dump: 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff069ce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff0440065c0 con 0x7ff044002860 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.054+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff05c011b60 con 0x7ff06413f5c0 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.055+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(98..98 src has 1..98) ==== 7316+0+0 (unknown 1470481556 0 0) 0x7ff05c04ea60 con 0x7ff06413f5c0 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.055+0000 7ff04bfff640 10 client.26538.objecter ms_dispatch 0x7ff064000b90 osd_map(98..98 src has 1..98) 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.055+0000 7ff04bfff640 3 client.26538.objecter handle_osd_map got epochs [98,98] > 0 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.055+0000 7ff04bfff640 3 client.26538.objecter handle_osd_map decoding full epoch 98 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.055+0000 7ff04bfff640 20 client.26538.objecter dump_active .. 0 homeless 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.055+0000 7ff062ffd640 10 monclient: get_auth_request con 0x7ff0500418e0 auth_method 0 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff0637fe640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6791/0 conn(0x7ff044002860 legacy=0x7ff044004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 client.26538.objecter ms_handle_connect 0x7ff044002860 2024-08-19T09:52:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 client.26538.objecter resend_mon_ops 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086850300 0 0) 0x7ff0440065c0 con 0x7ff044002860 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient(hunting): my global_id is 26231 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff06413a4d0 con 0x7ff044002860 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2157326285 0 0) 0x7ff06413a4d0 con 0x7ff044002860 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0440065c0 con 0x7ff044002860 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff054004180 con 0x7ff044002860 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient: dump: 2024-08-19T09:52:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2280722612 0 0) 0x7ff0440065c0 con 0x7ff044002860 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff04400a160 con 0x7ff044002860 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 client.26538.objecter ms_handle_connect 0x7ff0500418e0 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff0540043e0 con 0x7ff044002860 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6791/0 conn(0x7ff044002860 legacy=0x7ff044004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.057+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6791/0 conn(0x7ff044002860 legacy=0x7ff044004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.064+0000 7ff069ce5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.064+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff044009e70 con 0x7ff044009130 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff0637fe640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6791/0 conn(0x7ff044009130 legacy=0x7ff0440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 client.26538.objecter ms_handle_connect 0x7ff044009130 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 client.26538.objecter resend_mon_ops 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220922609 0 0) 0x7ff044009e70 con 0x7ff044009130 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient(hunting): my global_id is 26240 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0440065c0 con 0x7ff044009130 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4294365330 0 0) 0x7ff0440065c0 con 0x7ff044009130 2024-08-19T09:52:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff044009e70 con 0x7ff044009130 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff054005d70 con 0x7ff044009130 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 10 monclient: dump: 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 813936588 0 0) 0x7ff044009e70 con 0x7ff044009130 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.066+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff04400c410 con 0x7ff044009130 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7851 (unknown 0 0 3438629326) 0x7ff054007280 con 0x7ff044009130 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff04bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff04bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6791/0 conn(0x7ff044009130 legacy=0x7ff0440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff04bfff640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6791/0 conn(0x7ff044009130 legacy=0x7ff0440095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:44.088 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:44.088 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:44.088 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:44.088 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:44.088 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 170332, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T09:52:44.089 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:44.090 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:44.091 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:44.093 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:44.094 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.095 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:44.099 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:44.111 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 10 librados: watch_flush enter 2024-08-19T09:52:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 10 librados: watch_flush exit 2024-08-19T09:52:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 20 client.26538.objecter shutdown clearing up homeless session... 2024-08-19T09:52:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 10 client.26538.objecter successfully canceled tick 2024-08-19T09:52:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff0500418e0 legacy=0x7ff050043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 10 monclient: shutdown 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 >> v1:172.21.15.1:6792/0 conn(0x7ff06413f5c0 legacy=0x7ff0641d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff063fff640 1 -- 172.21.15.1:0/3125727019 reap_dead start 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff063fff640 1 -- 172.21.15.1:0/3125727019 reap_dead start 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 shutdown_connections 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 >> 172.21.15.1:0/3125727019 conn(0x7ff0640ac4b0 msgr2=0x7ff0640ab1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 shutdown_connections 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 -- 172.21.15.1:0/3125727019 wait complete. 2024-08-19T09:52:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.067+0000 7ff069ce5640 1 librados: shutdown 2024-08-19T09:52:44.115 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:44.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:52:44.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:52:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.214+0000 7fb3c8460640 10 monclient: get_monmap_and_config 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.214+0000 7fb3c8460640 10 monclient: build_initial_monmap 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.214+0000 7fb3c8460640 10 monclient: monmap: 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:44.215276+0000 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:44.215276+0000 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.214+0000 7fb3c8460640 1 Processor -- start 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.214+0000 7fb3c8460640 1 -- start start 2024-08-19T09:52:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.214+0000 7fb3c8460640 10 monclient: init 2024-08-19T09:52:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient: _add_conns ranks=[0,1,6,5,2,8,3,4,7] 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient(hunting): picked mon.noname-a con 0x7fb3c00a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient(hunting): picked mon.noname-f con 0x7fb3c00694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient(hunting): picked mon.noname-d con 0x7fb3c00a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c00a3c70 con 0x7fb3c00a6830 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c00a33d0 con 0x7fb3c00694b0 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c00a36b0 con 0x7fb3c00a46a0 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c8460640 10 monclient(hunting): authenticate will time out at 820.291698s 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c69fd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fb3c00a46a0 0x7fb3c0141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35728/0 (socket says 172.21.15.1:35728) 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c69fd640 1 -- 172.21.15.1:0/3706159788 learned_addr learned my addr 172.21.15.1:0/3706159788 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.215+0000 7fb3c61fc640 1 --1- 172.21.15.1:0/3706159788 >> v1:172.21.15.1:6789/0 conn(0x7fb3c00a6830 0x7fb3c00a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59014/0 (socket says 172.21.15.1:59014) 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.217+0000 7fb3c69fd640 1 -- 172.21.15.1:0/3706159788 >> v1:172.21.15.1:6792/0 conn(0x7fb3c00a46a0 legacy=0x7fb3c0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.217+0000 7fb3c61fc640 1 -- 172.21.15.1:0/3706159788 >> v1:172.21.15.1:6789/0 conn(0x7fb3c00a6830 legacy=0x7fb3c00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c59fb640 1 -- 172.21.15.1:0/3706159788 >> v1:172.21.15.82:6789/0 conn(0x7fb3c00694b0 legacy=0x7fb3c00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943505645 0 0) 0x7fb3c00a36b0 con 0x7fb3c00a46a0 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): my global_id is 26547 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3ac003520 con 0x7fb3c00a46a0 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955699649 0 0) 0x7fb3c00a33d0 con 0x7fb3c00694b0 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): my global_id is 28036 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3c00a36b0 con 0x7fb3c00694b0 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113497254 0 0) 0x7fb3c00a3c70 con 0x7fb3c00a6830 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient(hunting): my global_id is 134244 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3c00a33d0 con 0x7fb3c00a6830 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3350344814 0 0) 0x7fb3c00a33d0 con 0x7fb3c00a6830 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 >> v1:172.21.15.1:6792/0 conn(0x7fb3c00a46a0 legacy=0x7fb3c0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 >> v1:172.21.15.82:6789/0 conn(0x7fb3c00694b0 legacy=0x7fb3c00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: _finish_hunting 0 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 monclient: found mon.noname-a 2024-08-19T09:52:44.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c006dbb0 con 0x7fb3c00a6830 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: _finish_auth 0 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: _check_auth_tickets 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3bc001a80 con 0x7fb3c00a6830 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c8460640 5 monclient: authenticate success, global_id 134244 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c8460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: dump: 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c8460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3bc001d10 con 0x7fb3c00a6830 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c8460640 10 monclient: get_monmap_and_config success 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 1 -- 172.21.15.1:0/3706159788 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3bc004b20 con 0x7fb3c00a6830 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c8460640 4 set_mon_vals no callback set 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.218+0000 7fb3c51fa640 10 monclient: dump: 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.228 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 10 monclient: shutdown 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 -- 172.21.15.1:0/3706159788 >> v1:172.21.15.1:6789/0 conn(0x7fb3c00a6830 legacy=0x7fb3c00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 -- 172.21.15.1:0/3706159788 shutdown_connections 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 -- 172.21.15.1:0/3706159788 >> 172.21.15.1:0/3706159788 conn(0x7fb3c00a9020 msgr2=0x7fb3c0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 -- 172.21.15.1:0/3706159788 shutdown_connections 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 -- 172.21.15.1:0/3706159788 wait complete. 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 10 monclient: build_initial_monmap 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 10 monclient: monmap: 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:52:44.229 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 librados: starting msgr at 2024-08-19T09:52:44.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.219+0000 7fb3c8460640 1 librados: starting objecter 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 Processor -- start 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 -- start start 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 librados: setting wanted keys 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 librados: calling monclient init 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient: init 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient: _add_conns ranks=[7,1,8,6,2,3,4,5,0] 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient(hunting): picked mon.noname-i con 0x7fb3c00694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient(hunting): picked mon.noname-f con 0x7fb3c00a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient(hunting): picked mon.noname-e con 0x7fb3c00a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c01376a0 con 0x7fb3c00a46a0 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c00a9fa0 con 0x7fb3c00694b0 2024-08-19T09:52:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c0136460 con 0x7fb3c00a6830 2024-08-19T09:52:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c8460640 10 monclient(hunting): authenticate will time out at 820.296625s 2024-08-19T09:52:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c69fd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fb3c00a6830 0x7fb3c01d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:43722/0 (socket says 172.21.15.1:43722) 2024-08-19T09:52:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.220+0000 7fb3c69fd640 1 -- 172.21.15.1:0/3285138446 learned_addr learned my addr 172.21.15.1:0/3285138446 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:44.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.222+0000 7fb3c69fd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.1:6793/0 conn(0x7fb3c00a6830 legacy=0x7fb3c01d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.222+0000 7fb3b6ffd640 10 client.?.objecter ms_handle_connect 0x7fb3c00a6830 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.222+0000 7fb3b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c61fc640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3c00694b0 legacy=0x7fb3c0141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255699339 0 0) 0x7fb3c0136460 con 0x7fb3c00a6830 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): my global_id is 27044 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c59fb640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6789/0 conn(0x7fb3c00a46a0 legacy=0x7fb3c01d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a4004eb0 con 0x7fb3c00a6830 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 client.?.objecter ms_handle_connect 0x7fb3c00694b0 2024-08-19T09:52:44.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 client.?.objecter ms_handle_connect 0x7fb3c00a46a0 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3351790615 0 0) 0x7fb3c006dbb0 con 0x7fb3c00694b0 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): my global_id is 134341 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3c0136460 con 0x7fb3c00694b0 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338885938 0 0) 0x7fb3c01376a0 con 0x7fb3c00a46a0 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): my global_id is 28045 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3c006dbb0 con 0x7fb3c00a46a0 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1128481598 0 0) 0x7fb3a4004eb0 con 0x7fb3c00a6830 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3c01376a0 con 0x7fb3c00a6830 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3b8004b60 con 0x7fb3c00a6830 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): dump: 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 346505933 0 0) 0x7fb3c0136460 con 0x7fb3c00694b0 2024-08-19T09:52:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3a4004eb0 con 0x7fb3c00694b0 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3bc004b20 con 0x7fb3c00694b0 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient(hunting): dump: 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2659201203 0 0) 0x7fb3c01376a0 con 0x7fb3c00a6830 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3c00694b0 legacy=0x7fb3c0141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6789/0 conn(0x7fb3c00a46a0 legacy=0x7fb3c01d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 monclient: found mon.e 2024-08-19T09:52:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c01387c0 con 0x7fb3c00a6830 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: _finish_auth 0 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 5 monclient: authenticate success, global_id 27044 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 10 monclient: _renew_subs 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c0137f20 con 0x7fb3c00a6830 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 10 client.27044.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 10 monclient: _renew_subs 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:52:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3c01371a0 con 0x7fb3c00a6830 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3c8460640 1 librados: init done 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3b8004fb0 con 0x7fb3c00a6830 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3b8005520 con 0x7fb3c00a6830 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b7fff640 4 set_mon_vals no callback set 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.223+0000 7fb3b6ffd640 10 monclient: dump: 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.245 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb3b80056c0 con 0x7fb3c00a6830 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3c8460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb3a80065c0 con 0x7fb3a8002860 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(98..98 src has 1..98) ==== 7316+0+0 (unknown 1470481556 0 0) 0x7fb3b804ed50 con 0x7fb3c00a6830 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3b6ffd640 10 client.27044.objecter ms_dispatch 0x7fb3c0000b90 osd_map(98..98 src has 1..98) 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3b6ffd640 3 client.27044.objecter handle_osd_map got epochs [98,98] > 0 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3b6ffd640 3 client.27044.objecter handle_osd_map decoding full epoch 98 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3b6ffd640 20 client.27044.objecter dump_active .. 0 homeless 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.224+0000 7fb3c59fb640 10 monclient: get_auth_request con 0x7fb3a40414f0 auth_method 0 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.226+0000 7fb3c61fc640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3a8002860 legacy=0x7fb3a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.226+0000 7fb3b6ffd640 10 client.27044.objecter ms_handle_connect 0x7fb3a8002860 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.226+0000 7fb3b6ffd640 10 client.27044.objecter resend_mon_ops 2024-08-19T09:52:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533596723 0 0) 0x7fb3a80065c0 con 0x7fb3a8002860 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient(hunting): my global_id is 134350 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3c01376a0 con 0x7fb3a8002860 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 client.27044.objecter ms_handle_connect 0x7fb3a40414f0 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 737650021 0 0) 0x7fb3c01376a0 con 0x7fb3a8002860 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3a80065c0 con 0x7fb3a8002860 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3bc001d10 con 0x7fb3a8002860 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient: dump: 2024-08-19T09:52:44.247 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 468488410 0 0) 0x7fb3a80065c0 con 0x7fb3a8002860 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3a800a160 con 0x7fb3a8002860 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb3bc0073d0 con 0x7fb3a8002860 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3a8002860 legacy=0x7fb3a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.227+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3a8002860 legacy=0x7fb3a8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.233+0000 7fb3c8460640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:52:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.233+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb3a8009e70 con 0x7fb3a8009130 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3c61fc640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3a8009130 legacy=0x7fb3a80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 10 client.27044.objecter ms_handle_connect 0x7fb3a8009130 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 10 client.27044.objecter resend_mon_ops 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771037302 0 0) 0x7fb3a8009e70 con 0x7fb3a8009130 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 10 monclient(hunting): my global_id is 134359 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.236+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a80065c0 con 0x7fb3a8009130 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3606201727 0 0) 0x7fb3a80065c0 con 0x7fb3a8009130 2024-08-19T09:52:44.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3a8009e70 con 0x7fb3a8009130 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb3bc005b30 con 0x7fb3a8009130 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 10 monclient: dump: 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:52:44.252 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3725439071 0 0) 0x7fb3a8009e70 con 0x7fb3a8009130 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb3a800c410 con 0x7fb3a8009130 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 848167009) 0x7fb3bc007020 con 0x7fb3a8009130 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3a8009130 legacy=0x7fb3a80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.237+0000 7fb3b6ffd640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6792/0 conn(0x7fb3a8009130 legacy=0x7fb3a80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 20, 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 21298, 2024-08-19T09:52:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:52:44.258 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:52:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.260 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.261 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.263 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:52:44.264 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.267 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:52:44.268 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.269 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:52:44.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:52:44.272 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:52:44.273 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:52:44.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:52:44.274 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:52:44.274 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:52:44.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 10 librados: watch_flush enter 2024-08-19T09:52:44.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 10 librados: watch_flush exit 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 20 client.27044.objecter shutdown clearing up homeless session... 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 10 client.27044.objecter successfully canceled tick 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb3a40414f0 legacy=0x7fb3a40439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 10 monclient: shutdown 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 >> v1:172.21.15.1:6793/0 conn(0x7fb3c00a6830 legacy=0x7fb3c01d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c69fd640 1 -- 172.21.15.1:0/3285138446 reap_dead start 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c69fd640 1 -- 172.21.15.1:0/3285138446 reap_dead start 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 shutdown_connections 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 >> 172.21.15.1:0/3285138446 conn(0x7fb3c00a9020 msgr2=0x7fb3c00a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 shutdown_connections 2024-08-19T09:52:44.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 -- 172.21.15.1:0/3285138446 wait complete. 2024-08-19T09:52:44.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:44.238+0000 7fb3c8460640 1 librados: shutdown 2024-08-19T09:52:44.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:52:44.277 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b', 'i'] 2024-08-19T09:52:44.277 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['f'] 2024-08-19T09:52:44.277 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-19T09:52:44.277 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-19T09:52:44.278 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-19T09:52:44.278 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:52:44.290 INFO:tasks.ceph.mon.b:Stopped 2024-08-19T09:52:44.290 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-08-19T09:52:44.290 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-08-19T09:52:44.290 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-08-19T09:52:44.290 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:52:44.302 INFO:tasks.ceph.mon.i:Stopped 2024-08-19T09:52:44.302 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-08-19T09:52:44.302 INFO:tasks.ceph.mon.f:Sent signal 19 2024-08-19T09:52:44.302 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:52:59.303 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-08-19T09:52:59.303 INFO:tasks.ceph.mon.f:Sent signal 18 2024-08-19T09:52:59.303 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-08-19T09:52:59.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:52:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.413+0000 7fc10bf8a640 10 monclient: get_monmap_and_config 2024-08-19T09:52:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.413+0000 7fc10bf8a640 10 monclient: build_initial_monmap 2024-08-19T09:52:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.413+0000 7fc10bf8a640 10 monclient: monmap: 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:52:59.414406+0000 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:52:59.414406+0000 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:52:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:52:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:52:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:52:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:52:59.416 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:52:59.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 1 Processor -- start 2024-08-19T09:52:59.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 1 -- start start 2024-08-19T09:52:59.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 10 monclient: init 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 10 monclient: _reopen_session rank -1 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 10 monclient: _add_conns ranks=[2,3,7,5,0,4,8,1,6] 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 10 monclient(hunting): picked mon.noname-b con 0x7fc1040a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 10 monclient(hunting): picked mon.noname-g con 0x7fc1040a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 10 monclient(hunting): picked mon.noname-i con 0x7fc10413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040a3b40 con 0x7fc1040a67c0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc109d26640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fc1040a67c0 legacy=0x7fc1040a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.414+0000 7fc10bf8a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040a8130 con 0x7fc1040a6ba0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.415+0000 7fc10bf8a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10406ede0 con 0x7fc10413f5c0 2024-08-19T09:52:59.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.415+0000 7fc10bf8a640 10 monclient(hunting): _renew_subs 2024-08-19T09:52:59.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.415+0000 7fc10bf8a640 10 monclient(hunting): authenticate will time out at 835.491047s 2024-08-19T09:52:59.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.415+0000 7fc10a527640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7fc10413f5c0 legacy=0x7fc1041419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:52:59.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.415+0000 7fc109525640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fc1040a6ba0 0x7fc1040a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:46672/0 (socket says 172.21.15.1:46672) 2024-08-19T09:52:59.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.415+0000 7fc109525640 1 -- 172.21.15.1:0/2608674095 learned_addr learned my addr 172.21.15.1:0/2608674095 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:52:59.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.417+0000 7fc109525640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.82:6790/0 conn(0x7fc1040a6ba0 legacy=0x7fc1040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:52:59.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.616+0000 7fc109d26640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6790/0 conn(0x7fc1040a67c0 legacy=0x7fc1040a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:52:59.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:52:59.616+0000 7fc10a527640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.82:6792/0 conn(0x7fc10413f5c0 legacy=0x7fc1041419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:00.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:00.017+0000 7fc10a527640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.82:6792/0 conn(0x7fc10413f5c0 legacy=0x7fc1041419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:00.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:00.017+0000 7fc109d26640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6790/0 conn(0x7fc1040a67c0 legacy=0x7fc1040a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:00.819+0000 7fc10a527640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.82:6792/0 conn(0x7fc10413f5c0 legacy=0x7fc1041419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:00.819+0000 7fc109d26640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6790/0 conn(0x7fc1040a67c0 legacy=0x7fc1040a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): tick 2024-08-19T09:53:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 monclient(hunting): continuing hunt 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.82:6792/0 conn(0x7fc10413f5c0 legacy=0x7fc1041419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.82:6790/0 conn(0x7fc1040a6ba0 legacy=0x7fc1040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6790/0 conn(0x7fc1040a67c0 legacy=0x7fc1040a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient: _add_conns ranks=[8,6,4,5,0,1] 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): picked mon.noname-e con 0x7fc0e8003810 addr v1:172.21.15.1:6793/0 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): picked mon.noname-d con 0x7fc0e8007480 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e800b210 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 -- 172.21.15.1:0/2608674095 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040a92a0 con 0x7fc0e800b210 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 -- 172.21.15.1:0/2608674095 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040a3b40 con 0x7fc0e8007480 2024-08-19T09:53:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 1 -- 172.21.15.1:0/2608674095 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10406ede0 con 0x7fc0e8003810 2024-08-19T09:53:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.415+0000 7fc0fb7fe640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.417+0000 7fc10a527640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6791/0 conn(0x7fc0e800b210 legacy=0x7fc0e800d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.417+0000 7fc109d26640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6793/0 conn(0x7fc0e8003810 legacy=0x7fc0e8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:02.417+0000 7fc109525640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6792/0 conn(0x7fc0e8007480 legacy=0x7fc0e8009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.000+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2999662894 0 0) 0x7fc10406ede0 con 0x7fc0e8003810 2024-08-19T09:53:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.001+0000 7fc108d24640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.001+0000 7fc108d24640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.001+0000 7fc108d24640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.001+0000 7fc108d24640 10 monclient(hunting): my global_id is 27053 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.001+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0f0003520 con 0x7fc0e8003810 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 451795846 0 0) 0x7fc1040a3b40 con 0x7fc0e8007480 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 10 monclient(hunting): my global_id is 26556 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc10406ede0 con 0x7fc0e8007480 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc10a527640 1 --1- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6791/0 conn(0x7fc0e800b210 0x7fc0e800d6d0 :-1 s=OPENED pgs=3153 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fc1040a92a0 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2119750302 0 0) 0x7fc0f0003520 con 0x7fc0e8003810 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6792/0 conn(0x7fc0e8007480 legacy=0x7fc0e8009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6791/0 conn(0x7fc0e800b210 legacy=0x7fc0e800d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 1 monclient: found mon.noname-e 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc10a527640 1 -- 172.21.15.1:0/2608674095 reap_dead start 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.002+0000 7fc108d24640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e80120b0 con 0x7fc0e8003810 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc108d24640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc108d24640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc108d24640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc1000045b0 con 0x7fc0e8003810 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.003+0000 7fc108d24640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: dump: 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228392232 0 0) 0x7fc1040a92a0 con 0x7fc0e800b210 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1000047c0 con 0x7fc0e8003810 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 1 -- 172.21.15.1:0/2608674095 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc100004cb0 con 0x7fc0e8003810 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc108d24640 10 monclient: dump: 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.012 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc10bf8a640 5 monclient: authenticate success, global_id 27053 2024-08-19T09:53:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc10bf8a640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.004+0000 7fc10bf8a640 4 set_mon_vals no callback set 2024-08-19T09:53:04.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.005+0000 7fc10bf8a640 10 monclient: shutdown 2024-08-19T09:53:04.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.006+0000 7fc10bf8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.006+0000 7fc10bf8a640 1 -- 172.21.15.1:0/2608674095 >> v1:172.21.15.1:6793/0 conn(0x7fc0e8003810 legacy=0x7fc0e8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.006+0000 7fc10bf8a640 1 -- 172.21.15.1:0/2608674095 shutdown_connections 2024-08-19T09:53:04.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.006+0000 7fc10bf8a640 1 -- 172.21.15.1:0/2608674095 >> 172.21.15.1:0/2608674095 conn(0x7fc1040ac4b0 msgr2=0x7fc1040a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.007+0000 7fc10bf8a640 1 -- 172.21.15.1:0/2608674095 shutdown_connections 2024-08-19T09:53:04.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.007+0000 7fc10bf8a640 1 -- 172.21.15.1:0/2608674095 wait complete. 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.008+0000 7fc10bf8a640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.008+0000 7fc10bf8a640 10 monclient: monmap: 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.016 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.008+0000 7fc10bf8a640 1 librados: starting msgr at 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.008+0000 7fc10bf8a640 1 librados: starting objecter 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.010+0000 7fc10bf8a640 1 Processor -- start 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.012+0000 7fc10bf8a640 1 -- start start 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.012+0000 7fc10bf8a640 1 librados: setting wanted keys 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.012+0000 7fc10bf8a640 1 librados: calling monclient init 2024-08-19T09:53:04.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.012+0000 7fc10bf8a640 10 monclient: init 2024-08-19T09:53:04.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.013+0000 7fc10bf8a640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.013+0000 7fc10bf8a640 10 monclient: _add_conns ranks=[3,4,7,1,2,6,5,0,8] 2024-08-19T09:53:04.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.013+0000 7fc10bf8a640 10 monclient(hunting): picked mon.noname-g con 0x7fc0e8003810 addr v1:172.21.15.82:6790/0 2024-08-19T09:53:04.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.014+0000 7fc109d26640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fc0e8003810 0x7fc10413cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:46678/0 (socket says 172.21.15.1:46678) 2024-08-19T09:53:04.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.016+0000 7fc109d26640 1 -- 172.21.15.1:0/4219003431 learned_addr learned my addr 172.21.15.1:0/4219003431 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.016+0000 7fc10bf8a640 10 monclient(hunting): picked mon.noname-c con 0x7fc104126ce0 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.016+0000 7fc10bf8a640 10 monclient(hunting): picked mon.noname-i con 0x7fc1041ca780 addr v1:172.21.15.82:6792/0 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.016+0000 7fc10a527640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.82:6792/0 conn(0x7fc1041ca780 legacy=0x7fc1041d6770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.016+0000 7fc10bf8a640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040a1d40 con 0x7fc0e8003810 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.016+0000 7fc10bf8a640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc104126180 con 0x7fc104126ce0 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.017+0000 7fc10bf8a640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1041284e0 con 0x7fc1041ca780 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.017+0000 7fc10bf8a640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.017+0000 7fc10bf8a640 10 monclient(hunting): authenticate will time out at 840.094197s 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc109525640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.1:6791/0 conn(0x7fc104126ce0 legacy=0x7fc1041d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc109d26640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.82:6790/0 conn(0x7fc0e8003810 legacy=0x7fc10413cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc104126ce0 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc0e8003810 2024-08-19T09:53:04.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070010852 0 0) 0x7fc104126180 con 0x7fc104126ce0 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 10 monclient(hunting): my global_id is 26554 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.019+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0e800f820 con 0x7fc104126ce0 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2771844963 0 0) 0x7fc0e800f820 con 0x7fc104126ce0 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc104126180 con 0x7fc104126ce0 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0f4001c80 con 0x7fc104126ce0 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 10 monclient(hunting): dump: 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.024 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.026 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.020+0000 7fc0fa7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:04.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1848460761 0 0) 0x7fc104126180 con 0x7fc104126ce0 2024-08-19T09:53:04.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.82:6792/0 conn(0x7fc1041ca780 legacy=0x7fc1041d6770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:04.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.82:6790/0 conn(0x7fc0e8003810 legacy=0x7fc10413cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 1 monclient: found mon.c 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10413d250 con 0x7fc104126ce0 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc0fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc10bf8a640 5 monclient: authenticate success, global_id 26554 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc10bf8a640 10 monclient: _renew_subs 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc10bf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc10bf8a640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1041d7e90 con 0x7fc104126ce0 2024-08-19T09:53:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc10bf8a640 10 client.26554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.021+0000 7fc10bf8a640 10 monclient: _renew_subs 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.022+0000 7fc10bf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.022+0000 7fc10bf8a640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1041d8240 con 0x7fc104126ce0 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.022+0000 7fc10bf8a640 1 librados: init done 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.023+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0f4004b10 con 0x7fc104126ce0 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.023+0000 7fc0fa7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.023+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0f4005080 con 0x7fc104126ce0 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.023+0000 7fc0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.023+0000 7fc0fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.023+0000 7fc0fb7fe640 4 set_mon_vals no callback set 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.024+0000 7fc0fa7fc640 10 monclient: dump: 2024-08-19T09:53:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.031 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.024+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc0f40052a0 con 0x7fc104126ce0 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.024+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(99..99 src has 1..99) ==== 7707+0+0 (unknown 1385904680 0 0) 0x7fc0f404eb30 con 0x7fc104126ce0 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.024+0000 7fc0fa7fc640 10 client.26554.objecter ms_dispatch 0x7fc104000b90 osd_map(99..99 src has 1..99) 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0fa7fc640 3 client.26554.objecter handle_osd_map got epochs [99,99] > 0 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0fa7fc640 3 client.26554.objecter handle_osd_map decoding full epoch 99 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc109d26640 10 monclient: get_auth_request con 0x7fc0e8041d70 auth_method 0 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0fa7fc640 20 client.26554.objecter dump_active .. 0 homeless 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0e7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0e7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0e7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.025+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1040a4780 con 0x7fc104126ce0 2024-08-19T09:53:04.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.028+0000 7fc0fa7fc640 10 client.26554.objecter ms_handle_connect 0x7fc0e8041d70 2024-08-19T09:53:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.037+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fc1040a4780 con 0x7fc104126ce0 2024-08-19T09:53:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.037+0000 7fc0fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.037+0000 7fc0fa7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0e7fff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0e7fff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0e7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc104125c60 con 0x7fc104126ce0 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/4219003431 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2690 (unknown 254746657 0 4194856051) 0x7fc0f40175a0 con 0x7fc104126ce0 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.324+0000 7fc0fa7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:04.327 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":30,"quorum":[0,1,3,4,5,6,8],"quorum_names":["a","f","g","c","h","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 10 librados: watch_flush enter 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 10 librados: watch_flush exit 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 20 client.26554.objecter shutdown clearing up homeless session... 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 10 client.26554.objecter successfully canceled tick 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc0e8041d70 legacy=0x7fc0e8044230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 10 monclient: shutdown 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 >> v1:172.21.15.1:6791/0 conn(0x7fc104126ce0 legacy=0x7fc1041d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 shutdown_connections 2024-08-19T09:53:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.326+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 >> 172.21.15.1:0/4219003431 conn(0x7fc1040ac4b0 msgr2=0x7fc1040a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.327+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 shutdown_connections 2024-08-19T09:53:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.327+0000 7fc0e7fff640 1 -- 172.21.15.1:0/4219003431 wait complete. 2024-08-19T09:53:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.327+0000 7fc0e7fff640 1 librados: shutdown 2024-08-19T09:53:04.339 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-08-19T09:53:04.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-08-19T09:53:04.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.438+0000 7f37f03aa640 10 monclient: get_monmap_and_config 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.438+0000 7f37f03aa640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.438+0000 7f37f03aa640 10 monclient: monmap: 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:04.439992+0000 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:04.439992+0000 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.441 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.438+0000 7f37f03aa640 1 Processor -- start 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.438+0000 7f37f03aa640 1 -- start start 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.438+0000 7f37f03aa640 10 monclient: init 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient: _add_conns ranks=[5,6,2,3,8,7,4,1,0] 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient(hunting): picked mon.noname-h con 0x7f37e80a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient(hunting): picked mon.noname-d con 0x7f37e80694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient(hunting): picked mon.noname-b con 0x7f37e80a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e80a3c70 con 0x7f37e80a46a0 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e80a33d0 con 0x7f37e80a6830 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e80a36b0 con 0x7f37e80694b0 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37ee947640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f37e80a46a0 legacy=0x7f37e8141850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37f03aa640 10 monclient(hunting): authenticate will time out at 840.516494s 2024-08-19T09:53:04.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37ed945640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f37e80694b0 0x7f37e80a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:60766/0 (socket says 172.21.15.1:60766) 2024-08-19T09:53:04.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.439+0000 7f37ed945640 1 -- 172.21.15.1:0/845953796 learned_addr learned my addr 172.21.15.1:0/845953796 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed945640 1 -- 172.21.15.1:0/845953796 >> v1:172.21.15.1:6792/0 conn(0x7f37e80694b0 legacy=0x7f37e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ee146640 1 -- 172.21.15.1:0/845953796 >> v1:172.21.15.82:6791/0 conn(0x7f37e80a6830 legacy=0x7f37e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 795047149 0 0) 0x7f37e80a36b0 con 0x7f37e80694b0 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): my global_id is 26565 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d0003520 con 0x7f37e80694b0 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204511223 0 0) 0x7f37e80a33d0 con 0x7f37e80a6830 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.441+0000 7f37ed144640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient(hunting): my global_id is 26249 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e80a36b0 con 0x7f37e80a6830 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2212838238 0 0) 0x7f37d0003520 con 0x7f37e80694b0 2024-08-19T09:53:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 >> v1:172.21.15.82:6791/0 conn(0x7f37e80a6830 legacy=0x7f37e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 >> v1:172.21.15.1:6790/0 conn(0x7f37e80a46a0 legacy=0x7f37e8141850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 monclient: found mon.noname-d 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e806dbb0 con 0x7f37e80694b0 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37dc001ae0 con 0x7f37e80694b0 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 5 monclient: authenticate success, global_id 26565 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: dump: 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37dc0048a0 con 0x7f37e80694b0 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 1 -- 172.21.15.1:0/845953796 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37dc004e10 con 0x7f37e80694b0 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 4 set_mon_vals no callback set 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:53:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37ed144640 10 monclient: dump: 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.452 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 10 monclient: shutdown 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 1 -- 172.21.15.1:0/845953796 >> v1:172.21.15.1:6792/0 conn(0x7f37e80694b0 legacy=0x7f37e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 1 -- 172.21.15.1:0/845953796 shutdown_connections 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 1 -- 172.21.15.1:0/845953796 >> 172.21.15.1:0/845953796 conn(0x7f37e80a9020 msgr2=0x7f37e8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 1 -- 172.21.15.1:0/845953796 shutdown_connections 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.442+0000 7f37f03aa640 1 -- 172.21.15.1:0/845953796 wait complete. 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 10 monclient: monmap: 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:04.453 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:04.454 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:04.454 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.454 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.454 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.454 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.454 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.455 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 1 librados: starting msgr at 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 1 librados: starting objecter 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 1 Processor -- start 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 1 -- start start 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 1 librados: setting wanted keys 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 1 librados: calling monclient init 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.443+0000 7f37f03aa640 10 monclient: init 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient: _add_conns ranks=[5,6,8,2,3,1,7,0,4] 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient(hunting): picked mon.noname-h con 0x7f37e80694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient(hunting): picked mon.noname-d con 0x7f37e80a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient(hunting): picked mon.noname-e con 0x7f37e80a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:53:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e80a9dc0 con 0x7f37e80694b0 2024-08-19T09:53:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e8142a90 con 0x7f37e80a46a0 2024-08-19T09:53:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e8138e20 con 0x7f37e80a6830 2024-08-19T09:53:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37f03aa640 10 monclient(hunting): authenticate will time out at 840.521234s 2024-08-19T09:53:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37ed945640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f37e80a46a0 0x7f37e8135890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:60776/0 (socket says 172.21.15.1:60776) 2024-08-19T09:53:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.444+0000 7f37ed945640 1 -- 172.21.15.1:0/3037650950 learned_addr learned my addr 172.21.15.1:0/3037650950 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37ed945640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6792/0 conn(0x7f37e80a46a0 legacy=0x7f37e8135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 client.?.objecter ms_handle_connect 0x7f37e80a46a0 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 client.?.objecter ms_handle_connect 0x7f37e80a6830 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37ee947640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6793/0 conn(0x7f37e80a6830 legacy=0x7f37e81d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 client.?.objecter ms_handle_connect 0x7f37e80694b0 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37ee146640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.82:6791/0 conn(0x7f37e80694b0 legacy=0x7f37e8135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034671244 0 0) 0x7f37e806dbb0 con 0x7f37e80a46a0 2024-08-19T09:53:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): my global_id is 26574 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d4004ee0 con 0x7f37e80a46a0 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351478918 0 0) 0x7f37e8138e20 con 0x7f37e80a6830 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): my global_id is 27062 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e806dbb0 con 0x7f37e80a6830 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92806457 0 0) 0x7f37e80a36b0 con 0x7f37e80694b0 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): my global_id is 26258 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e8138e20 con 0x7f37e80694b0 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2796347566 0 0) 0x7f37d4004ee0 con 0x7f37e80a46a0 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37e80a36b0 con 0x7f37e80a46a0 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37dc001d30 con 0x7f37e80a46a0 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): dump: 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2807575096 0 0) 0x7f37e806dbb0 con 0x7f37e80a6830 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37d4004ee0 con 0x7f37e80a6830 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37d8004810 con 0x7f37e80a6830 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.446+0000 7f37e6ffd640 10 monclient(hunting): dump: 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4186325728 0 0) 0x7f37e80a36b0 con 0x7f37e80a46a0 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6793/0 conn(0x7f37e80a6830 legacy=0x7f37e81d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.82:6791/0 conn(0x7f37e80694b0 legacy=0x7f37e8135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 monclient: found mon.d 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e8142810 con 0x7f37e80a46a0 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 5 monclient: authenticate success, global_id 26574 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 10 monclient: _renew_subs 2024-08-19T09:53:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37e8136ac0 con 0x7f37e80a46a0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 10 client.26574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 10 monclient: _renew_subs 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f37e8138b40 con 0x7f37e80a46a0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 1 librados: init done 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37dc0048a0 con 0x7f37e80a46a0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37dc005380 con 0x7f37e80a46a0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37ec943640 4 set_mon_vals no callback set 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:53:04.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37e6ffd640 10 monclient: dump: 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.472 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.447+0000 7f37f03aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37c80065c0 con 0x7f37c8002860 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f37dc0117e0 con 0x7f37e80a46a0 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(99..99 src has 1..99) ==== 7707+0+0 (unknown 1385904680 0 0) 0x7f37dc04eaa0 con 0x7f37e80a46a0 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37e6ffd640 10 client.26574.objecter ms_dispatch 0x7f37e8000b90 osd_map(99..99 src has 1..99) 2024-08-19T09:53:04.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37e6ffd640 3 client.26574.objecter handle_osd_map got epochs [99,99] > 0 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37e6ffd640 3 client.26574.objecter handle_osd_map decoding full epoch 99 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37ee947640 10 monclient: get_auth_request con 0x7f37d40415e0 auth_method 0 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.448+0000 7f37e6ffd640 20 client.26574.objecter dump_active .. 0 homeless 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 client.26574.objecter ms_handle_connect 0x7f37c8002860 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37ee146640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6789/0 conn(0x7f37c8002860 legacy=0x7f37c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 client.26574.objecter resend_mon_ops 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380150903 0 0) 0x7f37c80065c0 con 0x7f37c8002860 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient(hunting): my global_id is 134253 2024-08-19T09:53:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e80a36b0 con 0x7f37c8002860 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 892261288 0 0) 0x7f37e80a36b0 con 0x7f37c8002860 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c80065c0 con 0x7f37c8002860 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37e0003f10 con 0x7f37c8002860 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 10 monclient: dump: 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.475 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2454966363 0 0) 0x7f37c80065c0 con 0x7f37c8002860 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.450+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37c800a160 con 0x7f37c8002860 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.451+0000 7f37e6ffd640 10 client.26574.objecter ms_handle_connect 0x7f37d40415e0 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.451+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f37e0004190 con 0x7f37c8002860 2024-08-19T09:53:04.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.451+0000 7f37e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.451+0000 7f37e6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.451+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6789/0 conn(0x7f37c8002860 legacy=0x7f37c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.451+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6789/0 conn(0x7f37c8002860 legacy=0x7f37c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.457+0000 7f37f03aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.457+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37c8009e50 con 0x7f37c8009110 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37ee146640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6789/0 conn(0x7f37c8009110 legacy=0x7f37c80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 client.26574.objecter ms_handle_connect 0x7f37c8009110 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 client.26574.objecter resend_mon_ops 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4168427930 0 0) 0x7f37c8009e50 con 0x7f37c8009110 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient(hunting): my global_id is 134262 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c80065c0 con 0x7f37c8009110 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3821844813 0 0) 0x7f37c80065c0 con 0x7f37c8009110 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c8009e50 con 0x7f37c8009110 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37e0005b40 con 0x7f37c8009110 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 10 monclient: dump: 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3453851805 0 0) 0x7f37c8009e50 con 0x7f37c8009110 2024-08-19T09:53:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.459+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37c800c3f0 con 0x7f37c8009110 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7663 (unknown 0 0 4289743833) 0x7f37e0005fa0 con 0x7f37c8009110 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37e6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6789/0 conn(0x7f37c8009110 legacy=0x7f37c80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37e6ffd640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6789/0 conn(0x7f37c8009110 legacy=0x7f37c80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 41523, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:04.502 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 10 librados: watch_flush enter 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 10 librados: watch_flush exit 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 20 client.26574.objecter shutdown clearing up homeless session... 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 10 client.26574.objecter successfully canceled tick 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.82:6812/2549104415 conn(0x7f37d40415e0 legacy=0x7f37d4043aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 10 monclient: shutdown 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 >> v1:172.21.15.1:6792/0 conn(0x7f37e80a46a0 legacy=0x7f37e8135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37ee947640 1 -- 172.21.15.1:0/3037650950 reap_dead start 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37ee947640 1 -- 172.21.15.1:0/3037650950 reap_dead start 2024-08-19T09:53:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 shutdown_connections 2024-08-19T09:53:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 >> 172.21.15.1:0/3037650950 conn(0x7f37e80a9020 msgr2=0x7f37e80a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 shutdown_connections 2024-08-19T09:53:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 -- 172.21.15.1:0/3037650950 wait complete. 2024-08-19T09:53:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.460+0000 7f37f03aa640 1 librados: shutdown 2024-08-19T09:53:04.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:04.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-08-19T09:53:04.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:53:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.603+0000 7f5a9ecc7640 10 monclient: get_monmap_and_config 2024-08-19T09:53:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.603+0000 7f5a9ecc7640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.603+0000 7f5a9ecc7640 10 monclient: monmap: 2024-08-19T09:53:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:04.605416+0000 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:04.605416+0000 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.607 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.607 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 1 Processor -- start 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 1 -- start start 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient: init 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient: _add_conns ranks=[6,4,3,2,7,5,1,0,8] 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient(hunting): picked mon.noname-d con 0x7f5a980a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient(hunting): picked mon.noname-c con 0x7f5a980a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:04.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient(hunting): picked mon.noname-g con 0x7f5a9813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a980a3b40 con 0x7f5a9813f5c0 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a980a8130 con 0x7f5a980a6ba0 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9806ede0 con 0x7f5a980a67c0 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ecc7640 10 monclient(hunting): authenticate will time out at 840.681769s 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ca63640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5a980a67c0 0x7f5a980a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:60786/0 (socket says 172.21.15.1:60786) 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a9ca63640 1 -- 172.21.15.1:0/3184433078 learned_addr learned my addr 172.21.15.1:0/3184433078 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.604+0000 7f5a97fff640 1 --1- 172.21.15.1:0/3184433078 >> v1:172.21.15.1:6791/0 conn(0x7f5a980a6ba0 0x7f5a980a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44462/0 (socket says 172.21.15.1:44462) 2024-08-19T09:53:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ca63640 1 -- 172.21.15.1:0/3184433078 >> v1:172.21.15.1:6792/0 conn(0x7f5a980a67c0 legacy=0x7f5a980a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9d264640 1 -- 172.21.15.1:0/3184433078 >> v1:172.21.15.82:6790/0 conn(0x7f5a9813f5c0 legacy=0x7f5a981419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a97fff640 1 -- 172.21.15.1:0/3184433078 >> v1:172.21.15.1:6791/0 conn(0x7f5a980a6ba0 legacy=0x7f5a980a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757947375 0 0) 0x7f5a9806ede0 con 0x7f5a980a67c0 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): my global_id is 26583 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a88003520 con 0x7f5a980a67c0 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593484605 0 0) 0x7f5a980a3b40 con 0x7f5a9813f5c0 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): my global_id is 134247 2024-08-19T09:53:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a9806ede0 con 0x7f5a9813f5c0 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1640021320 0 0) 0x7f5a980a8130 con 0x7f5a980a6ba0 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient(hunting): my global_id is 26563 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a980a3b40 con 0x7f5a980a6ba0 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1195362997 0 0) 0x7f5a9806ede0 con 0x7f5a9813f5c0 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ca63640 1 --1- 172.21.15.1:0/3184433078 >> v1:172.21.15.1:6792/0 conn(0x7f5a980a67c0 0x7f5a980a3f60 :-1 s=OPENED pgs=3218 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f5a8c001ad0 mon_map magic: 0 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 >> v1:172.21.15.1:6792/0 conn(0x7f5a980a67c0 legacy=0x7f5a980a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 >> v1:172.21.15.1:6791/0 conn(0x7f5a980a6ba0 legacy=0x7f5a980a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 monclient: found mon.noname-g 2024-08-19T09:53:04.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a980a92a0 con 0x7f5a9813f5c0 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ecc7640 5 monclient: authenticate success, global_id 134247 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ecc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a90001e90 con 0x7f5a9813f5c0 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: dump: 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.620 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ecc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a8c001ad0 con 0x7f5a980a67c0 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a90003990 con 0x7f5a9813f5c0 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ecc7640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 1 -- 172.21.15.1:0/3184433078 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a90003f40 con 0x7f5a9813f5c0 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a9ecc7640 4 set_mon_vals no callback set 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.607+0000 7f5a977fe640 10 monclient: dump: 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.621 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.622 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.623 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.623 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.623 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.623 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 10 monclient: shutdown 2024-08-19T09:53:04.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/3184433078 >> v1:172.21.15.82:6790/0 conn(0x7f5a9813f5c0 legacy=0x7f5a981419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/3184433078 shutdown_connections 2024-08-19T09:53:04.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/3184433078 >> 172.21.15.1:0/3184433078 conn(0x7f5a980ac4b0 msgr2=0x7f5a980a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/3184433078 shutdown_connections 2024-08-19T09:53:04.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/3184433078 wait complete. 2024-08-19T09:53:04.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 10 monclient: monmap: 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.625 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.626 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.626 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.626 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.626 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.626 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.626 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.627 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.627 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.627 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.627 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 librados: starting msgr at 2024-08-19T09:53:04.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.608+0000 7f5a9ecc7640 1 librados: starting objecter 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 Processor -- start 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 -- start start 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 librados: setting wanted keys 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 librados: calling monclient init 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient: init 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient: _add_conns ranks=[3,2,5,8,4,6,0,7,1] 2024-08-19T09:53:04.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient(hunting): picked mon.noname-g con 0x7f5a980a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:53:04.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient(hunting): picked mon.noname-b con 0x7f5a980a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:04.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient(hunting): picked mon.noname-h con 0x7f5a9813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:04.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a980a2890 con 0x7f5a980a6ba0 2024-08-19T09:53:04.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a980a15c0 con 0x7f5a980a67c0 2024-08-19T09:53:04.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9813a860 con 0x7f5a9813f5c0 2024-08-19T09:53:04.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ecc7640 10 monclient(hunting): authenticate will time out at 840.686679s 2024-08-19T09:53:04.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a97fff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f5a980a6ba0 legacy=0x7f5a980a0610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:04.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ca63640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f5a980a67c0 0x7f5a9809ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:46694/0 (socket says 172.21.15.1:46694) 2024-08-19T09:53:04.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.609+0000 7f5a9ca63640 1 -- 172.21.15.1:0/4006922736 learned_addr learned my addr 172.21.15.1:0/4006922736 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.611+0000 7f5a9ca63640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.82:6790/0 conn(0x7f5a980a67c0 legacy=0x7f5a9809ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.611+0000 7f5a9d264640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.82:6791/0 conn(0x7f5a9813f5c0 legacy=0x7f5a981d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.611+0000 7f5a957fa640 10 client.?.objecter ms_handle_connect 0x7f5a980a67c0 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 client.?.objecter ms_handle_connect 0x7f5a9813f5c0 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471352508 0 0) 0x7f5a980a15c0 con 0x7f5a980a67c0 2024-08-19T09:53:04.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): my global_id is 134256 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a7c004eb0 con 0x7f5a980a67c0 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810110175 0 0) 0x7f5a9813a860 con 0x7f5a9813f5c0 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): my global_id is 26267 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a980a15c0 con 0x7f5a9813f5c0 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3887005587 0 0) 0x7f5a7c004eb0 con 0x7f5a980a67c0 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a9813a860 con 0x7f5a980a67c0 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a8c001a20 con 0x7f5a980a67c0 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:53:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): dump: 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1866894315 0 0) 0x7f5a980a15c0 con 0x7f5a9813f5c0 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a7c004eb0 con 0x7f5a9813f5c0 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a90003e80 con 0x7f5a9813f5c0 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient(hunting): dump: 2024-08-19T09:53:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.639 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.640 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 462054325 0 0) 0x7f5a9813a860 con 0x7f5a980a67c0 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.82:6791/0 conn(0x7f5a9813f5c0 legacy=0x7f5a981d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6790/0 conn(0x7f5a980a6ba0 legacy=0x7f5a980a0610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 monclient: found mon.g 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:53:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a98139fc0 con 0x7f5a980a67c0 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 5 monclient: authenticate success, global_id 134256 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 10 monclient: _renew_subs 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9813f1b0 con 0x7f5a980a67c0 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 10 client.134256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 10 monclient: _renew_subs 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a980a0d20 con 0x7f5a980a67c0 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a9ecc7640 1 librados: init done 2024-08-19T09:53:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a8c004e50 con 0x7f5a980a67c0 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a8c005380 con 0x7f5a980a67c0 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a96ffd640 4 set_mon_vals no callback set 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.612+0000 7f5a957fa640 10 monclient: dump: 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.643 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a9ecc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a780065c0 con 0x7f5a78002860 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5a8c005520 con 0x7f5a980a67c0 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(99..99 src has 1..99) ==== 7707+0+0 (unknown 1385904680 0 0) 0x7f5a8c04ecd0 con 0x7f5a980a67c0 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a957fa640 10 client.134256.objecter ms_dispatch 0x7f5a98000b90 osd_map(99..99 src has 1..99) 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a957fa640 3 client.134256.objecter handle_osd_map got epochs [99,99] > 0 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a957fa640 3 client.134256.objecter handle_osd_map decoding full epoch 99 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a957fa640 20 client.134256.objecter dump_active .. 0 homeless 2024-08-19T09:53:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.613+0000 7f5a9d264640 10 monclient: get_auth_request con 0x7f5a7c041460 auth_method 0 2024-08-19T09:53:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a97fff640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6791/0 conn(0x7f5a78002860 legacy=0x7f5a78004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 client.134256.objecter ms_handle_connect 0x7f5a78002860 2024-08-19T09:53:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 client.134256.objecter resend_mon_ops 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632765117 0 0) 0x7f5a780065c0 con 0x7f5a78002860 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient(hunting): my global_id is 26572 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a9813a860 con 0x7f5a78002860 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 650330975 0 0) 0x7f5a9813a860 con 0x7f5a78002860 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a780065c0 con 0x7f5a78002860 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a84001930 con 0x7f5a78002860 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.615+0000 7f5a957fa640 10 monclient: dump: 2024-08-19T09:53:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 490851051 0 0) 0x7f5a780065c0 con 0x7f5a78002860 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a7800a160 con 0x7f5a78002860 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 10 client.134256.objecter ms_handle_connect 0x7f5a7c041460 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5a84006840 con 0x7f5a78002860 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6791/0 conn(0x7f5a78002860 legacy=0x7f5a78004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.616+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6791/0 conn(0x7f5a78002860 legacy=0x7f5a78004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.622+0000 7f5a9ecc7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.622+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a78009e70 con 0x7f5a78009130 2024-08-19T09:53:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.624+0000 7f5a957fa640 10 client.134256.objecter ms_handle_connect 0x7f5a78009130 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 client.134256.objecter resend_mon_ops 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.624+0000 7f5a97fff640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6791/0 conn(0x7f5a78009130 legacy=0x7f5a780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1327562037 0 0) 0x7f5a78009e70 con 0x7f5a78009130 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient(hunting): my global_id is 26581 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a780065c0 con 0x7f5a78009130 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3682618244 0 0) 0x7f5a780065c0 con 0x7f5a78009130 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a78009e70 con 0x7f5a78009130 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a84004f10 con 0x7f5a78009130 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient: dump: 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3789973360 0 0) 0x7f5a78009e70 con 0x7f5a78009130 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a7800c410 con 0x7f5a78009130 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7662 (unknown 0 0 3367647629) 0x7f5a84006500 con 0x7f5a78009130 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6791/0 conn(0x7f5a78009130 legacy=0x7f5a780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.625+0000 7f5a957fa640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.1:6791/0 conn(0x7f5a78009130 legacy=0x7f5a780095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 190897, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.656 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.659 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.660 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.661 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:04.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.663 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.666 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:04.667 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:04.667 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:04.671 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.672 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:04.676 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 10 librados: watch_flush enter 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 10 librados: watch_flush exit 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 20 client.134256.objecter shutdown clearing up homeless session... 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 10 client.134256.objecter successfully canceled tick 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5a7c041460 legacy=0x7f5a7c043920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 10 monclient: shutdown 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 >> v1:172.21.15.82:6790/0 conn(0x7f5a980a67c0 legacy=0x7f5a9809ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9d264640 1 -- 172.21.15.1:0/4006922736 reap_dead start 2024-08-19T09:53:04.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9d264640 1 -- 172.21.15.1:0/4006922736 reap_dead start 2024-08-19T09:53:04.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 shutdown_connections 2024-08-19T09:53:04.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 >> 172.21.15.1:0/4006922736 conn(0x7f5a980ac4b0 msgr2=0x7f5a980a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 shutdown_connections 2024-08-19T09:53:04.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 -- 172.21.15.1:0/4006922736 wait complete. 2024-08-19T09:53:04.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.626+0000 7f5a9ecc7640 1 librados: shutdown 2024-08-19T09:53:04.679 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:04.679 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-08-19T09:53:04.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:53:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.777+0000 7f6f848cc640 10 monclient: get_monmap_and_config 2024-08-19T09:53:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.777+0000 7f6f848cc640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.777+0000 7f6f848cc640 10 monclient: monmap: 2024-08-19T09:53:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:04.779064+0000 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:04.779064+0000 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.780 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.777+0000 7f6f848cc640 1 Processor -- start 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.777+0000 7f6f848cc640 1 -- start start 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.777+0000 7f6f848cc640 10 monclient: init 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient: _add_conns ranks=[4,5,1,0,7,8,2,3,6] 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient(hunting): picked mon.noname-h con 0x7f6f800694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:04.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient(hunting): picked mon.noname-f con 0x7f6f800a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f800a3c70 con 0x7f6f800a46a0 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f800a33d0 con 0x7f6f800a6830 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f800a36b0 con 0x7f6f800694b0 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f848cc640 10 monclient(hunting): authenticate will time out at 840.855420s 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f7e59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f6f800a6830 0x7f6f800a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44490/0 (socket says 172.21.15.1:44490) 2024-08-19T09:53:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.778+0000 7f6f7e59c640 1 -- 172.21.15.1:0/1515380987 learned_addr learned my addr 172.21.15.1:0/1515380987 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7e59c640 1 -- 172.21.15.1:0/1515380987 >> v1:172.21.15.1:6791/0 conn(0x7f6f800a6830 legacy=0x7f6f800a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7dd9b640 1 -- 172.21.15.1:0/1515380987 >> v1:172.21.15.82:6791/0 conn(0x7f6f800694b0 legacy=0x7f6f800a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7ed9d640 1 -- 172.21.15.1:0/1515380987 >> v1:172.21.15.82:6789/0 conn(0x7f6f800a46a0 legacy=0x7f6f80141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 37265932 0 0) 0x7f6f800a33d0 con 0x7f6f800a6830 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7d59a640 10 monclient(hunting): my global_id is 26590 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.780+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f6c003520 con 0x7f6f800a6830 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2863055499 0 0) 0x7f6f800a36b0 con 0x7f6f800694b0 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient(hunting): my global_id is 26276 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f800a33d0 con 0x7f6f800694b0 2024-08-19T09:53:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1515741197 0 0) 0x7f6f6c003520 con 0x7f6f800a6830 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 >> v1:172.21.15.82:6791/0 conn(0x7f6f800694b0 legacy=0x7f6f800a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 >> v1:172.21.15.82:6789/0 conn(0x7f6f800a46a0 legacy=0x7f6f80141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 monclient: found mon.noname-c 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8006dbb0 con 0x7f6f800a6830 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f70001e60 con 0x7f6f800a6830 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 5 monclient: authenticate success, global_id 26590 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: dump: 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.796 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977235749 0 0) 0x7f6f800a3c70 con 0x7f6f800a46a0 2024-08-19T09:53:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:53:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f70004c20 con 0x7f6f800a6830 2024-08-19T09:53:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 1 -- 172.21.15.1:0/1515380987 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f70005190 con 0x7f6f800a6830 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 4 set_mon_vals no callback set 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f7d59a640 10 monclient: dump: 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 10 monclient: shutdown 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 1 -- 172.21.15.1:0/1515380987 >> v1:172.21.15.1:6791/0 conn(0x7f6f800a6830 legacy=0x7f6f800a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.781+0000 7f6f848cc640 1 -- 172.21.15.1:0/1515380987 shutdown_connections 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 1 -- 172.21.15.1:0/1515380987 >> 172.21.15.1:0/1515380987 conn(0x7f6f800a9020 msgr2=0x7f6f80068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 1 -- 172.21.15.1:0/1515380987 shutdown_connections 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 1 -- 172.21.15.1:0/1515380987 wait complete. 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 10 monclient: monmap: 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.801 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 1 librados: starting msgr at 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.782+0000 7f6f848cc640 1 librados: starting objecter 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 Processor -- start 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 -- start start 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 librados: setting wanted keys 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 librados: calling monclient init 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient: init 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient: _add_conns ranks=[0,2,4,7,3,1,8,6,5] 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient(hunting): picked mon.noname-a con 0x7f6f800694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient(hunting): picked mon.noname-b con 0x7f6f800a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8013a800 con 0x7f6f800694b0 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f800a9fa0 con 0x7f6f800a46a0 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f801392e0 con 0x7f6f800a6830 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f7dd9b640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f6f800a46a0 legacy=0x7f6f801373a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f848cc640 10 monclient(hunting): authenticate will time out at 840.860688s 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f7e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f800694b0 0x7f6f80136c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49342/0 (socket says 172.21.15.1:49342) 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.783+0000 7f6f7e59c640 1 -- 172.21.15.1:0/3282949839 learned_addr learned my addr 172.21.15.1:0/3282949839 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.785+0000 7f6f7e59c640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6789/0 conn(0x7f6f800694b0 legacy=0x7f6f80136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.785+0000 7f6f76ffd640 10 client.?.objecter ms_handle_connect 0x7f6f800694b0 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.785+0000 7f6f76ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.785+0000 7f6f76ffd640 10 client.?.objecter ms_handle_connect 0x7f6f800a6830 2024-08-19T09:53:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.785+0000 7f6f76ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.785+0000 7f6f7ed9d640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6791/0 conn(0x7f6f800a6830 legacy=0x7f6f801d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 973445009 0 0) 0x7f6f8013a800 con 0x7f6f800694b0 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): my global_id is 134271 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f60004ee0 con 0x7f6f800694b0 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342940108 0 0) 0x7f6f801392e0 con 0x7f6f800a6830 2024-08-19T09:53:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): my global_id is 26599 2024-08-19T09:53:04.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8013a800 con 0x7f6f800a6830 2024-08-19T09:53:04.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3206429800 0 0) 0x7f6f8013a800 con 0x7f6f800a6830 2024-08-19T09:53:04.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f801392e0 con 0x7f6f800a6830 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f78003610 con 0x7f6f800a6830 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): dump: 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.806 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1332346256 0 0) 0x7f6f60004ee0 con 0x7f6f800694b0 2024-08-19T09:53:04.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f8013a800 con 0x7f6f800694b0 2024-08-19T09:53:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f700020b0 con 0x7f6f800694b0 2024-08-19T09:53:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:53:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient(hunting): dump: 2024-08-19T09:53:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 182651482 0 0) 0x7f6f801392e0 con 0x7f6f800a6830 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6790/0 conn(0x7f6f800a46a0 legacy=0x7f6f801373a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6789/0 conn(0x7f6f800694b0 legacy=0x7f6f80136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 monclient: found mon.c 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80137ab0 con 0x7f6f800a6830 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 5 monclient: authenticate success, global_id 26599 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2118461501 0 0) 0x7f6f8013a800 con 0x7f6f800694b0 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 10 monclient: _renew_subs 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8013a520 con 0x7f6f800a6830 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 10 client.26599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 10 monclient: _renew_subs 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f80142380 con 0x7f6f800a6830 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f848cc640 1 librados: init done 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f78003b10 con 0x7f6f800a6830 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f78004000 con 0x7f6f800a6830 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f7cd99640 4 set_mon_vals no callback set 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.786+0000 7f6f76ffd640 10 monclient: dump: 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f848cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f640065c0 con 0x7f6f64002860 2024-08-19T09:53:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6f780041a0 con 0x7f6f800a6830 2024-08-19T09:53:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(99..99 src has 1..99) ==== 7707+0+0 (unknown 1385904680 0 0) 0x7f6f7804e990 con 0x7f6f800a6830 2024-08-19T09:53:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f76ffd640 10 client.26599.objecter ms_dispatch 0x7f6f80000b90 osd_map(99..99 src has 1..99) 2024-08-19T09:53:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f76ffd640 3 client.26599.objecter handle_osd_map got epochs [99,99] > 0 2024-08-19T09:53:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.787+0000 7f6f76ffd640 3 client.26599.objecter handle_osd_map decoding full epoch 99 2024-08-19T09:53:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.788+0000 7f6f76ffd640 20 client.26599.objecter dump_active .. 0 homeless 2024-08-19T09:53:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.788+0000 7f6f7dd9b640 10 monclient: get_auth_request con 0x7f6f60041580 auth_method 0 2024-08-19T09:53:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f7e59c640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6792/0 conn(0x7f6f64002860 legacy=0x7f6f64004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 10 client.26599.objecter ms_handle_connect 0x7f6f64002860 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 10 client.26599.objecter resend_mon_ops 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204127537 0 0) 0x7f6f640065c0 con 0x7f6f64002860 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 10 monclient(hunting): my global_id is 26592 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.789+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8013a800 con 0x7f6f64002860 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 766532595 0 0) 0x7f6f8013a800 con 0x7f6f64002860 2024-08-19T09:53:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f640065c0 con 0x7f6f64002860 2024-08-19T09:53:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f70004c20 con 0x7f6f64002860 2024-08-19T09:53:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:53:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 10 monclient: dump: 2024-08-19T09:53:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3991207204 0 0) 0x7f6f640065c0 con 0x7f6f64002860 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f6400a160 con 0x7f6f64002860 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 10 client.26599.objecter ms_handle_connect 0x7f6f60041580 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6f70007970 con 0x7f6f64002860 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6792/0 conn(0x7f6f64002860 legacy=0x7f6f64004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.790+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6792/0 conn(0x7f6f64002860 legacy=0x7f6f64004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.796+0000 7f6f848cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.796+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f64009e50 con 0x7f6f64009110 2024-08-19T09:53:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f7e59c640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6792/0 conn(0x7f6f64009110 legacy=0x7f6f640095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 client.26599.objecter ms_handle_connect 0x7f6f64009110 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 client.26599.objecter resend_mon_ops 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718148867 0 0) 0x7f6f64009e50 con 0x7f6f64009110 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient(hunting): my global_id is 26601 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f640065c0 con 0x7f6f64009110 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1226626926 0 0) 0x7f6f640065c0 con 0x7f6f64009110 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f64009e50 con 0x7f6f64009110 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6f70006200 con 0x7f6f64009110 2024-08-19T09:53:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient: dump: 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1823836629 0 0) 0x7f6f64009e50 con 0x7f6f64009110 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6f6400c3f0 con 0x7f6f64009110 2024-08-19T09:53:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7662 (unknown 0 0 2592147774) 0x7f6f70007770 con 0x7f6f64009110 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6792/0 conn(0x7f6f64009110 legacy=0x7f6f640095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.799+0000 7f6f76ffd640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6792/0 conn(0x7f6f64009110 legacy=0x7f6f640095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:04.826 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 191056, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.828 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:53:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:53:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:04.853 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 10 librados: watch_flush enter 2024-08-19T09:53:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 10 librados: watch_flush exit 2024-08-19T09:53:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 20 client.26599.objecter shutdown clearing up homeless session... 2024-08-19T09:53:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 10 client.26599.objecter successfully canceled tick 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6f60041580 legacy=0x7f6f60043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 10 monclient: shutdown 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 >> v1:172.21.15.1:6791/0 conn(0x7f6f800a6830 legacy=0x7f6f801d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f7ed9d640 1 -- 172.21.15.1:0/3282949839 reap_dead start 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f7ed9d640 1 -- 172.21.15.1:0/3282949839 reap_dead start 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 shutdown_connections 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 >> 172.21.15.1:0/3282949839 conn(0x7f6f800a9020 msgr2=0x7f6f800ab150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 shutdown_connections 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 -- 172.21.15.1:0/3282949839 wait complete. 2024-08-19T09:53:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.800+0000 7f6f848cc640 1 librados: shutdown 2024-08-19T09:53:04.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:04.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-08-19T09:53:04.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:53:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.955+0000 7f4eecec5640 10 monclient: get_monmap_and_config 2024-08-19T09:53:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.955+0000 7f4eecec5640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.955+0000 7f4eecec5640 10 monclient: monmap: 2024-08-19T09:53:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:04.957012+0000 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:04.957012+0000 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.959 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.955+0000 7f4eecec5640 1 Processor -- start 2024-08-19T09:53:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.955+0000 7f4eecec5640 1 -- start start 2024-08-19T09:53:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.955+0000 7f4eecec5640 10 monclient: init 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient: _add_conns ranks=[5,2,3,7,1,8,6,4,0] 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient(hunting): picked mon.noname-h con 0x7f4ee80a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee80694b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient(hunting): picked mon.noname-g con 0x7f4ee80a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee80a3c70 con 0x7f4ee80694b0 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee80a33d0 con 0x7f4ee80a46a0 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee80a36b0 con 0x7f4ee80a6830 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4eecec5640 10 monclient(hunting): authenticate will time out at 841.033386s 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4ee5d9b640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f4ee80694b0 legacy=0x7f4ee80a7320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4ee659c640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f4ee80a6830 0x7f4ee80a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:38068/0 (socket says 172.21.15.1:38068) 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.956+0000 7f4ee659c640 1 -- 172.21.15.1:0/3857103265 learned_addr learned my addr 172.21.15.1:0/3857103265 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee659c640 1 -- 172.21.15.1:0/3857103265 >> v1:172.21.15.82:6791/0 conn(0x7f4ee80a6830 legacy=0x7f4ee80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee6d9d640 1 -- 172.21.15.1:0/3857103265 >> v1:172.21.15.82:6790/0 conn(0x7f4ee80a46a0 legacy=0x7f4ee8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366195874 0 0) 0x7f4ee80a36b0 con 0x7f4ee80a6830 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): my global_id is 26285 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed4003520 con 0x7f4ee80a6830 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016473485 0 0) 0x7f4ee80a33d0 con 0x7f4ee80a46a0 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 10 monclient(hunting): my global_id is 134265 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.958+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee80a36b0 con 0x7f4ee80a46a0 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1237753674 0 0) 0x7f4ed4003520 con 0x7f4ee80a6830 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 >> v1:172.21.15.82:6790/0 conn(0x7f4ee80a46a0 legacy=0x7f4ee8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 >> v1:172.21.15.1:6790/0 conn(0x7f4ee80694b0 legacy=0x7f4ee80a7320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 monclient: found mon.noname-h 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee806dbb0 con 0x7f4ee80a6830 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 5 monclient: authenticate success, global_id 26285 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ed8001e60 con 0x7f4ee80a6830 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: dump: 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ed8004c20 con 0x7f4ee80a6830 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 1 -- 172.21.15.1:0/3857103265 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ed8005190 con 0x7f4ee80a6830 2024-08-19T09:53:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 4 set_mon_vals no callback set 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4ee559a640 10 monclient: dump: 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 10 monclient: shutdown 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 1 -- 172.21.15.1:0/3857103265 >> v1:172.21.15.82:6791/0 conn(0x7f4ee80a6830 legacy=0x7f4ee80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 1 -- 172.21.15.1:0/3857103265 shutdown_connections 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 1 -- 172.21.15.1:0/3857103265 >> 172.21.15.1:0/3857103265 conn(0x7f4ee80a9020 msgr2=0x7f4ee8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 1 -- 172.21.15.1:0/3857103265 shutdown_connections 2024-08-19T09:53:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.959+0000 7f4eecec5640 1 -- 172.21.15.1:0/3857103265 wait complete. 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.960+0000 7f4eecec5640 10 monclient: build_initial_monmap 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.960+0000 7f4eecec5640 10 monclient: monmap: 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.960+0000 7f4eecec5640 1 librados: starting msgr at 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.960+0000 7f4eecec5640 1 librados: starting objecter 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.960+0000 7f4eecec5640 1 Processor -- start 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 1 -- start start 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 1 librados: setting wanted keys 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 1 librados: calling monclient init 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient: init 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient: _add_conns ranks=[0,7,4,3,5,6,8,2,1] 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee80694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient(hunting): picked mon.noname-i con 0x7f4ee80a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee80a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee8138c60 con 0x7f4ee80694b0 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee8135e30 con 0x7f4ee80a6830 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee80a9f10 con 0x7f4ee80a46a0 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4eecec5640 10 monclient(hunting): authenticate will time out at 841.038483s 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4ee659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4ee80694b0 0x7f4ee8135010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49350/0 (socket says 172.21.15.1:49350) 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4ee659c640 1 -- 172.21.15.1:0/2100627891 learned_addr learned my addr 172.21.15.1:0/2100627891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.961+0000 7f4ee5d9b640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.82:6792/0 conn(0x7f4ee80a46a0 legacy=0x7f4ee8135720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4ee659c640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6789/0 conn(0x7f4ee80694b0 legacy=0x7f4ee8135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 client.?.objecter ms_handle_connect 0x7f4ee80694b0 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 client.?.objecter ms_handle_connect 0x7f4ee80a6830 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4ee6d9d640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6791/0 conn(0x7f4ee80a6830 legacy=0x7f4ee81d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727964478 0 0) 0x7f4ee8138c60 con 0x7f4ee80694b0 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): my global_id is 134280 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec8004eb0 con 0x7f4ee80694b0 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799997633 0 0) 0x7f4ee8135e30 con 0x7f4ee80a6830 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 10 monclient(hunting): my global_id is 26608 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.963+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee8138c60 con 0x7f4ee80a6830 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1050923201 0 0) 0x7f4ee8138c60 con 0x7f4ee80a6830 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ee8135e30 con 0x7f4ee80a6830 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ee0003610 con 0x7f4ee80a6830 2024-08-19T09:53:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): dump: 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2829905413 0 0) 0x7f4ec8004eb0 con 0x7f4ee80694b0 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ee8138c60 con 0x7f4ee80694b0 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ed80020b0 con 0x7f4ee80694b0 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient(hunting): dump: 2024-08-19T09:53:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2388417921 0 0) 0x7f4ee8135e30 con 0x7f4ee80a6830 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.82:6792/0 conn(0x7f4ee80a46a0 legacy=0x7f4ee8135720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6789/0 conn(0x7f4ee80694b0 legacy=0x7f4ee8135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: _finish_hunting 0 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 monclient: found mon.c 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee8137df0 con 0x7f4ee80a6830 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: _finish_auth 0 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: _check_auth_tickets 2024-08-19T09:53:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 5 monclient: authenticate success, global_id 26608 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 10 monclient: _renew_subs 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee8136900 con 0x7f4ee80a6830 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 10 client.26608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 10 monclient: _renew_subs 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ee8137730 con 0x7f4ee80a6830 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4eecec5640 1 librados: init done 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ee0003a60 con 0x7f4ee80a6830 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ee0003f50 con 0x7f4ee80a6830 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4ee4d99640 4 set_mon_vals no callback set 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.964+0000 7f4edeffd640 10 monclient: dump: 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4eecec5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ecc0065c0 con 0x7f4ecc002860 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4ee00040f0 con 0x7f4ee80a6830 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(99..99 src has 1..99) ==== 7707+0+0 (unknown 1385904680 0 0) 0x7f4ee004ea30 con 0x7f4ee80a6830 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4edeffd640 10 client.26608.objecter ms_dispatch 0x7f4ee8000b90 osd_map(99..99 src has 1..99) 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4edeffd640 3 client.26608.objecter handle_osd_map got epochs [99,99] > 0 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4edeffd640 3 client.26608.objecter handle_osd_map decoding full epoch 99 2024-08-19T09:53:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4edeffd640 20 client.26608.objecter dump_active .. 0 homeless 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.965+0000 7f4ee5d9b640 10 monclient: get_auth_request con 0x7f4ec8041620 auth_method 0 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 client.26608.objecter ms_handle_connect 0x7f4ecc002860 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 client.26608.objecter resend_mon_ops 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4ee659c640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6793/0 conn(0x7f4ecc002860 legacy=0x7f4ecc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1077553471 0 0) 0x7f4ecc0065c0 con 0x7f4ecc002860 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient(hunting): my global_id is 27071 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee8135e30 con 0x7f4ecc002860 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1158974780 0 0) 0x7f4ee8135e30 con 0x7f4ecc002860 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ecc0065c0 con 0x7f4ecc002860 2024-08-19T09:53:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ed8004c20 con 0x7f4ecc002860 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.967+0000 7f4edeffd640 10 monclient: dump: 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3219642485 0 0) 0x7f4ecc0065c0 con 0x7f4ecc002860 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ecc00a160 con 0x7f4ecc002860 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 10 client.26608.objecter ms_handle_connect 0x7f4ec8041620 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4ed80057e0 con 0x7f4ecc002860 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6793/0 conn(0x7f4ecc002860 legacy=0x7f4ecc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.968+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6793/0 conn(0x7f4ecc002860 legacy=0x7f4ecc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.974+0000 7f4eecec5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.974+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4ecc009e70 con 0x7f4ecc009130 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4ee659c640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6793/0 conn(0x7f4ecc009130 legacy=0x7f4ecc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 client.26608.objecter ms_handle_connect 0x7f4ecc009130 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 client.26608.objecter resend_mon_ops 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1505111437 0 0) 0x7f4ecc009e70 con 0x7f4ecc009130 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient(hunting): my global_id is 27080 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ecc0065c0 con 0x7f4ecc009130 2024-08-19T09:53:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2890027105 0 0) 0x7f4ecc0065c0 con 0x7f4ecc009130 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ecc009e70 con 0x7f4ecc009130 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4ed8005f00 con 0x7f4ecc009130 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient: dump: 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 492187319 0 0) 0x7f4ecc009e70 con 0x7f4ecc009130 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ecc00c410 con 0x7f4ecc009130 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 2660717536) 0x7f4ed8008280 con 0x7f4ecc009130 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6793/0 conn(0x7f4ecc009130 legacy=0x7f4ecc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.977+0000 7f4edeffd640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6793/0 conn(0x7f4ecc009130 legacy=0x7f4ecc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:04.995 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 191250, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:04.996 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.997 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:04.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:05.015 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 10 librados: watch_flush enter 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 10 librados: watch_flush exit 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 20 client.26608.objecter shutdown clearing up homeless session... 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 10 client.26608.objecter successfully canceled tick 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4ec8041620 legacy=0x7f4ec8043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 10 monclient: shutdown 2024-08-19T09:53:05.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 >> v1:172.21.15.1:6791/0 conn(0x7f4ee80a6830 legacy=0x7f4ee81d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4ee6d9d640 1 -- 172.21.15.1:0/2100627891 reap_dead start 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4ee6d9d640 1 -- 172.21.15.1:0/2100627891 reap_dead start 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 shutdown_connections 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 >> 172.21.15.1:0/2100627891 conn(0x7f4ee80a9020 msgr2=0x7f4ee8133790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 shutdown_connections 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 -- 172.21.15.1:0/2100627891 wait complete. 2024-08-19T09:53:05.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:04.978+0000 7f4eecec5640 1 librados: shutdown 2024-08-19T09:53:05.018 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:05.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-08-19T09:53:05.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:53:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.135+0000 7f1333bf2640 10 monclient: get_monmap_and_config 2024-08-19T09:53:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.135+0000 7f1333bf2640 10 monclient: build_initial_monmap 2024-08-19T09:53:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.135+0000 7f1333bf2640 10 monclient: monmap: 2024-08-19T09:53:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:05.137269+0000 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:05.137269+0000 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:05.139 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.135+0000 7f1333bf2640 1 Processor -- start 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 1 -- start start 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient: init 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient: _add_conns ranks=[6,7,8,0,4,5,3,1,2] 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient(hunting): picked mon.noname-d con 0x7f132c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient(hunting): picked mon.noname-i con 0x7f132c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:53:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient(hunting): picked mon.noname-e con 0x7f132c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0a3b40 con 0x7f132c0a67c0 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0a8130 con 0x7f132c0a6ba0 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c06ede0 con 0x7f132c13f5c0 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f1333bf2640 10 monclient(hunting): authenticate will time out at 841.213599s 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f133118d640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f132c0a6ba0 legacy=0x7f132c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f133198e640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f132c0a67c0 0x7f132c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:60804/0 (socket says 172.21.15.1:60804) 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f133198e640 1 -- 172.21.15.1:0/338502772 learned_addr learned my addr 172.21.15.1:0/338502772 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.136+0000 7f133218f640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f132c13f5c0 0x7f132c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:38838/0 (socket says 172.21.15.1:38838) 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133218f640 1 -- 172.21.15.1:0/338502772 >> v1:172.21.15.1:6793/0 conn(0x7f132c13f5c0 legacy=0x7f132c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133198e640 1 -- 172.21.15.1:0/338502772 >> v1:172.21.15.1:6792/0 conn(0x7f132c0a67c0 legacy=0x7f132c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1809738938 0 0) 0x7f132c06ede0 con 0x7f132c13f5c0 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): my global_id is 27089 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1314003520 con 0x7f132c13f5c0 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285093154 0 0) 0x7f132c0a3b40 con 0x7f132c0a67c0 2024-08-19T09:53:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient(hunting): my global_id is 26610 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f132c06ede0 con 0x7f132c0a67c0 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 581974776 0 0) 0x7f1314003520 con 0x7f132c13f5c0 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 >> v1:172.21.15.82:6792/0 conn(0x7f132c0a6ba0 legacy=0x7f132c0a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 >> v1:172.21.15.1:6792/0 conn(0x7f132c0a67c0 legacy=0x7f132c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: _finish_hunting 0 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 monclient: found mon.noname-e 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:53:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0a92a0 con 0x7f132c13f5c0 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: _finish_auth 0 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: _check_auth_tickets 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f1333bf2640 5 monclient: authenticate success, global_id 27089 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f131c001e50 con 0x7f132c13f5c0 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f1333bf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: dump: 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f1333bf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1324001ac0 con 0x7f132c0a67c0 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f131c004b60 con 0x7f132c13f5c0 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 1 -- 172.21.15.1:0/338502772 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f131c005090 con 0x7f132c13f5c0 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f1333bf2640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f1333bf2640 4 set_mon_vals no callback set 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.139+0000 7f133098c640 10 monclient: dump: 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 10 monclient: shutdown 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 -- 172.21.15.1:0/338502772 >> v1:172.21.15.1:6793/0 conn(0x7f132c13f5c0 legacy=0x7f132c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 -- 172.21.15.1:0/338502772 shutdown_connections 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 -- 172.21.15.1:0/338502772 >> 172.21.15.1:0/338502772 conn(0x7f132c0ac4b0 msgr2=0x7f132c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 -- 172.21.15.1:0/338502772 shutdown_connections 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 -- 172.21.15.1:0/338502772 wait complete. 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 10 monclient: build_initial_monmap 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 10 monclient: monmap: 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 librados: starting msgr at 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.140+0000 7f1333bf2640 1 librados: starting objecter 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 Processor -- start 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 -- start start 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 librados: setting wanted keys 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 librados: calling monclient init 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient: init 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient: _add_conns ranks=[0,6,2,7,3,5,8,4,1] 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient(hunting): picked mon.noname-a con 0x7f132c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient(hunting): picked mon.noname-d con 0x7f132c0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient(hunting): picked mon.noname-b con 0x7f132c13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0a9f70 con 0x7f132c0a67c0 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c13a370 con 0x7f132c13f5c0 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c13a650 con 0x7f132c0a6ba0 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f133218f640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f132c13f5c0 legacy=0x7f132c1d62a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f1333bf2640 10 monclient(hunting): authenticate will time out at 841.218602s 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f133118d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f132c0a6ba0 0x7f132c1d2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:60808/0 (socket says 172.21.15.1:60808) 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f133198e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 0x7f132c13ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49362/0 (socket says 172.21.15.1:49362) 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.141+0000 7f133118d640 1 -- 172.21.15.1:0/1122054548 learned_addr learned my addr 172.21.15.1:0/1122054548 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f133198e640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 legacy=0x7f132c13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f133118d640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6792/0 conn(0x7f132c0a6ba0 legacy=0x7f132c1d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 client.?.objecter ms_handle_connect 0x7f132c0a67c0 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 client.?.objecter ms_handle_connect 0x7f132c0a6ba0 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2513820817 0 0) 0x7f1324001580 con 0x7f132c0a67c0 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): my global_id is 134289 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1318004ee0 con 0x7f132c0a67c0 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3168679735 0 0) 0x7f132c13a650 con 0x7f132c0a6ba0 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.143+0000 7f132a7fc640 10 monclient(hunting): my global_id is 26619 2024-08-19T09:53:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1324001580 con 0x7f132c0a6ba0 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f133198e640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 legacy=0x7f132c13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f133198e640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 legacy=0x7f132c13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f133198e640 1 --1- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 0x7f132c13ebb0 :-1 s=OPENED pgs=181 cs=1 l=1).handle_message read tag failed 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f133198e640 1 --1- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 0x7f132c13ebb0 :-1 s=OPENED pgs=181 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6789/0 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2701554687 0 0) 0x7f1324001580 con 0x7f132c0a6ba0 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f132c13a650 con 0x7f132c0a6ba0 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13200045b0 con 0x7f132c0a6ba0 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient(hunting): dump: 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.163 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3324579641 0 0) 0x7f132c13a650 con 0x7f132c0a6ba0 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6790/0 conn(0x7f132c13f5c0 legacy=0x7f132c1d62a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6789/0 conn(0x7f132c0a67c0 legacy=0x7f132c13ebb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 monclient: found mon.d 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0a21e0 con 0x7f132c0a6ba0 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 5 monclient: authenticate success, global_id 26619 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 10 monclient: _renew_subs 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f132c0a5540 con 0x7f132c0a6ba0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 10 client.26619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 10 monclient: _renew_subs 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f132c139db0 con 0x7f132c0a6ba0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f1333bf2640 1 librados: init done 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1320004b30 con 0x7f132c0a6ba0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1320005060 con 0x7f132c0a6ba0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132bfff640 4 set_mon_vals no callback set 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.144+0000 7f132a7fc640 10 monclient: dump: 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f1333bf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f130c0065c0 con 0x7f130c002860 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1320005200 con 0x7f132c0a6ba0 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(100..100 src has 1..100) ==== 7677+0+0 (unknown 4039362983 0 0) 0x7f132004ea70 con 0x7f132c0a6ba0 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f132a7fc640 10 client.26619.objecter ms_dispatch 0x7f132c000b90 osd_map(100..100 src has 1..100) 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f132a7fc640 3 client.26619.objecter handle_osd_map got epochs [100,100] > 0 2024-08-19T09:53:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f132a7fc640 3 client.26619.objecter handle_osd_map decoding full epoch 100 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f132a7fc640 20 client.26619.objecter dump_active .. 0 homeless 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.145+0000 7f133218f640 10 monclient: get_auth_request con 0x7f1318040ef0 auth_method 0 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.147+0000 7f132a7fc640 10 client.26619.objecter ms_handle_connect 0x7f130c002860 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.147+0000 7f133198e640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6789/0 conn(0x7f130c002860 legacy=0x7f130c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.147+0000 7f132a7fc640 10 client.26619.objecter resend_mon_ops 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3172467412 0 0) 0x7f130c0065c0 con 0x7f130c002860 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient(hunting): my global_id is 28063 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f132c13a650 con 0x7f130c002860 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 client.26619.objecter ms_handle_connect 0x7f1318040ef0 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1153586389 0 0) 0x7f132c13a650 con 0x7f130c002860 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f130c0065c0 con 0x7f130c002860 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1324004b20 con 0x7f130c002860 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient: dump: 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.170 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 49127871 0 0) 0x7f130c0065c0 con 0x7f130c002860 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f130c00a160 con 0x7f130c002860 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f13240060e0 con 0x7f130c002860 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6789/0 conn(0x7f130c002860 legacy=0x7f130c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.148+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6789/0 conn(0x7f130c002860 legacy=0x7f130c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.154+0000 7f1333bf2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.154+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f130c009e30 con 0x7f130c009110 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f133198e640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6789/0 conn(0x7f130c009110 legacy=0x7f130c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 client.26619.objecter ms_handle_connect 0x7f130c009110 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 client.26619.objecter resend_mon_ops 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220200384 0 0) 0x7f130c009e30 con 0x7f130c009110 2024-08-19T09:53:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient(hunting): my global_id is 28072 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f130c0065c0 con 0x7f130c009110 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1079780350 0 0) 0x7f130c0065c0 con 0x7f130c009110 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f130c009e30 con 0x7f130c009110 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1324006a00 con 0x7f130c009110 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.157+0000 7f132a7fc640 10 monclient: dump: 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2800061632 0 0) 0x7f130c009e30 con 0x7f130c009110 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f130c00c3f0 con 0x7f130c009110 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7662 (unknown 0 0 4251231146) 0x7f1324004b20 con 0x7f130c009110 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6789/0 conn(0x7f130c009110 legacy=0x7f130c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f132a7fc640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6789/0 conn(0x7f130c009110 legacy=0x7f130c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.176 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 191419, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:05.177 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-08-19T09:53:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:05.197 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 10 librados: watch_flush enter 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 10 librados: watch_flush exit 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 20 client.26619.objecter shutdown clearing up homeless session... 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 10 client.26619.objecter successfully canceled tick 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1318040ef0 legacy=0x7f13180433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 10 monclient: shutdown 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 >> v1:172.21.15.1:6792/0 conn(0x7f132c0a6ba0 legacy=0x7f132c1d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f133218f640 1 -- 172.21.15.1:0/1122054548 reap_dead start 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f133218f640 1 -- 172.21.15.1:0/1122054548 reap_dead start 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 shutdown_connections 2024-08-19T09:53:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 >> 172.21.15.1:0/1122054548 conn(0x7f132c0ac4b0 msgr2=0x7f132c0a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 shutdown_connections 2024-08-19T09:53:05.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 -- 172.21.15.1:0/1122054548 wait complete. 2024-08-19T09:53:05.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.158+0000 7f1333bf2640 1 librados: shutdown 2024-08-19T09:53:05.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:05.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-08-19T09:53:05.200 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:53:05.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.297+0000 7f9de81e7640 10 monclient: get_monmap_and_config 2024-08-19T09:53:05.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.297+0000 7f9de81e7640 10 monclient: build_initial_monmap 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.297+0000 7f9de81e7640 10 monclient: monmap: 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:05.299574+0000 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:05.299574+0000 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:05.301 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:05.301 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:05.301 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:05.301 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:05.301 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 1 Processor -- start 2024-08-19T09:53:05.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 1 -- start start 2024-08-19T09:53:05.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient: init 2024-08-19T09:53:05.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:05.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient: _add_conns ranks=[5,2,4,3,1,7,6,0,8] 2024-08-19T09:53:05.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient(hunting): picked mon.noname-h con 0x7f9de00a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient(hunting): picked mon.noname-b con 0x7f9de00a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient(hunting): picked mon.noname-c con 0x7f9de013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de00a3b40 con 0x7f9de00a6ba0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de00a8130 con 0x7f9de013f5c0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de006ede0 con 0x7f9de00a67c0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de5782640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f9de00a6ba0 legacy=0x7f9de00a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.298+0000 7f9de81e7640 10 monclient(hunting): authenticate will time out at 841.375950s 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.299+0000 7f9de6784640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f9de013f5c0 0x7f9de01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44518/0 (socket says 172.21.15.1:44518) 2024-08-19T09:53:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.299+0000 7f9de6784640 1 -- 172.21.15.1:0/587736494 learned_addr learned my addr 172.21.15.1:0/587736494 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de6784640 1 -- 172.21.15.1:0/587736494 >> v1:172.21.15.1:6791/0 conn(0x7f9de013f5c0 legacy=0x7f9de01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253193918 0 0) 0x7f9de00a8130 con 0x7f9de013f5c0 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de5f83640 1 -- 172.21.15.1:0/587736494 >> v1:172.21.15.82:6791/0 conn(0x7f9de00a67c0 legacy=0x7f9de00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): my global_id is 26617 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dcc003520 con 0x7f9de013f5c0 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946393190 0 0) 0x7f9de006ede0 con 0x7f9de00a67c0 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient(hunting): my global_id is 26294 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de00a8130 con 0x7f9de00a67c0 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 49115696 0 0) 0x7f9dcc003520 con 0x7f9de013f5c0 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 >> v1:172.21.15.82:6791/0 conn(0x7f9de00a67c0 legacy=0x7f9de00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 >> v1:172.21.15.1:6790/0 conn(0x7f9de00a6ba0 legacy=0x7f9de00a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: _finish_hunting 0 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 monclient: found mon.noname-c 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de00a92a0 con 0x7f9de013f5c0 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: _finish_auth 0 2024-08-19T09:53:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: _check_auth_tickets 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9dd8001e60 con 0x7f9de013f5c0 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de81e7640 5 monclient: authenticate success, global_id 26617 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de81e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: dump: 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de81e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dd8004c20 con 0x7f9de013f5c0 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 1 -- 172.21.15.1:0/587736494 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9dd8005190 con 0x7f9de013f5c0 2024-08-19T09:53:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de4f81640 10 monclient: dump: 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de81e7640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.301+0000 7f9de81e7640 4 set_mon_vals no callback set 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 10 monclient: shutdown 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 1 -- 172.21.15.1:0/587736494 >> v1:172.21.15.1:6791/0 conn(0x7f9de013f5c0 legacy=0x7f9de01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 1 -- 172.21.15.1:0/587736494 shutdown_connections 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 1 -- 172.21.15.1:0/587736494 >> 172.21.15.1:0/587736494 conn(0x7f9de00ac4b0 msgr2=0x7f9de00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 1 -- 172.21.15.1:0/587736494 shutdown_connections 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 1 -- 172.21.15.1:0/587736494 wait complete. 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 10 monclient: build_initial_monmap 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.302+0000 7f9de81e7640 10 monclient: monmap: 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 librados: starting msgr at 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 librados: starting objecter 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 Processor -- start 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 -- start start 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 librados: setting wanted keys 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 librados: calling monclient init 2024-08-19T09:53:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient: init 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient: _add_conns ranks=[1,2,4,7,0,5,8,6,3] 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient(hunting): picked mon.noname-f con 0x7f9de00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient(hunting): picked mon.noname-b con 0x7f9de00a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient(hunting): picked mon.noname-c con 0x7f9de013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de013a200 con 0x7f9de00a67c0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de00aa210 con 0x7f9de00a6ba0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de5782640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f9de00a6ba0 legacy=0x7f9de00a01f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de013a7c0 con 0x7f9de013f5c0 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de81e7640 10 monclient(hunting): authenticate will time out at 841.380798s 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de6784640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f9de013f5c0 0x7f9de00a0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44532/0 (socket says 172.21.15.1:44532) 2024-08-19T09:53:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.303+0000 7f9de6784640 1 -- 172.21.15.1:0/2965712041 learned_addr learned my addr 172.21.15.1:0/2965712041 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de6784640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.1:6791/0 conn(0x7f9de013f5c0 legacy=0x7f9de00a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 client.?.objecter ms_handle_connect 0x7f9de013f5c0 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de5f83640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6789/0 conn(0x7f9de00a67c0 legacy=0x7f9de009fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 client.?.objecter ms_handle_connect 0x7f9de00a67c0 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3830337180 0 0) 0x7f9de013a7c0 con 0x7f9de013f5c0 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): my global_id is 26626 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dc4004eb0 con 0x7f9de013f5c0 2024-08-19T09:53:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232099385 0 0) 0x7f9de013a200 con 0x7f9de00a67c0 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): my global_id is 28081 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de013a7c0 con 0x7f9de00a67c0 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1212611894 0 0) 0x7f9dc4004eb0 con 0x7f9de013f5c0 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9de013a200 con 0x7f9de013f5c0 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9dd80020b0 con 0x7f9de013f5c0 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): dump: 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3243952584 0 0) 0x7f9de013a7c0 con 0x7f9de00a67c0 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dc4004eb0 con 0x7f9de00a67c0 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9ddc003610 con 0x7f9de00a67c0 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient(hunting): dump: 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2633184135 0 0) 0x7f9de013a200 con 0x7f9de013f5c0 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.1:6790/0 conn(0x7f9de00a6ba0 legacy=0x7f9de00a01f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6789/0 conn(0x7f9de00a67c0 legacy=0x7f9de009fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient: _finish_hunting 0 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 monclient: found mon.c 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de0139f20 con 0x7f9de013f5c0 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient: _finish_auth 0 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 10 monclient: _check_auth_tickets 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9dd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 5 monclient: authenticate success, global_id 26626 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 10 monclient: _renew_subs 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9de00a1d50 con 0x7f9de013f5c0 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 10 client.26626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 10 monclient: _renew_subs 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9de01d7440 con 0x7f9de013f5c0 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.306+0000 7f9de81e7640 1 librados: init done 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dd80053c0 con 0x7f9de013f5c0 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9dd80058b0 con 0x7f9de013f5c0 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd7fff640 4 set_mon_vals no callback set 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 10 monclient: dump: 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.327 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9de81e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.307+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9dd8011d10 con 0x7f9de013f5c0 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.308+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(100..100 src has 1..100) ==== 7677+0+0 (unknown 4039362983 0 0) 0x7f9dd804edf0 con 0x7f9de013f5c0 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.308+0000 7f9dd67fc640 10 client.26626.objecter ms_dispatch 0x7f9de0000b90 osd_map(100..100 src has 1..100) 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.308+0000 7f9dd67fc640 3 client.26626.objecter handle_osd_map got epochs [100,100] > 0 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.308+0000 7f9dd67fc640 3 client.26626.objecter handle_osd_map decoding full epoch 100 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.308+0000 7f9dd67fc640 20 client.26626.objecter dump_active .. 0 homeless 2024-08-19T09:53:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.308+0000 7f9de5782640 10 monclient: get_auth_request con 0x7f9dc40415f0 auth_method 0 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 client.26626.objecter ms_handle_connect 0x7f9dc8002860 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 client.26626.objecter resend_mon_ops 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9de5f83640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6790/0 conn(0x7f9dc8002860 legacy=0x7f9dc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290528045 0 0) 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient(hunting): my global_id is 134274 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de013a200 con 0x7f9dc8002860 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 597655574 0 0) 0x7f9de013a200 con 0x7f9dc8002860 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9ddc003da0 con 0x7f9dc8002860 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 monclient: dump: 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 329047378 0 0) 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T09:53:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9dc800a160 con 0x7f9dc8002860 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.310+0000 7f9dd67fc640 10 client.26626.objecter ms_handle_connect 0x7f9dc40415f0 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.311+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9ddc0041a0 con 0x7f9dc8002860 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.311+0000 7f9dd67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.311+0000 7f9dd67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.311+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6790/0 conn(0x7f9dc8002860 legacy=0x7f9dc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.311+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6790/0 conn(0x7f9dc8002860 legacy=0x7f9dc8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.317+0000 7f9de81e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.317+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9dc8009e50 con 0x7f9dc8009110 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9de5f83640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6790/0 conn(0x7f9dc8009110 legacy=0x7f9dc80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 10 client.26626.objecter ms_handle_connect 0x7f9dc8009110 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 10 client.26626.objecter resend_mon_ops 2024-08-19T09:53:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778878653 0 0) 0x7f9dc8009e50 con 0x7f9dc8009110 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 10 monclient(hunting): my global_id is 134283 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.319+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dc80065c0 con 0x7f9dc8009110 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1531265521 0 0) 0x7f9dc80065c0 con 0x7f9dc8009110 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dc8009e50 con 0x7f9dc8009110 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9ddc005bc0 con 0x7f9dc8009110 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 10 monclient: dump: 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.336 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 334238811 0 0) 0x7f9dc8009e50 con 0x7f9dc8009110 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9dc800c3f0 con 0x7f9dc8009110 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7494 (unknown 0 0 224685382) 0x7f9ddc007090 con 0x7f9dc8009110 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6790/0 conn(0x7f9dc8009110 legacy=0x7f9dc80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.320+0000 7f9dd67fc640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6790/0 conn(0x7f9dc8009110 legacy=0x7f9dc80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:05.337 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 42382, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.338 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.339 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.340 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:05.341 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.342 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.346 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:05.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:05.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.355 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.357 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 10 librados: watch_flush enter 2024-08-19T09:53:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 10 librados: watch_flush exit 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 20 client.26626.objecter shutdown clearing up homeless session... 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 10 client.26626.objecter successfully canceled tick 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9dc40415f0 legacy=0x7f9dc4043ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 10 monclient: shutdown 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 >> v1:172.21.15.1:6791/0 conn(0x7f9de013f5c0 legacy=0x7f9de00a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de6784640 1 -- 172.21.15.1:0/2965712041 reap_dead start 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de6784640 1 -- 172.21.15.1:0/2965712041 reap_dead start 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 shutdown_connections 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 >> 172.21.15.1:0/2965712041 conn(0x7f9de00ac4b0 msgr2=0x7f9de00a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 shutdown_connections 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 -- 172.21.15.1:0/2965712041 wait complete. 2024-08-19T09:53:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.321+0000 7f9de81e7640 1 librados: shutdown 2024-08-19T09:53:05.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:05.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-08-19T09:53:05.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.457+0000 7f709c993640 10 monclient: get_monmap_and_config 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.457+0000 7f709c993640 10 monclient: build_initial_monmap 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.457+0000 7f709c993640 10 monclient: monmap: 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:53:05.459875+0000 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:53:05.459875+0000 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.460 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:05.461 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.458+0000 7f709c993640 1 Processor -- start 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.458+0000 7f709c993640 1 -- start start 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.458+0000 7f709c993640 10 monclient: init 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient: _add_conns ranks=[5,2,6,3,8,1,7,4,0] 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient(hunting): picked mon.noname-h con 0x7f70980a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient(hunting): picked mon.noname-b con 0x7f70980a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient(hunting): picked mon.noname-d con 0x7f709813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70980a3b40 con 0x7f70980a6ba0 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70980a8130 con 0x7f70980a67c0 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709806ede0 con 0x7f709813f5c0 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f709c993640 10 monclient(hunting): authenticate will time out at 841.536206s 2024-08-19T09:53:05.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f708ffff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f70980a6ba0 legacy=0x7f70980a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:05.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f7096d9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f709813f5c0 0x7f70981419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:60814/0 (socket says 172.21.15.1:60814) 2024-08-19T09:53:05.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.459+0000 7f7096d9d640 1 -- 172.21.15.1:0/2801416898 learned_addr learned my addr 172.21.15.1:0/2801416898 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7096d9d640 1 -- 172.21.15.1:0/2801416898 >> v1:172.21.15.1:6792/0 conn(0x7f709813f5c0 legacy=0x7f70981419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848944008 0 0) 0x7f709806ede0 con 0x7f709813f5c0 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f709659c640 1 -- 172.21.15.1:0/2801416898 >> v1:172.21.15.82:6791/0 conn(0x7f70980a67c0 legacy=0x7f70980a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): my global_id is 26628 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7084003520 con 0x7f709813f5c0 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261180443 0 0) 0x7f70980a8130 con 0x7f70980a67c0 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient(hunting): my global_id is 26303 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f709806ede0 con 0x7f70980a67c0 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 658190705 0 0) 0x7f7084003520 con 0x7f709813f5c0 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 >> v1:172.21.15.82:6791/0 conn(0x7f70980a67c0 legacy=0x7f70980a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 >> v1:172.21.15.1:6790/0 conn(0x7f70980a6ba0 legacy=0x7f70980a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 monclient: found mon.noname-d 2024-08-19T09:53:05.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70980a92a0 con 0x7f709813f5c0 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: _finish_auth 0 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7090001e50 con 0x7f709813f5c0 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f7095d9b640 10 monclient: dump: 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f709c993640 5 monclient: authenticate success, global_id 26628 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.461+0000 7f709c993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7090004c20 con 0x7f709813f5c0 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f7095d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f7095d9b640 1 -- 172.21.15.1:0/2801416898 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7090005190 con 0x7f709813f5c0 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 10 monclient: get_monmap_and_config success 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f7095d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 4 set_mon_vals no callback set 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f7095d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:53:05.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f7095d9b640 10 monclient: dump: 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.476 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 10 monclient: shutdown 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 1 -- 172.21.15.1:0/2801416898 >> v1:172.21.15.1:6792/0 conn(0x7f709813f5c0 legacy=0x7f70981419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 1 -- 172.21.15.1:0/2801416898 shutdown_connections 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 1 -- 172.21.15.1:0/2801416898 >> 172.21.15.1:0/2801416898 conn(0x7f70980ac4b0 msgr2=0x7f70980a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 1 -- 172.21.15.1:0/2801416898 shutdown_connections 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.462+0000 7f709c993640 1 -- 172.21.15.1:0/2801416898 wait complete. 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 10 monclient: build_initial_monmap 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 10 monclient: monmap: 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:53:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 1 librados: starting msgr at 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 1 librados: starting objecter 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 1 Processor -- start 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 1 -- start start 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 1 librados: setting wanted keys 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 1 librados: calling monclient init 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.463+0000 7f709c993640 10 monclient: init 2024-08-19T09:53:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient: _reopen_session rank -1 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient: _add_conns ranks=[7,2,8,6,5,0,3,4,1] 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient(hunting): picked mon.noname-i con 0x7f70980a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient(hunting): picked mon.noname-b con 0x7f70980a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient(hunting): picked mon.noname-e con 0x7f709813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709813a9b0 con 0x7f70980a6ba0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70981cf020 con 0x7f70980a67c0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70980a2880 con 0x7f709813f5c0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient(hunting): _renew_subs 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709c993640 10 monclient(hunting): authenticate will time out at 841.541189s 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f708ffff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f70980a6ba0 legacy=0x7f70981ce910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f709659c640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f70980a67c0 legacy=0x7f70981ce200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f7096d9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f709813f5c0 0x7f70981d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:38846/0 (socket says 172.21.15.1:38846) 2024-08-19T09:53:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.464+0000 7f7096d9d640 1 -- 172.21.15.1:0/3278484437 learned_addr learned my addr 172.21.15.1:0/3278484437 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f7096d9d640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.1:6793/0 conn(0x7f709813f5c0 legacy=0x7f70981d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 client.?.objecter ms_handle_connect 0x7f709813f5c0 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818532464 0 0) 0x7f70980a2880 con 0x7f709813f5c0 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): my global_id is 27098 2024-08-19T09:53:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7078004eb0 con 0x7f709813f5c0 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 470704282 0 0) 0x7f7078004eb0 con 0x7f709813f5c0 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70980a2880 con 0x7f709813f5c0 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70900020a0 con 0x7f709813f5c0 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): dump: 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3854783740 0 0) 0x7f70980a2880 con 0x7f709813f5c0 2024-08-19T09:53:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6792/0 conn(0x7f70980a67c0 legacy=0x7f70981ce200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.1:6790/0 conn(0x7f70980a6ba0 legacy=0x7f70981ce910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient: _finish_hunting 0 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 monclient: found mon.e 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.466+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70980a1620 con 0x7f709813f5c0 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 10 monclient: _finish_auth 0 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 10 monclient: _check_auth_tickets 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 5 monclient: authenticate success, global_id 27098 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 10 monclient: _renew_subs 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f709813a6d0 con 0x7f709813f5c0 2024-08-19T09:53:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 10 client.27098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:53:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 10 monclient: _renew_subs 2024-08-19T09:53:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:53:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f70980a0ee0 con 0x7f709813f5c0 2024-08-19T09:53:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 1 librados: init done 2024-08-19T09:53:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7090004c20 con 0x7f709813f5c0 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7090005700 con 0x7f709813f5c0 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709559a640 4 set_mon_vals no callback set 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 10 monclient: dump: 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f707c0065c0 con 0x7f707c002860 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.467+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7090011b60 con 0x7f709813f5c0 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.468+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(100..100 src has 1..100) ==== 7677+0+0 (unknown 4039362983 0 0) 0x7f709004edd0 con 0x7f709813f5c0 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.468+0000 7f708effd640 10 client.27098.objecter ms_dispatch 0x7f7098000b90 osd_map(100..100 src has 1..100) 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.468+0000 7f708effd640 3 client.27098.objecter handle_osd_map got epochs [100,100] > 0 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.468+0000 7f708effd640 3 client.27098.objecter handle_osd_map decoding full epoch 100 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.468+0000 7f708effd640 20 client.27098.objecter dump_active .. 0 homeless 2024-08-19T09:53:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.468+0000 7f708ffff640 10 monclient: get_auth_request con 0x7f7078040da0 auth_method 0 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f709659c640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6791/0 conn(0x7f707c002860 legacy=0x7f707c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 client.27098.objecter ms_handle_connect 0x7f707c002860 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 client.27098.objecter resend_mon_ops 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706955439 0 0) 0x7f707c0065c0 con 0x7f707c002860 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient(hunting): my global_id is 26312 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70980a2880 con 0x7f707c002860 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 client.27098.objecter ms_handle_connect 0x7f7078040da0 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1592246440 0 0) 0x7f70980a2880 con 0x7f707c002860 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f707c0065c0 con 0x7f707c002860 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7088001d20 con 0x7f707c002860 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.470+0000 7f708effd640 10 monclient: dump: 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2917520211 0 0) 0x7f707c0065c0 con 0x7f707c002860 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f707c00a160 con 0x7f707c002860 2024-08-19T09:53:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f70880057c0 con 0x7f707c002860 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6791/0 conn(0x7f707c002860 legacy=0x7f707c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.471+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6791/0 conn(0x7f707c002860 legacy=0x7f707c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.477+0000 7f709c993640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.477+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f707c009e70 con 0x7f707c009130 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f709659c640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6791/0 conn(0x7f707c009130 legacy=0x7f707c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 client.27098.objecter ms_handle_connect 0x7f707c009130 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 client.27098.objecter resend_mon_ops 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894748377 0 0) 0x7f707c009e70 con 0x7f707c009130 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient(hunting): _negotiate 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:53:05.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient(hunting): my global_id is 26321 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f707c0065c0 con 0x7f707c009130 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 389142939 0 0) 0x7f707c0065c0 con 0x7f707c009130 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f707c009e70 con 0x7f707c009130 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7088003e50 con 0x7f707c009130 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient: dump: 2024-08-19T09:53:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:53:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1107701098 0 0) 0x7f707c009e70 con 0x7f707c009130 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f707c00c410 con 0x7f707c009130 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7829 (unknown 0 0 2762462573) 0x7f7088004170 con 0x7f707c009130 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6791/0 conn(0x7f707c009130 legacy=0x7f707c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.480+0000 7f708effd640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6791/0 conn(0x7f707c009130 legacy=0x7f707c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:53:05.497 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 191747, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:53:05.498 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.499 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.500 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:53:05.502 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.503 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.504 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:53:05.506 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.507 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.510 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:53:05.511 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.512 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.514 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.517 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:53:05.518 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:53:05.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 10 librados: watch_flush enter 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 10 librados: watch_flush exit 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 20 client.27098.objecter shutdown clearing up homeless session... 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 10 client.27098.objecter successfully canceled tick 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7078040da0 legacy=0x7f7078043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 10 monclient: shutdown 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 >> v1:172.21.15.1:6793/0 conn(0x7f709813f5c0 legacy=0x7f70981d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:53:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f7096d9d640 1 -- 172.21.15.1:0/3278484437 reap_dead start 2024-08-19T09:53:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f7096d9d640 1 -- 172.21.15.1:0/3278484437 reap_dead start 2024-08-19T09:53:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 shutdown_connections 2024-08-19T09:53:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 >> 172.21.15.1:0/3278484437 conn(0x7f70980ac4b0 msgr2=0x7f70980a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:53:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 shutdown_connections 2024-08-19T09:53:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 -- 172.21.15.1:0/3278484437 wait complete. 2024-08-19T09:53:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:53:05.481+0000 7f709c993640 1 librados: shutdown 2024-08-19T09:53:05.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T09:53:05.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T09:53:11.978 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSna api_aio: Running main() from gmock_main.cc 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [----------] Global test environment set-up. 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.TooBig (3184 ms) 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3063 ms) 2024-08-19T09:53:11.979 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3030 ms) 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3588 ms) 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2625 ms) 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3650 ms) 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3026 ms) 2024-08-19T09:53:11.980 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2315 ms) 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3026 ms) 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2992 ms) 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-08-19T09:53:11.981 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3031 ms) 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3000 ms) 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2968 ms) 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3356 ms) 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-08-19T09:53:11.982 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.Flush (3018 ms) 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2992 ms) 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3013 ms) 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2592 ms) 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3803 ms) 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2617 ms) 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-08-19T09:53:11.983 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (8658 ms) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (11018 ms) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3030 ms) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (23124 ms) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2930 ms) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3002 ms) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [----------] 26 tests from LibRadosAio (112673 ms total) 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: 2024-08-19T09:53:11.984 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-08-19T09:53:13.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrit api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-08-19T09:53:13.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-08-19T09:53:13.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=12 expected=12 2024-08-19T09:53:13.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-08-19T09:53:13.085 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=3 expected=3 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=4 expected=4 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=1 expected=1 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-08-19T09:53:13.086 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=6 expected=6 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=8 expected=8 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=5 expected=5 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-08-19T09:53:13.087 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: : entry=11 expected=11 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (234 ms) 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (68706 ms) 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (37122 ms) 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] 7 tests from LibRadosList (106609 ms total) 2024-08-19T09:53:13.088 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1037 ms) 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset foo1,foo2,foo3 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo1 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo2 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo3 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset foo1,foo4,foo5 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo4 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo5 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo1 2024-08-19T09:53:13.089 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset foo6,foo7 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo7 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: foo6 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns1:foo4 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns1:foo5 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns2:foo7 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns2:foo6 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: ns1:foo1 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: :foo1 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: :foo2 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: :foo3 2024-08-19T09:53:13.090 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (44 ms) 2024-08-19T09:53:13.091 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-08-19T09:53:13.091 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 1 0 2024-08-19T09:53:13.091 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 10 0 2024-08-19T09:53:13.091 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 13 0 2024-08-19T09:53:19.187 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 7 0 2024-08-19T09:53:19.187 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 14 0 2024-08-19T09:53:19.187 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 0 0 2024-08-19T09:53:19.187 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 15 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 11 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 5 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 8 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 6 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 3 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 4 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 12 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 9 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 2 0 2024-08-19T09:53:19.188 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (89 ms) 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] 3 tests from LibRadosListEC (1170 ms total) 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2855 ms) 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] 1 test from LibRadosListNP (2855 ms total) 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [----------] Global test environment tear-down 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [==========] 11 tests from 3 test suites ran. (119821 ms total) 2024-08-19T09:53:19.189 INFO:tasks.workunit.client.0.smithi082.stdout: api_list: [ PASSED ] 11 tests. 2024-08-19T09:53:19.192 INFO:tasks.workunit.client.0.smithi082.stdout: (71471 ms total) 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (112 ms) 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (112 ms total) 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-08-19T09:53:19.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (21 ms) 2024-08-19T09:53:19.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-08-19T09:53:19.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-08-19T09:53:19.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (25 ms total) 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (30 ms) 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (12 ms) 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (42 ms total) 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (35 ms) 2024-08-19T09:53:19.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (7 ms) 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (42 ms total) 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1028 ms) 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1028 ms total) 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (119767 ms total) 2024-08-19T09:53:19.196 INFO:tasks.workunit.client.0.smithi082.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-08-19T09:53:27.013 INFO:tasks.workunit.client.0.smithi082.stdout:pshotsECPP.SnapRemovePP (1998 ms) 2024-08-19T09:53:27.013 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (21106 ms) 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2022 ms) 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (35172 ms total) 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4377 ms) 2024-08-19T09:53:27.014 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3845 ms) 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3784 ms) 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12006 ms total) 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (127587 ms total) 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-08-19T09:53:27.015 INFO:tasks.workunit.client.0.smithi082.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-08-19T09:53:36.006 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-08-19T09:53:36.006 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3191 ms) 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19007 ms) 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5325 ms) 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2975 ms) 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-08-19T09:53:36.007 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3022 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3030 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2964 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3351 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3019 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3013 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3007 ms) 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-08-19T09:53:36.008 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2581 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3810 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2608 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (8665 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (11017 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3030 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (23119 ms) 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-08-19T09:53:36.009 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3250 ms) 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2685 ms) 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3357 ms) 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2656 ms) 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3214 ms) 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2776 ms) 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-08-19T09:53:36.010 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3036 ms) 2024-08-19T09:53:36.011 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-08-19T09:53:36.011 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2959 ms) 2024-08-19T09:53:36.011 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-08-19T09:54:35.524 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-19T09:54:35.524 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-19T09:54:35.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-19T09:54:35.529 INFO:tasks.ceph.mon.b:Started 2024-08-19T09:54:35.529 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-08-19T09:54:35.530 INFO:tasks.ceph.mon.i:Restarting daemon 2024-08-19T09:54:35.530 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-08-19T09:54:35.533 INFO:tasks.ceph.mon.i:Started 2024-08-19T09:54:35.533 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-08-19T09:54:35.533 INFO:tasks.ceph.mon.f:Sent signal 19 2024-08-19T09:54:35.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:54:50.534 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-08-19T09:54:50.535 INFO:tasks.ceph.mon.f:Sent signal 18 2024-08-19T09:54:50.535 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:54:50.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:54:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.653+0000 7fdacc7ba640 10 monclient: get_monmap_and_config 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.653+0000 7fdacc7ba640 10 monclient: build_initial_monmap 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.653+0000 7fdacc7ba640 10 monclient: monmap: 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:50.654655+0000 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:50.654655+0000 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:50.656 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.654+0000 7fdacc7ba640 1 Processor -- start 2024-08-19T09:54:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.654+0000 7fdacc7ba640 1 -- start start 2024-08-19T09:54:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.654+0000 7fdacc7ba640 10 monclient: init 2024-08-19T09:54:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.654+0000 7fdacc7ba640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 10 monclient: _add_conns ranks=[4,3,7,5,6,0,8,1,2] 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 10 monclient(hunting): picked mon.noname-c con 0x7fdac40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 10 monclient(hunting): picked mon.noname-g con 0x7fdac40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 10 monclient(hunting): picked mon.noname-i con 0x7fdac413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac40a3b40 con 0x7fdac40a6ba0 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac40a8130 con 0x7fdac40a67c0 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac406ede0 con 0x7fdac413f5c0 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdacc7ba640 10 monclient(hunting): authenticate will time out at 946.731102s 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdaca556640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fdac40a67c0 0x7fdac40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59348/0 (socket says 172.21.15.1:59348) 2024-08-19T09:54:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.655+0000 7fdaca556640 1 -- 172.21.15.1:0/3800546765 learned_addr learned my addr 172.21.15.1:0/3800546765 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdaca556640 1 -- 172.21.15.1:0/3800546765 >> v1:172.21.15.1:6791/0 conn(0x7fdac40a67c0 legacy=0x7fdac40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9d55640 1 -- 172.21.15.1:0/3800546765 >> v1:172.21.15.82:6790/0 conn(0x7fdac40a6ba0 legacy=0x7fdac40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdacad57640 1 -- 172.21.15.1:0/3800546765 >> v1:172.21.15.82:6792/0 conn(0x7fdac413f5c0 legacy=0x7fdac41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 878673720 0 0) 0x7fdac406ede0 con 0x7fdac413f5c0 2024-08-19T09:54:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9554640 10 monclient(hunting): _negotiate 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9554640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9554640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9554640 10 monclient(hunting): my global_id is 144106 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.657+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab0003520 con 0x7fdac413f5c0 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 127488481 0 0) 0x7fdab0003520 con 0x7fdac413f5c0 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 >> v1:172.21.15.1:6791/0 conn(0x7fdac40a67c0 legacy=0x7fdac40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 >> v1:172.21.15.82:6790/0 conn(0x7fdac40a6ba0 legacy=0x7fdac40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: _finish_hunting 0 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 monclient: found mon.noname-i 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac40a92a0 con 0x7fdac413f5c0 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: _finish_auth 0 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: _check_auth_tickets 2024-08-19T09:54:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdabc001ea0 con 0x7fdac413f5c0 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 5 monclient: authenticate success, global_id 144106 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: dump: 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdabc003990 con 0x7fdac413f5c0 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 1 -- 172.21.15.1:0/3800546765 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdabc003f00 con 0x7fdac413f5c0 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 4 set_mon_vals no callback set 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdac9554640 10 monclient: dump: 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.672 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:50.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 10 monclient: shutdown 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3800546765 >> v1:172.21.15.82:6792/0 conn(0x7fdac413f5c0 legacy=0x7fdac41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3800546765 shutdown_connections 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3800546765 >> 172.21.15.1:0/3800546765 conn(0x7fdac40ac4b0 msgr2=0x7fdac40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3800546765 shutdown_connections 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.658+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3800546765 wait complete. 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 10 monclient: build_initial_monmap 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 10 monclient: monmap: 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:50.674 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:50.675 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 1 librados: starting msgr at 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 1 librados: starting objecter 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 1 Processor -- start 2024-08-19T09:54:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 1 -- start start 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 1 librados: setting wanted keys 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 1 librados: calling monclient init 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.659+0000 7fdacc7ba640 10 monclient: init 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient: _add_conns ranks=[6,8,1,7,0,2,3,5,4] 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient(hunting): picked mon.noname-d con 0x7fdac40a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient(hunting): picked mon.noname-e con 0x7fdac40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient(hunting): picked mon.noname-f con 0x7fdac413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac40a9cc0 con 0x7fdac413f5c0 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac413bbc0 con 0x7fdac40a67c0 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac413bea0 con 0x7fdac40a6ba0 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdaca556640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdac40a67c0 0x7fdac41ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52772/0 (socket says 172.21.15.1:52772) 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdacc7ba640 10 monclient(hunting): authenticate will time out at 946.736451s 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.660+0000 7fdaca556640 1 -- 172.21.15.1:0/3250727087 learned_addr learned my addr 172.21.15.1:0/3250727087 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdaca556640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.1:6792/0 conn(0x7fdac40a67c0 legacy=0x7fdac41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdabaffd640 10 client.?.objecter ms_handle_connect 0x7fdac40a67c0 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdabaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdabaffd640 10 client.?.objecter ms_handle_connect 0x7fdac40a6ba0 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdabaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdac9d55640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.1:6793/0 conn(0x7fdac40a6ba0 legacy=0x7fdac41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdabaffd640 10 client.?.objecter ms_handle_connect 0x7fdac413f5c0 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdabaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.662+0000 7fdacad57640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.82:6789/0 conn(0x7fdac413f5c0 legacy=0x7fdac41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:50.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205138513 0 0) 0x7fdac40a92a0 con 0x7fdac413f5c0 2024-08-19T09:54:50.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:50.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): my global_id is 28279 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaa8004ee0 con 0x7fdac413f5c0 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022425181 0 0) 0x7fdac413bea0 con 0x7fdac40a6ba0 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): my global_id is 27350 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac40a92a0 con 0x7fdac40a6ba0 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 373493019 0 0) 0x7fdaa8004ee0 con 0x7fdac413f5c0 2024-08-19T09:54:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdac413bea0 con 0x7fdac413f5c0 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdabc003e80 con 0x7fdac413f5c0 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): dump: 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3485498661 0 0) 0x7fdac40a92a0 con 0x7fdac40a6ba0 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdaa8004ee0 con 0x7fdac40a6ba0 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdab4003610 con 0x7fdac40a6ba0 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient(hunting): dump: 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2379443756 0 0) 0x7fdac413bea0 con 0x7fdac413f5c0 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.1:6793/0 conn(0x7fdac40a6ba0 legacy=0x7fdac41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.1:6792/0 conn(0x7fdac40a67c0 legacy=0x7fdac41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 monclient: found mon.f 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac41cf020 con 0x7fdac413f5c0 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient: _finish_auth 0 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:54:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdabaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdacc7ba640 5 monclient: authenticate success, global_id 28279 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdacc7ba640 10 monclient: _renew_subs 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdacc7ba640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.677+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdac413a6a0 con 0x7fdac413f5c0 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdacc7ba640 10 client.28279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdacc7ba640 10 monclient: _renew_subs 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdacc7ba640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdac413e390 con 0x7fdac413f5c0 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdacc7ba640 1 librados: init done 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdabc001e40 con 0x7fdac413f5c0 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdabc004610 con 0x7fdac413f5c0 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdac8d53640 4 set_mon_vals no callback set 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 10 monclient: dump: 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.678+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fdabc011e00 con 0x7fdac413f5c0 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(193..193 src has 1..193) ==== 5860+0+0 (unknown 1784841602 0 0) 0x7fdabc04e600 con 0x7fdac413f5c0 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdabaffd640 10 client.28279.objecter ms_dispatch 0x7fdac4000b90 osd_map(193..193 src has 1..193) 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdabaffd640 3 client.28279.objecter handle_osd_map got epochs [193,193] > 0 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdabaffd640 3 client.28279.objecter handle_osd_map decoding full epoch 193 2024-08-19T09:54:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdacc7ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdacc7ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdacc7ba640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaac005080 con 0x7fdac413f5c0 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdabaffd640 20 client.28279.objecter dump_active .. 0 homeless 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.679+0000 7fdaca556640 10 monclient: get_auth_request con 0x7fdaa8041520 auth_method 0 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.681+0000 7fdabaffd640 10 client.28279.objecter ms_handle_connect 0x7fdaa8041520 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.684+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fdac40a9cc0 con 0x7fdac413f5c0 2024-08-19T09:54:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.684+0000 7fdabaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:50.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.684+0000 7fdabaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.938+0000 7fdacc7ba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:54:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.938+0000 7fdacc7ba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:54:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.938+0000 7fdacc7ba640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:54:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.938+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdaac005370 con 0x7fdac413f5c0 2024-08-19T09:54:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.939+0000 7fdabaffd640 1 -- 172.21.15.1:0/3250727087 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 3532190111) 0x7fdabc017860 con 0x7fdac413f5c0 2024-08-19T09:54:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.939+0000 7fdabaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:54:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.939+0000 7fdabaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:50.940 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:54:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 10 librados: watch_flush enter 2024-08-19T09:54:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 10 librados: watch_flush exit 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 20 client.28279.objecter shutdown clearing up homeless session... 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 10 client.28279.objecter successfully canceled tick 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.82:6812/2549104415 conn(0x7fdaa8041520 legacy=0x7fdaa80439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 10 monclient: shutdown 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 >> v1:172.21.15.82:6789/0 conn(0x7fdac413f5c0 legacy=0x7fdac41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 shutdown_connections 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 >> 172.21.15.1:0/3250727087 conn(0x7fdac40ac4b0 msgr2=0x7fdac4141e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 shutdown_connections 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 -- 172.21.15.1:0/3250727087 wait complete. 2024-08-19T09:54:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:50.941+0000 7fdacc7ba640 1 librados: shutdown 2024-08-19T09:54:50.952 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:54:50.952 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:54:50.953 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.051+0000 7f5a85fa0640 10 monclient: get_monmap_and_config 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.051+0000 7f5a85fa0640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.051+0000 7f5a85fa0640 10 monclient: monmap: 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:51.052740+0000 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:51.052740+0000 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.053 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.054 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.052+0000 7f5a85fa0640 1 Processor -- start 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.052+0000 7f5a85fa0640 1 -- start start 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.052+0000 7f5a85fa0640 10 monclient: init 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.052+0000 7f5a85fa0640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 10 monclient: _add_conns ranks=[8,4,7,1,2,6,0,3,5] 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 10 monclient(hunting): picked mon.noname-e con 0x7f5a800a8ce0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 10 monclient(hunting): picked mon.noname-c con 0x7f5a800a5720 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 10 monclient(hunting): picked mon.noname-i con 0x7f5a800a5b00 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a80136660 con 0x7f5a800a5720 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a8009ded0 con 0x7f5a800a5b00 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a8009e1b0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a85fa0640 10 monclient(hunting): authenticate will time out at 947.129090s 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a84f9e640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f5a800a8ce0 0x7f5a80135f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40554/0 (socket says 172.21.15.1:40554) 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a84f9e640 1 -- 172.21.15.1:0/3331180361 learned_addr learned my addr 172.21.15.1:0/3331180361 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.053+0000 7f5a7ffff640 1 --1- 172.21.15.1:0/3331180361 >> v1:172.21.15.1:6791/0 conn(0x7f5a800a5720 0x7f5a8013e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59350/0 (socket says 172.21.15.1:59350) 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a84f9e640 1 -- 172.21.15.1:0/3331180361 >> v1:172.21.15.1:6793/0 conn(0x7f5a800a8ce0 legacy=0x7f5a80135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7ffff640 1 -- 172.21.15.1:0/3331180361 >> v1:172.21.15.1:6791/0 conn(0x7f5a800a5720 legacy=0x7f5a8013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a8579f640 1 -- 172.21.15.1:0/3331180361 >> v1:172.21.15.82:6792/0 conn(0x7f5a800a5b00 legacy=0x7f5a801418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593968245 0 0) 0x7f5a8009e1b0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): my global_id is 27359 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a68003520 con 0x7f5a800a8ce0 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2869764130 0 0) 0x7f5a80136660 con 0x7f5a800a5720 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient(hunting): my global_id is 26869 2024-08-19T09:54:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a8009e1b0 con 0x7f5a800a5720 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2257733627 0 0) 0x7f5a68003520 con 0x7f5a800a8ce0 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 >> v1:172.21.15.82:6792/0 conn(0x7f5a800a5b00 legacy=0x7f5a801418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 >> v1:172.21.15.1:6791/0 conn(0x7f5a800a5720 legacy=0x7f5a8013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 monclient: found mon.noname-e 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a800a5f50 con 0x7f5a800a8ce0 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a78001e60 con 0x7f5a800a8ce0 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a85fa0640 5 monclient: authenticate success, global_id 27359 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: dump: 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a85fa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a78004c20 con 0x7f5a800a8ce0 2024-08-19T09:54:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 1 -- 172.21.15.1:0/3331180361 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a78005190 con 0x7f5a800a8ce0 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a85fa0640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a85fa0640 4 set_mon_vals no callback set 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.055+0000 7f5a7f7fe640 10 monclient: dump: 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.075 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 10 monclient: shutdown 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 1 -- 172.21.15.1:0/3331180361 >> v1:172.21.15.1:6793/0 conn(0x7f5a800a8ce0 legacy=0x7f5a80135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 1 -- 172.21.15.1:0/3331180361 shutdown_connections 2024-08-19T09:54:51.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 1 -- 172.21.15.1:0/3331180361 >> 172.21.15.1:0/3331180361 conn(0x7f5a800ac100 msgr2=0x7f5a800a85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 1 -- 172.21.15.1:0/3331180361 shutdown_connections 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 1 -- 172.21.15.1:0/3331180361 wait complete. 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.056+0000 7f5a85fa0640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient: monmap: 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.077 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 librados: starting msgr at 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 librados: starting objecter 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 Processor -- start 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 -- start start 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 librados: setting wanted keys 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 librados: calling monclient init 2024-08-19T09:54:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient: init 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient: _add_conns ranks=[3,1,5,6,0,7,2,8,4] 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient(hunting): picked mon.noname-g con 0x7f5a800a5720 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient(hunting): picked mon.noname-f con 0x7f5a800a5b00 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient(hunting): picked mon.noname-h con 0x7f5a800a8ce0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a3ae0 con 0x7f5a800a5b00 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a2c70 con 0x7f5a800a5720 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a0bb0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.057+0000 7f5a85fa0640 10 monclient(hunting): authenticate will time out at 947.133984s 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.058+0000 7f5a84f9e640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f5a800a5720 0x7f5a8009fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:52514/0 (socket says 172.21.15.1:52514) 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.058+0000 7f5a8579f640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f5a800a8ce0 0x7f5a801d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:41152/0 (socket says 172.21.15.1:41152) 2024-08-19T09:54:51.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.058+0000 7f5a84f9e640 1 -- 172.21.15.1:0/4221638726 learned_addr learned my addr 172.21.15.1:0/4221638726 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a84f9e640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6790/0 conn(0x7f5a800a5720 legacy=0x7f5a8009fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 client.?.objecter ms_handle_connect 0x7f5a800a5720 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 client.?.objecter ms_handle_connect 0x7f5a800a5b00 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7ffff640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6789/0 conn(0x7f5a800a5b00 legacy=0x7f5a800a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 client.?.objecter ms_handle_connect 0x7f5a800a8ce0 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a8579f640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6791/0 conn(0x7f5a800a8ce0 legacy=0x7f5a801d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883664082 0 0) 0x7f5a800a3ae0 con 0x7f5a800a5b00 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 28288 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a6c004ee0 con 0x7f5a800a5b00 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 661418833 0 0) 0x7f5a800a2c70 con 0x7f5a800a5720 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 134580 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a800a3ae0 con 0x7f5a800a5720 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315776532 0 0) 0x7f5a800a0bb0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 26582 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.060+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a800a2c70 con 0x7f5a800a8ce0 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 724556877 0 0) 0x7f5a800a2c70 con 0x7f5a800a8ce0 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a800a0bb0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a70001d10 con 0x7f5a800a8ce0 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): dump: 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.082 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.083 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.083 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.083 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.083 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.084 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1923051257 0 0) 0x7f5a800a3ae0 con 0x7f5a800a5720 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a800a2c70 con 0x7f5a800a5720 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2250908057 0 0) 0x7f5a6c004ee0 con 0x7f5a800a5b00 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a800a3ae0 con 0x7f5a800a5b00 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a780020b0 con 0x7f5a800a5720 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): dump: 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a740047e0 con 0x7f5a800a5b00 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient(hunting): dump: 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1048561260 0 0) 0x7f5a800a0bb0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6790/0 conn(0x7f5a800a5720 legacy=0x7f5a8009fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6789/0 conn(0x7f5a800a5b00 legacy=0x7f5a800a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 monclient: found mon.h 2024-08-19T09:54:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a800a2580 con 0x7f5a800a8ce0 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 5 monclient: authenticate success, global_id 26582 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 10 monclient: _renew_subs 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a800a5340 con 0x7f5a800a8ce0 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 10 client.26582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 10 monclient: _renew_subs 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:54:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a801d7b50 con 0x7f5a800a8ce0 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a85fa0640 1 librados: init done 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a700038c0 con 0x7f5a800a8ce0 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a70003e70 con 0x7f5a800a8ce0 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7effd640 4 set_mon_vals no callback set 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.061+0000 7f5a7d7fa640 10 monclient: dump: 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a85fa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a600065c0 con 0x7f5a60002860 2024-08-19T09:54:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5a70004010 con 0x7f5a800a8ce0 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(193..193 src has 1..193) ==== 5860+0+0 (unknown 1784841602 0 0) 0x7f5a7004e280 con 0x7f5a800a8ce0 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7d7fa640 10 client.26582.objecter ms_dispatch 0x7f5a80000b90 osd_map(193..193 src has 1..193) 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7d7fa640 3 client.26582.objecter handle_osd_map got epochs [193,193] > 0 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7d7fa640 3 client.26582.objecter handle_osd_map decoding full epoch 193 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7d7fa640 20 client.26582.objecter dump_active .. 0 homeless 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.062+0000 7f5a7ffff640 10 monclient: get_auth_request con 0x7f5a6c041cd0 auth_method 0 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 client.26582.objecter ms_handle_connect 0x7f5a60002860 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 client.26582.objecter resend_mon_ops 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a84f9e640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.1:6789/0 conn(0x7f5a60002860 legacy=0x7f5a60004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242166404 0 0) 0x7f5a600065c0 con 0x7f5a60002860 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 134559 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a800a0bb0 con 0x7f5a60002860 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3942584221 0 0) 0x7f5a800a0bb0 con 0x7f5a60002860 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a600065c0 con 0x7f5a60002860 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a78004ba0 con 0x7f5a60002860 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 10 monclient: dump: 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3192076654 0 0) 0x7f5a600065c0 con 0x7f5a60002860 2024-08-19T09:54:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.064+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a6000a160 con 0x7f5a60002860 2024-08-19T09:54:51.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.065+0000 7f5a7d7fa640 10 client.26582.objecter ms_handle_connect 0x7f5a6c041cd0 2024-08-19T09:54:51.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.065+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5a78005450 con 0x7f5a60002860 2024-08-19T09:54:51.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.065+0000 7f5a7d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.065+0000 7f5a7d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:51.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.065+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.1:6789/0 conn(0x7f5a60002860 legacy=0x7f5a60004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.065+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.1:6789/0 conn(0x7f5a60002860 legacy=0x7f5a60004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.071+0000 7f5a85fa0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.071+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a60009e70 con 0x7f5a60009130 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 client.26582.objecter ms_handle_connect 0x7f5a60009130 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 client.26582.objecter resend_mon_ops 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a84f9e640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.1:6789/0 conn(0x7f5a60009130 legacy=0x7f5a600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3903372428 0 0) 0x7f5a60009e70 con 0x7f5a60009130 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 134568 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a600065c0 con 0x7f5a60009130 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 38954724 0 0) 0x7f5a600065c0 con 0x7f5a60009130 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a60009e70 con 0x7f5a60009130 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a78005c70 con 0x7f5a60009130 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.073+0000 7f5a7d7fa640 10 monclient: dump: 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 61872619 0 0) 0x7f5a60009e70 con 0x7f5a60009130 2024-08-19T09:54:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a6000c410 con 0x7f5a60009130 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7686 (unknown 0 0 4084241807) 0x7f5a78005f80 con 0x7f5a60009130 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.1:6789/0 conn(0x7f5a60009130 legacy=0x7f5a600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a7d7fa640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.1:6789/0 conn(0x7f5a60009130 legacy=0x7f5a600095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:54:51.103 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 148137, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.104 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.105 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.106 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.107 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.108 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:51.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.113 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.114 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.119 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:51.120 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.121 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.122 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.123 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:51.124 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 10 librados: watch_flush enter 2024-08-19T09:54:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 10 librados: watch_flush exit 2024-08-19T09:54:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 20 client.26582.objecter shutdown clearing up homeless session... 2024-08-19T09:54:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 10 client.26582.objecter successfully canceled tick 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5a6c041cd0 legacy=0x7f5a6c044190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 10 monclient: shutdown 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 >> v1:172.21.15.82:6791/0 conn(0x7f5a800a8ce0 legacy=0x7f5a801d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a8579f640 1 -- 172.21.15.1:0/4221638726 reap_dead start 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a8579f640 1 -- 172.21.15.1:0/4221638726 reap_dead start 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 shutdown_connections 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 >> 172.21.15.1:0/4221638726 conn(0x7f5a800ac100 msgr2=0x7f5a800aaf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 shutdown_connections 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 -- 172.21.15.1:0/4221638726 wait complete. 2024-08-19T09:54:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.074+0000 7f5a85fa0640 1 librados: shutdown 2024-08-19T09:54:51.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:51.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:54:51.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.226+0000 7f11404c5640 10 monclient: get_monmap_and_config 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.226+0000 7f11404c5640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.226+0000 7f11404c5640 10 monclient: monmap: 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:51.227100+0000 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:51.227100+0000 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.228 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.226+0000 7f11404c5640 1 Processor -- start 2024-08-19T09:54:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.226+0000 7f11404c5640 1 -- start start 2024-08-19T09:54:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.226+0000 7f11404c5640 10 monclient: init 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient: _add_conns ranks=[3,0,5,6,2,4,8,1,7] 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient(hunting): picked mon.noname-g con 0x7f11380a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient(hunting): picked mon.noname-a con 0x7f11380694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient(hunting): picked mon.noname-h con 0x7f11380a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11380a3c70 con 0x7f11380694b0 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11380a33d0 con 0x7f11380a6830 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11380a36b0 con 0x7f11380a46a0 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f11404c5640 10 monclient(hunting): authenticate will time out at 947.303453s 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f113da60640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f11380694b0 0x7f11380a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37148/0 (socket says 172.21.15.1:37148) 2024-08-19T09:54:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.227+0000 7f113da60640 1 -- 172.21.15.1:0/3565000744 learned_addr learned my addr 172.21.15.1:0/3565000744 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113da60640 1 -- 172.21.15.1:0/3565000744 >> v1:172.21.15.1:6789/0 conn(0x7f11380694b0 legacy=0x7f11380a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380305307 0 0) 0x7f11380a3c70 con 0x7f11380694b0 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113d25f640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113d25f640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113d25f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113d25f640 10 monclient(hunting): my global_id is 134577 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.229+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1120003520 con 0x7f11380694b0 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113e261640 1 -- 172.21.15.1:0/3565000744 >> v1:172.21.15.82:6790/0 conn(0x7f11380a6830 legacy=0x7f11380a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113ea62640 1 -- 172.21.15.1:0/3565000744 >> v1:172.21.15.82:6791/0 conn(0x7f11380a46a0 legacy=0x7f1138141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3644450917 0 0) 0x7f1120003520 con 0x7f11380694b0 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 >> v1:172.21.15.82:6791/0 conn(0x7f11380a46a0 legacy=0x7f1138141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 >> v1:172.21.15.82:6790/0 conn(0x7f11380a6830 legacy=0x7f11380a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 monclient: found mon.noname-a 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113806dbb0 con 0x7f11380694b0 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 5 monclient: authenticate success, global_id 134577 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f112c001ae0 con 0x7f11380694b0 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: dump: 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f112c0048a0 con 0x7f11380694b0 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 1 -- 172.21.15.1:0/3565000744 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f112c004dd0 con 0x7f11380694b0 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 4 set_mon_vals no callback set 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f113d25f640 10 monclient: dump: 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.238 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 10 monclient: shutdown 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 1 -- 172.21.15.1:0/3565000744 >> v1:172.21.15.1:6789/0 conn(0x7f11380694b0 legacy=0x7f11380a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 1 -- 172.21.15.1:0/3565000744 shutdown_connections 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 1 -- 172.21.15.1:0/3565000744 >> 172.21.15.1:0/3565000744 conn(0x7f11380a9020 msgr2=0x7f1138068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 1 -- 172.21.15.1:0/3565000744 shutdown_connections 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.230+0000 7f11404c5640 1 -- 172.21.15.1:0/3565000744 wait complete. 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 10 monclient: monmap: 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 1 librados: starting msgr at 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 1 librados: starting objecter 2024-08-19T09:54:51.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 1 Processor -- start 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 1 -- start start 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 1 librados: setting wanted keys 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 1 librados: calling monclient init 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.231+0000 7f11404c5640 10 monclient: init 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient: _add_conns ranks=[6,8,5,2,7,0,1,3,4] 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient(hunting): picked mon.noname-d con 0x7f11380694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient(hunting): picked mon.noname-e con 0x7f11380a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient(hunting): picked mon.noname-h con 0x7f11380a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1138139660 con 0x7f11380a6830 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11380aa050 con 0x7f11380694b0 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f113813ae30 con 0x7f11380a46a0 2024-08-19T09:54:51.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f11404c5640 10 monclient(hunting): authenticate will time out at 947.308420s 2024-08-19T09:54:51.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f113e261640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f11380694b0 0x7f1138136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52774/0 (socket says 172.21.15.1:52774) 2024-08-19T09:54:51.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.232+0000 7f113e261640 1 -- 172.21.15.1:0/3235003542 learned_addr learned my addr 172.21.15.1:0/3235003542 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f113e261640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6792/0 conn(0x7f11380694b0 legacy=0x7f1138136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f113da60640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6793/0 conn(0x7f11380a46a0 legacy=0x7f1138137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f1136ffd640 10 client.?.objecter ms_handle_connect 0x7f11380694b0 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f1136ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f1136ffd640 10 client.?.objecter ms_handle_connect 0x7f11380a46a0 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f1136ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f113ea62640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.82:6791/0 conn(0x7f11380a6830 legacy=0x7f11381d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f1136ffd640 10 client.?.objecter ms_handle_connect 0x7f11380a6830 2024-08-19T09:54:51.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.234+0000 7f1136ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956052963 0 0) 0x7f113813ae30 con 0x7f11380a46a0 2024-08-19T09:54:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): my global_id is 27368 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1124004eb0 con 0x7f11380a46a0 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836592442 0 0) 0x7f11380a33d0 con 0x7f11380694b0 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): my global_id is 26943 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f113813ae30 con 0x7f11380694b0 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 291788136 0 0) 0x7f1138139660 con 0x7f11380a6830 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 10 monclient(hunting): my global_id is 26591 2024-08-19T09:54:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.355+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11380a33d0 con 0x7f11380a6830 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.360+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 507745377 0 0) 0x7f113813ae30 con 0x7f11380694b0 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.360+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1138139660 con 0x7f11380694b0 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1128003990 con 0x7f11380694b0 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): dump: 2024-08-19T09:54:51.363 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.364 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.364 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.364 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.364 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.364 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.364 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2975108394 0 0) 0x7f1124004eb0 con 0x7f11380a46a0 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f113813ae30 con 0x7f11380a46a0 2024-08-19T09:54:51.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f112c001d30 con 0x7f11380a46a0 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient(hunting): dump: 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 663786783 0 0) 0x7f113813ae30 con 0x7f11380a46a0 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6792/0 conn(0x7f11380694b0 legacy=0x7f1138136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.82:6791/0 conn(0x7f11380a6830 legacy=0x7f11381d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 monclient: found mon.e 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1138138730 con 0x7f11380a46a0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f1136ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 5 monclient: authenticate success, global_id 27368 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 10 monclient: _renew_subs 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f113813ab50 con 0x7f11380a46a0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 10 client.27368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 10 monclient: _renew_subs 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.361+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f1138137d60 con 0x7f11380a46a0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.362+0000 7f11404c5640 1 librados: init done 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.362+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f112c001d30 con 0x7f11380a46a0 2024-08-19T09:54:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.362+0000 7f1136ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.362+0000 7f113ca5e640 4 set_mon_vals no callback set 2024-08-19T09:54:51.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.363+0000 7f11404c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.363+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11180065c0 con 0x7f1118002860 2024-08-19T09:54:51.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.365+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f112c0048a0 con 0x7f11380a46a0 2024-08-19T09:54:51.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.365+0000 7f1136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.365+0000 7f1136ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.365+0000 7f1136ffd640 10 monclient: dump: 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.373 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.366+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f112c011300 con 0x7f11380a46a0 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.366+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7f112c04dda0 con 0x7f11380a46a0 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.366+0000 7f1136ffd640 10 client.27368.objecter ms_dispatch 0x7f1138000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.366+0000 7f1136ffd640 3 client.27368.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.366+0000 7f1136ffd640 3 client.27368.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.366+0000 7f1136ffd640 20 client.27368.objecter dump_active .. 0 homeless 2024-08-19T09:54:51.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.367+0000 7f113ea62640 10 monclient: get_auth_request con 0x7f1124041aa0 auth_method 0 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.367+0000 7f113e261640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6790/0 conn(0x7f1118002860 legacy=0x7f1118004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 10 client.27368.objecter ms_handle_connect 0x7f1118002860 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 10 client.27368.objecter resend_mon_ops 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654241233 0 0) 0x7f11180065c0 con 0x7f1118002860 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 10 monclient(hunting): my global_id is 144101 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.368+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f113813ae30 con 0x7f1118002860 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.369+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 794203041 0 0) 0x7f113813ae30 con 0x7f1118002860 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.369+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11180065c0 con 0x7f1118002860 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.369+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f11280044c0 con 0x7f1118002860 2024-08-19T09:54:51.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.369+0000 7f1136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.369+0000 7f1136ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.369+0000 7f1136ffd640 10 monclient: dump: 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.376 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1452532068 0 0) 0x7f11180065c0 con 0x7f1118002860 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f111800a160 con 0x7f1118002860 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1128004770 con 0x7f1118002860 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6790/0 conn(0x7f1118002860 legacy=0x7f1118004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.370+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6790/0 conn(0x7f1118002860 legacy=0x7f1118004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.371+0000 7f1136ffd640 10 client.27368.objecter ms_handle_connect 0x7f1124041aa0 2024-08-19T09:54:51.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.378+0000 7f11404c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:51.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.378+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1118009e70 con 0x7f1118009130 2024-08-19T09:54:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.380+0000 7f1136ffd640 10 client.27368.objecter ms_handle_connect 0x7f1118009130 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.380+0000 7f1136ffd640 10 client.27368.objecter resend_mon_ops 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.380+0000 7f113e261640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6790/0 conn(0x7f1118009130 legacy=0x7f11180095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423515393 0 0) 0x7f1118009e70 con 0x7f1118009130 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient(hunting): my global_id is 144110 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11180065c0 con 0x7f1118009130 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4000362466 0 0) 0x7f11180065c0 con 0x7f1118009130 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1118009e70 con 0x7f1118009130 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f11280060b0 con 0x7f1118009130 2024-08-19T09:54:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient: dump: 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.385 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 478749603 0 0) 0x7f1118009e70 con 0x7f1118009130 2024-08-19T09:54:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f111800c3f0 con 0x7f1118009130 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 1288733750) 0x7f1128008370 con 0x7f1118009130 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6790/0 conn(0x7f1118009130 legacy=0x7f11180095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.381+0000 7f1136ffd640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6790/0 conn(0x7f1118009130 legacy=0x7f11180095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:51.386 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15763, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:51.387 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.388 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.389 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:51.390 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.391 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:51.392 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:51.396 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.397 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.401 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:51.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.403 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.405 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.406 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.406 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.406 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:51.406 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 10 librados: watch_flush enter 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 10 librados: watch_flush exit 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 20 client.27368.objecter shutdown clearing up homeless session... 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 10 client.27368.objecter successfully canceled tick 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1124041aa0 legacy=0x7f1124043f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 10 monclient: shutdown 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 >> v1:172.21.15.1:6793/0 conn(0x7f11380a46a0 legacy=0x7f1138137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f113ea62640 1 -- 172.21.15.1:0/3235003542 reap_dead start 2024-08-19T09:54:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f113ea62640 1 -- 172.21.15.1:0/3235003542 reap_dead start 2024-08-19T09:54:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 shutdown_connections 2024-08-19T09:54:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 >> 172.21.15.1:0/3235003542 conn(0x7f11380a9020 msgr2=0x7f11380a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 shutdown_connections 2024-08-19T09:54:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 -- 172.21.15.1:0/3235003542 wait complete. 2024-08-19T09:54:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.382+0000 7f11404c5640 1 librados: shutdown 2024-08-19T09:54:51.409 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:51.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:54:51.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:54:51.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.507+0000 7f7592d86640 10 monclient: get_monmap_and_config 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.507+0000 7f7592d86640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.507+0000 7f7592d86640 10 monclient: monmap: 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:51.508889+0000 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:51.508889+0000 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.510 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.508+0000 7f7592d86640 1 Processor -- start 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.508+0000 7f7592d86640 1 -- start start 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.508+0000 7f7592d86640 10 monclient: init 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient: _add_conns ranks=[7,8,3,1,6,0,4,2,5] 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient(hunting): picked mon.noname-i con 0x7f758c0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient(hunting): picked mon.noname-e con 0x7f758c0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient(hunting): picked mon.noname-g con 0x7f758c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758c0a3b40 con 0x7f758c13f5c0 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758c0a8130 con 0x7f758c0a67c0 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758c06ede0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f7592d86640 10 monclient(hunting): authenticate will time out at 947.585360s 2024-08-19T09:54:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f758bfff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f758c0a6ba0 0x7f758c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40566/0 (socket says 172.21.15.1:40566) 2024-08-19T09:54:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.509+0000 7f758bfff640 1 -- 172.21.15.1:0/275237678 learned_addr learned my addr 172.21.15.1:0/275237678 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758bfff640 1 -- 172.21.15.1:0/275237678 >> v1:172.21.15.1:6793/0 conn(0x7f758c0a6ba0 legacy=0x7f758c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f7591323640 1 -- 172.21.15.1:0/275237678 >> v1:172.21.15.82:6790/0 conn(0x7f758c13f5c0 legacy=0x7f758c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434557026 0 0) 0x7f758c06ede0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): my global_id is 27377 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f7590b22640 1 -- 172.21.15.1:0/275237678 >> v1:172.21.15.82:6792/0 conn(0x7f758c0a67c0 legacy=0x7f758c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7578003520 con 0x7f758c0a6ba0 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728429811 0 0) 0x7f758c0a8130 con 0x7f758c0a67c0 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.511+0000 7f758b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient(hunting): my global_id is 144115 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758c06ede0 con 0x7f758c0a67c0 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2537271506 0 0) 0x7f758c0a3b40 con 0x7f758c13f5c0 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient(hunting): my global_id is 134589 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758c0a8130 con 0x7f758c13f5c0 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1008430900 0 0) 0x7f7578003520 con 0x7f758c0a6ba0 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 >> v1:172.21.15.82:6792/0 conn(0x7f758c0a67c0 legacy=0x7f758c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 >> v1:172.21.15.82:6790/0 conn(0x7f758c13f5c0 legacy=0x7f758c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 monclient: found mon.noname-e 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c0a92a0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7574001ad0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: dump: 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.520 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 5 monclient: authenticate success, global_id 27377 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75740048a0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 1 -- 172.21.15.1:0/275237678 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7574004e10 con 0x7f758c0a6ba0 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f758b7fe640 10 monclient: dump: 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 4 set_mon_vals no callback set 2024-08-19T09:54:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 10 monclient: shutdown 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 1 -- 172.21.15.1:0/275237678 >> v1:172.21.15.1:6793/0 conn(0x7f758c0a6ba0 legacy=0x7f758c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 1 -- 172.21.15.1:0/275237678 shutdown_connections 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 1 -- 172.21.15.1:0/275237678 >> 172.21.15.1:0/275237678 conn(0x7f758c0ac4b0 msgr2=0x7f758c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 1 -- 172.21.15.1:0/275237678 shutdown_connections 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.512+0000 7f7592d86640 1 -- 172.21.15.1:0/275237678 wait complete. 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 10 monclient: monmap: 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 1 librados: starting msgr at 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 1 librados: starting objecter 2024-08-19T09:54:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 1 Processor -- start 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 1 -- start start 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 1 librados: setting wanted keys 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 1 librados: calling monclient init 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.513+0000 7f7592d86640 10 monclient: init 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient: _add_conns ranks=[0,3,7,1,8,2,5,4,6] 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient(hunting): picked mon.noname-a con 0x7f758c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient(hunting): picked mon.noname-g con 0x7f758c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient(hunting): picked mon.noname-i con 0x7f758c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758c13ba10 con 0x7f758c0a67c0 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758c13bcf0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758c13ae80 con 0x7f758c13f5c0 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7592d86640 10 monclient(hunting): authenticate will time out at 947.590415s 2024-08-19T09:54:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7590b22640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f758c0a67c0 0x7f758c13c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37154/0 (socket says 172.21.15.1:37154) 2024-08-19T09:54:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.514+0000 7f7590b22640 1 -- 172.21.15.1:0/3312565728 learned_addr learned my addr 172.21.15.1:0/3312565728 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f7590b22640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6789/0 conn(0x7f758c0a67c0 legacy=0x7f758c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 client.?.objecter ms_handle_connect 0x7f758c0a67c0 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 client.?.objecter ms_handle_connect 0x7f758c13f5c0 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 client.?.objecter ms_handle_connect 0x7f758c0a6ba0 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f7591323640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.82:6792/0 conn(0x7f758c13f5c0 legacy=0x7f758c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f758bfff640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.82:6790/0 conn(0x7f758c0a6ba0 legacy=0x7f758c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1002668201 0 0) 0x7f758c13ba10 con 0x7f758c0a67c0 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): my global_id is 134586 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f756c004ee0 con 0x7f758c0a67c0 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1305162246 0 0) 0x7f758c13ae80 con 0x7f758c13f5c0 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): my global_id is 144124 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758c13ba10 con 0x7f758c13f5c0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336889944 0 0) 0x7f758c13bcf0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 10 monclient(hunting): my global_id is 134598 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.516+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758c13ae80 con 0x7f758c0a6ba0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4021601234 0 0) 0x7f758c13ba10 con 0x7f758c13f5c0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f758c13bcf0 con 0x7f758c13f5c0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7584001cb0 con 0x7f758c13f5c0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:54:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): dump: 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.536 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 303506150 0 0) 0x7f756c004ee0 con 0x7f758c0a67c0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f758c13ba10 con 0x7f758c0a67c0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3674891375 0 0) 0x7f758c13ae80 con 0x7f758c0a6ba0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f756c004ee0 con 0x7f758c0a6ba0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f757c003990 con 0x7f758c0a67c0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): dump: 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.537 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.538 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7574004d90 con 0x7f758c0a6ba0 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient(hunting): dump: 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.539 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2767255608 0 0) 0x7f758c13bcf0 con 0x7f758c13f5c0 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.82:6790/0 conn(0x7f758c0a6ba0 legacy=0x7f758c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6789/0 conn(0x7f758c0a67c0 legacy=0x7f758c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 monclient: found mon.i 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c13e560 con 0x7f758c13f5c0 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 5 monclient: authenticate success, global_id 144124 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 10 monclient: _renew_subs 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f758c0a28b0 con 0x7f758c13f5c0 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 10 client.144124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 10 monclient: _renew_subs 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f758c13d1f0 con 0x7f758c13f5c0 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f7592d86640 1 librados: init done 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75840039f0 con 0x7f758c13f5c0 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7584003ee0 con 0x7f758c13f5c0 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f758affd640 4 set_mon_vals no callback set 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.517+0000 7f75897fa640 10 monclient: dump: 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.542 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.543 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f7592d86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f75700065c0 con 0x7f7570002860 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7584004140 con 0x7f758c13f5c0 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7f758404e220 con 0x7f758c13f5c0 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f75897fa640 10 client.144124.objecter ms_dispatch 0x7f758c000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f75897fa640 3 client.144124.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f75897fa640 3 client.144124.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f75897fa640 20 client.144124.objecter dump_active .. 0 homeless 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.518+0000 7f758bfff640 10 monclient: get_auth_request con 0x7f756c041a30 auth_method 0 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f7590b22640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6791/0 conn(0x7f7570002860 legacy=0x7f7570004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 client.144124.objecter ms_handle_connect 0x7f7570002860 2024-08-19T09:54:51.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 client.144124.objecter resend_mon_ops 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257251430 0 0) 0x7f75700065c0 con 0x7f7570002860 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient(hunting): my global_id is 26878 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758c13bcf0 con 0x7f7570002860 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2693792619 0 0) 0x7f758c13bcf0 con 0x7f7570002860 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75700065c0 con 0x7f7570002860 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f757c0041d0 con 0x7f7570002860 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 10 monclient: dump: 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2869603139 0 0) 0x7f75700065c0 con 0x7f7570002860 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.520+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f757000a160 con 0x7f7570002860 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.521+0000 7f75897fa640 10 client.144124.objecter ms_handle_connect 0x7f756c041a30 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.521+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f757c005b40 con 0x7f7570002860 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.521+0000 7f75897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.521+0000 7f75897fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.521+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6791/0 conn(0x7f7570002860 legacy=0x7f7570004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.521+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6791/0 conn(0x7f7570002860 legacy=0x7f7570004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.527+0000 7f7592d86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.527+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7570009e50 con 0x7f7570009110 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f7590b22640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6791/0 conn(0x7f7570009110 legacy=0x7f75700095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 10 client.144124.objecter ms_handle_connect 0x7f7570009110 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 10 client.144124.objecter resend_mon_ops 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860471067 0 0) 0x7f7570009e50 con 0x7f7570009110 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 10 monclient(hunting): my global_id is 26887 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.529+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75700065c0 con 0x7f7570009110 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 797979645 0 0) 0x7f75700065c0 con 0x7f7570009110 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7570009e50 con 0x7f7570009110 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f757c0046f0 con 0x7f7570009110 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 10 monclient: dump: 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.549 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3729044099 0 0) 0x7f7570009e50 con 0x7f7570009110 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f757000c3a0 con 0x7f7570009110 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1556028084) 0x7f757c008110 con 0x7f7570009110 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6791/0 conn(0x7f7570009110 legacy=0x7f75700095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f75897fa640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.1:6791/0 conn(0x7f7570009110 legacy=0x7f75700095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 297802, 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:51.551 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:51.552 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:51.553 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:51.555 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.559 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:51.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.561 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.564 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.565 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.569 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.570 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.571 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.572 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.572 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:51.572 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 10 librados: watch_flush enter 2024-08-19T09:54:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 10 librados: watch_flush exit 2024-08-19T09:54:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 20 client.144124.objecter shutdown clearing up homeless session... 2024-08-19T09:54:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 10 client.144124.objecter successfully canceled tick 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.82:6812/2549104415 conn(0x7f756c041a30 legacy=0x7f756c043ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 10 monclient: shutdown 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 >> v1:172.21.15.82:6792/0 conn(0x7f758c13f5c0 legacy=0x7f758c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7591323640 1 -- 172.21.15.1:0/3312565728 reap_dead start 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7591323640 1 -- 172.21.15.1:0/3312565728 reap_dead start 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 shutdown_connections 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.530+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 >> 172.21.15.1:0/3312565728 conn(0x7f758c0ac4b0 msgr2=0x7f758c13ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.531+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 shutdown_connections 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.531+0000 7f7592d86640 1 -- 172.21.15.1:0/3312565728 wait complete. 2024-08-19T09:54:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.531+0000 7f7592d86640 1 librados: shutdown 2024-08-19T09:54:51.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:51.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:54:51.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.675+0000 7f490ba56640 10 monclient: get_monmap_and_config 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.675+0000 7f490ba56640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.675+0000 7f490ba56640 10 monclient: monmap: 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:51.676793+0000 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:51.676793+0000 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.676+0000 7f490ba56640 1 Processor -- start 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.676+0000 7f490ba56640 1 -- start start 2024-08-19T09:54:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.676+0000 7f490ba56640 10 monclient: init 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient: _add_conns ranks=[5,1,8,6,3,2,7,0,4] 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient(hunting): picked mon.noname-h con 0x7f49040a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient(hunting): picked mon.noname-f con 0x7f49040694b0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient(hunting): picked mon.noname-e con 0x7f49040a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49040a3c70 con 0x7f49040694b0 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49040a33d0 con 0x7f49040a6830 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49040a36b0 con 0x7f49040a46a0 2024-08-19T09:54:51.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f490ba56640 10 monclient(hunting): authenticate will time out at 947.753173s 2024-08-19T09:54:51.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f4909ff3640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f49040a46a0 0x7f4904141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40568/0 (socket says 172.21.15.1:40568) 2024-08-19T09:54:51.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.677+0000 7f4909ff3640 1 -- 172.21.15.1:0/3123015248 learned_addr learned my addr 172.21.15.1:0/3123015248 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.679+0000 7f4909ff3640 1 -- 172.21.15.1:0/3123015248 >> v1:172.21.15.1:6793/0 conn(0x7f49040a46a0 legacy=0x7f4904141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.679+0000 7f49097f2640 1 -- 172.21.15.1:0/3123015248 >> v1:172.21.15.82:6791/0 conn(0x7f49040a6830 legacy=0x7f49040a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.679+0000 7f4908ff1640 1 -- 172.21.15.1:0/3123015248 >> v1:172.21.15.82:6789/0 conn(0x7f49040694b0 legacy=0x7f49040a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964825418 0 0) 0x7f49040a3c70 con 0x7f49040694b0 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient(hunting): my global_id is 28297 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48f0003520 con 0x7f49040694b0 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1309830398 0 0) 0x7f48f0003520 con 0x7f49040694b0 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 >> v1:172.21.15.1:6793/0 conn(0x7f49040a46a0 legacy=0x7f4904141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 >> v1:172.21.15.82:6791/0 conn(0x7f49040a6830 legacy=0x7f49040a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 monclient: found mon.noname-f 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490406dbb0 con 0x7f49040694b0 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f48f4001a60 con 0x7f49040694b0 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: dump: 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.685 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f490ba56640 5 monclient: authenticate success, global_id 28297 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f490ba56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48f4003610 con 0x7f49040694b0 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 1 -- 172.21.15.1:0/3123015248 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f48f4003b80 con 0x7f49040694b0 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f490ba56640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f490ba56640 4 set_mon_vals no callback set 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.681+0000 7f48fbfff640 10 monclient: dump: 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.687 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 10 monclient: shutdown 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 1 -- 172.21.15.1:0/3123015248 >> v1:172.21.15.82:6789/0 conn(0x7f49040694b0 legacy=0x7f49040a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 1 -- 172.21.15.1:0/3123015248 shutdown_connections 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 1 -- 172.21.15.1:0/3123015248 >> 172.21.15.1:0/3123015248 conn(0x7f49040a9020 msgr2=0x7f4904068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 1 -- 172.21.15.1:0/3123015248 shutdown_connections 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 1 -- 172.21.15.1:0/3123015248 wait complete. 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.682+0000 7f490ba56640 10 monclient: monmap: 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.689 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 librados: starting msgr at 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 librados: starting objecter 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 Processor -- start 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 -- start start 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 librados: setting wanted keys 2024-08-19T09:54:51.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 librados: calling monclient init 2024-08-19T09:54:51.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient: init 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient: _add_conns ranks=[2,8,1,4,6,7,0,3,5] 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient(hunting): picked mon.noname-b con 0x7f49040694b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient(hunting): picked mon.noname-e con 0x7f49040a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient(hunting): picked mon.noname-f con 0x7f49040a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f490413a8f0 con 0x7f49040a6830 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f490413abd0 con 0x7f49040694b0 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49041424b0 con 0x7f49040a46a0 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f490ba56640 10 monclient(hunting): authenticate will time out at 947.759789s 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f49097f2640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f49040694b0 0x7f4904136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34800/0 (socket says 172.21.15.1:34800) 2024-08-19T09:54:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f4908ff1640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f49040a46a0 0x7f49041371e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40582/0 (socket says 172.21.15.1:40582) 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.683+0000 7f49097f2640 1 -- 172.21.15.1:0/2901168309 learned_addr learned my addr 172.21.15.1:0/2901168309 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f4908ff1640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6793/0 conn(0x7f49040a46a0 legacy=0x7f49041371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f49097f2640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6790/0 conn(0x7f49040694b0 legacy=0x7f4904136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 client.?.objecter ms_handle_connect 0x7f49040a46a0 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 client.?.objecter ms_handle_connect 0x7f49040694b0 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f4909ff3640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.82:6789/0 conn(0x7f49040a6830 legacy=0x7f49041d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 client.?.objecter ms_handle_connect 0x7f49040a6830 2024-08-19T09:54:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24763919 0 0) 0x7f490413abd0 con 0x7f49040694b0 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): my global_id is 144119 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48e8004eb0 con 0x7f49040694b0 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791773083 0 0) 0x7f49041424b0 con 0x7f49040a46a0 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): my global_id is 27386 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f490413abd0 con 0x7f49040a46a0 2024-08-19T09:54:51.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980358782 0 0) 0x7f490413a8f0 con 0x7f49040a6830 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): my global_id is 28306 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49041424b0 con 0x7f49040a6830 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 944299475 0 0) 0x7f490413abd0 con 0x7f49040a46a0 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f490413a8f0 con 0x7f49040a46a0 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f48f4003b00 con 0x7f49040a46a0 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): dump: 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.700 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.701 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 66706639 0 0) 0x7f48e8004eb0 con 0x7f49040694b0 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f490413abd0 con 0x7f49040694b0 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1606957894 0 0) 0x7f49041424b0 con 0x7f49040a6830 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48e8004eb0 con 0x7f49040a6830 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4900003610 con 0x7f49040694b0 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): dump: 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f48fc002000 con 0x7f49040a6830 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient(hunting): dump: 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2892911751 0 0) 0x7f490413a8f0 con 0x7f49040a46a0 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6790/0 conn(0x7f49040694b0 legacy=0x7f4904136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.82:6789/0 conn(0x7f49040a6830 legacy=0x7f49041d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 monclient: found mon.e 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4904139db0 con 0x7f49040a46a0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 5 monclient: authenticate success, global_id 27386 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 10 monclient: _renew_subs 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49041384d0 con 0x7f49040a46a0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48f4001a00 con 0x7f49040a46a0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f48f4004290 con 0x7f49040a46a0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48f9ffb640 10 monclient: dump: 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 10 client.27386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 10 monclient: _renew_subs 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f48fb7fe640 4 set_mon_vals no callback set 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f490413a330 con 0x7f49040a46a0 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.686+0000 7f490ba56640 1 librados: init done 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f48f4011a80 con 0x7f49040a46a0 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f490ba56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4904051b00 con 0x7f4904139030 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7f48f404e150 con 0x7f49040a46a0 2024-08-19T09:54:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f48f9ffb640 10 client.27386.objecter ms_dispatch 0x7f4904000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f48f9ffb640 3 client.27386.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.687+0000 7f48f9ffb640 3 client.27386.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.688+0000 7f48f9ffb640 20 client.27386.objecter dump_active .. 0 homeless 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.688+0000 7f4909ff3640 10 monclient: get_auth_request con 0x7f48e8041970 auth_method 0 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.689+0000 7f49097f2640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6792/0 conn(0x7f4904139030 legacy=0x7f49040a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.689+0000 7f48f9ffb640 10 client.27386.objecter ms_handle_connect 0x7f4904139030 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.689+0000 7f48f9ffb640 10 client.27386.objecter resend_mon_ops 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287185080 0 0) 0x7f4904051b00 con 0x7f4904139030 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient(hunting): my global_id is 26952 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f490413a8f0 con 0x7f4904139030 2024-08-19T09:54:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2467206253 0 0) 0x7f490413a8f0 con 0x7f4904139030 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4904051b00 con 0x7f4904139030 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4900003d60 con 0x7f4904139030 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient: dump: 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 client.27386.objecter ms_handle_connect 0x7f48e8041970 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 392418249 0 0) 0x7f4904051b00 con 0x7f4904139030 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4904142cf0 con 0x7f4904139030 2024-08-19T09:54:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4900004090 con 0x7f4904139030 2024-08-19T09:54:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6792/0 conn(0x7f4904139030 legacy=0x7f49040a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.690+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6792/0 conn(0x7f4904139030 legacy=0x7f49040a0c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.696+0000 7f490ba56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.696+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f490413a500 con 0x7f490402e1d0 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f49097f2640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6792/0 conn(0x7f490402e1d0 legacy=0x7f49040a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 client.27386.objecter ms_handle_connect 0x7f490402e1d0 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 client.27386.objecter resend_mon_ops 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524532749 0 0) 0x7f490413a500 con 0x7f490402e1d0 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient(hunting): my global_id is 26961 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4904051b00 con 0x7f490402e1d0 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 687002825 0 0) 0x7f4904051b00 con 0x7f490402e1d0 2024-08-19T09:54:51.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f490413a500 con 0x7f490402e1d0 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4900005b00 con 0x7f490402e1d0 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 10 monclient: dump: 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 953412946 0 0) 0x7f490413a500 con 0x7f490402e1d0 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.699+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4904139410 con 0x7f490402e1d0 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3320153945) 0x7f4900005eb0 con 0x7f490402e1d0 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f48f9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f48f9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6792/0 conn(0x7f490402e1d0 legacy=0x7f49040a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f48f9ffb640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6792/0 conn(0x7f490402e1d0 legacy=0x7f49040a1330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 297957, 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:51.720 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:51.721 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:51.722 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.723 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:51.724 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.725 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:51.726 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.729 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.733 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:51.735 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:51.740 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 10 librados: watch_flush enter 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 10 librados: watch_flush exit 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 20 client.27386.objecter shutdown clearing up homeless session... 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 10 client.27386.objecter successfully canceled tick 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.82:6812/2549104415 conn(0x7f48e8041970 legacy=0x7f48e8043e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 10 monclient: shutdown 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 >> v1:172.21.15.1:6793/0 conn(0x7f49040a46a0 legacy=0x7f49041371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f4909ff3640 1 -- 172.21.15.1:0/2901168309 reap_dead start 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f4909ff3640 1 -- 172.21.15.1:0/2901168309 reap_dead start 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 shutdown_connections 2024-08-19T09:54:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 >> 172.21.15.1:0/2901168309 conn(0x7f49040a9020 msgr2=0x7f49040a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 shutdown_connections 2024-08-19T09:54:51.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 -- 172.21.15.1:0/2901168309 wait complete. 2024-08-19T09:54:51.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.700+0000 7f490ba56640 1 librados: shutdown 2024-08-19T09:54:51.743 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:51.743 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:54:51.743 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:54:51.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.842+0000 7fd7bc306640 10 monclient: get_monmap_and_config 2024-08-19T09:54:51.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.842+0000 7fd7bc306640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.842+0000 7fd7bc306640 10 monclient: monmap: 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:51.843696+0000 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:51.843696+0000 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.845 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.845 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.843+0000 7fd7bc306640 1 Processor -- start 2024-08-19T09:54:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.843+0000 7fd7bc306640 1 -- start start 2024-08-19T09:54:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.843+0000 7fd7bc306640 10 monclient: init 2024-08-19T09:54:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.843+0000 7fd7bc306640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 10 monclient: _add_conns ranks=[5,4,0,8,6,3,2,7,1] 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 10 monclient(hunting): picked mon.noname-h con 0x7fd7b40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b413f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b40a3b40 con 0x7fd7b413f5c0 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b40a8130 con 0x7fd7b40a6ba0 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b406ede0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7bc306640 10 monclient(hunting): authenticate will time out at 947.920092s 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7b98a1640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd7b40a6ba0 0x7fd7b40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59370/0 (socket says 172.21.15.1:59370) 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7ba8a3640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7b413f5c0 0x7fd7b41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37164/0 (socket says 172.21.15.1:37164) 2024-08-19T09:54:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.844+0000 7fd7b98a1640 1 -- 172.21.15.1:0/1181674919 learned_addr learned my addr 172.21.15.1:0/1181674919 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b98a1640 1 -- 172.21.15.1:0/1181674919 >> v1:172.21.15.1:6791/0 conn(0x7fd7b40a6ba0 legacy=0x7fd7b40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7ba8a3640 1 -- 172.21.15.1:0/1181674919 >> v1:172.21.15.1:6789/0 conn(0x7fd7b413f5c0 legacy=0x7fd7b41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459573721 0 0) 0x7fd7b40a3b40 con 0x7fd7b413f5c0 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): my global_id is 134595 2024-08-19T09:54:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd79c003520 con 0x7fd7b413f5c0 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116957868 0 0) 0x7fd7b40a8130 con 0x7fd7b40a6ba0 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient(hunting): my global_id is 26896 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b40a3b40 con 0x7fd7b40a6ba0 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7ba0a2640 1 -- 172.21.15.1:0/1181674919 >> v1:172.21.15.82:6791/0 conn(0x7fd7b40a67c0 legacy=0x7fd7b40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2650893907 0 0) 0x7fd79c003520 con 0x7fd7b413f5c0 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 >> v1:172.21.15.82:6791/0 conn(0x7fd7b40a67c0 legacy=0x7fd7b40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 >> v1:172.21.15.1:6791/0 conn(0x7fd7b40a6ba0 legacy=0x7fd7b40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 monclient: found mon.noname-a 2024-08-19T09:54:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b40a92a0 con 0x7fd7b413f5c0 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7a4001e60 con 0x7fd7b413f5c0 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7bc306640 5 monclient: authenticate success, global_id 134595 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7bc306640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7b90a0640 10 monclient: dump: 2024-08-19T09:54:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.846+0000 7fd7bc306640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7a4004b60 con 0x7fd7b413f5c0 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7b90a0640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7b90a0640 1 -- 172.21.15.1:0/1181674919 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7a40050d0 con 0x7fd7b413f5c0 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7b90a0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7b90a0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7b90a0640 10 monclient: dump: 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.859 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 4 set_mon_vals no callback set 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 10 monclient: shutdown 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 1 -- 172.21.15.1:0/1181674919 >> v1:172.21.15.1:6789/0 conn(0x7fd7b413f5c0 legacy=0x7fd7b41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 1 -- 172.21.15.1:0/1181674919 shutdown_connections 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 1 -- 172.21.15.1:0/1181674919 >> 172.21.15.1:0/1181674919 conn(0x7fd7b40ac4b0 msgr2=0x7fd7b40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 1 -- 172.21.15.1:0/1181674919 shutdown_connections 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.847+0000 7fd7bc306640 1 -- 172.21.15.1:0/1181674919 wait complete. 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient: build_initial_monmap 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient: monmap: 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:51.862 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 librados: starting msgr at 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 librados: starting objecter 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 Processor -- start 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 -- start start 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 librados: setting wanted keys 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 librados: calling monclient init 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient: init 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient: _add_conns ranks=[8,3,7,4,6,5,0,1,2] 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient(hunting): picked mon.noname-e con 0x7fd7b40a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient(hunting): picked mon.noname-g con 0x7fd7b40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 10 monclient(hunting): picked mon.noname-i con 0x7fd7b413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.848+0000 7fd7bc306640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b413a2b0 con 0x7fd7b40a6ba0 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.849+0000 7fd7bc306640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b40aa260 con 0x7fd7b413f5c0 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.849+0000 7fd7bc306640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b40a9d50 con 0x7fd7b40a67c0 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.849+0000 7fd7bc306640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.849+0000 7fd7bc306640 10 monclient(hunting): authenticate will time out at 947.925056s 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.849+0000 7fd7ba0a2640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd7b40a67c0 0x7fd7b40a0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40598/0 (socket says 172.21.15.1:40598) 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.849+0000 7fd7ba0a2640 1 -- 172.21.15.1:0/788454765 learned_addr learned my addr 172.21.15.1:0/788454765 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7ba0a2640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd7b40a67c0 legacy=0x7fd7b40a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd7b40a67c0 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205455940 0 0) 0x7fd7b40a9d50 con 0x7fd7b40a67c0 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): my global_id is 27395 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7ba8a3640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.82:6792/0 conn(0x7fd7b413f5c0 legacy=0x7fd7b41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7a0004eb0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd7b413f5c0 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b98a1640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.82:6790/0 conn(0x7fd7b40a6ba0 legacy=0x7fd7b40a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd7b40a6ba0 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1666463797 0 0) 0x7fd7b40aa260 con 0x7fd7b413f5c0 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): my global_id is 144133 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b40a9d50 con 0x7fd7b413f5c0 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462774586 0 0) 0x7fd7b413a2b0 con 0x7fd7b40a6ba0 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): my global_id is 134607 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b40aa260 con 0x7fd7b40a6ba0 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4150434481 0 0) 0x7fd7a0004eb0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7b413a2b0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7ac003610 con 0x7fd7b40a67c0 2024-08-19T09:54:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): dump: 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.867 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3615720645 0 0) 0x7fd7b40a9d50 con 0x7fd7b413f5c0 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7a0004eb0 con 0x7fd7b413f5c0 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7a40020b0 con 0x7fd7b413f5c0 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient(hunting): dump: 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.868 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.869 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 110042425 0 0) 0x7fd7b413a2b0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.82:6792/0 conn(0x7fd7b413f5c0 legacy=0x7fd7b41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.82:6790/0 conn(0x7fd7b40a6ba0 legacy=0x7fd7b40a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 monclient: found mon.e 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b413a870 con 0x7fd7b40a67c0 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient: _finish_auth 0 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 5 monclient: authenticate success, global_id 27395 2024-08-19T09:54:51.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 10 monclient: _renew_subs 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b40a1ec0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 10 client.27395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 10 monclient: _renew_subs 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.851+0000 7fd7bc306640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7b40a1150 con 0x7fd7b40a67c0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7bc306640 1 librados: init done 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7ac003b90 con 0x7fd7b40a67c0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7ac0040c0 con 0x7fd7b40a67c0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b889f640 4 set_mon_vals no callback set 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 10 monclient: dump: 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.872 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7bc306640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd7940065c0 con 0x7fd794002860 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.852+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd7ac004260 con 0x7fd7b40a67c0 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.853+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7fd7ac04e380 con 0x7fd7b40a67c0 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.853+0000 7fd7b2ffd640 10 client.27395.objecter ms_dispatch 0x7fd7b4000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.853+0000 7fd7b2ffd640 3 client.27395.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.853+0000 7fd7b2ffd640 3 client.27395.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.853+0000 7fd7b2ffd640 20 client.27395.objecter dump_active .. 0 homeless 2024-08-19T09:54:51.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.853+0000 7fd7ba8a3640 10 monclient: get_auth_request con 0x7fd7a0041600 auth_method 0 2024-08-19T09:54:51.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 client.27395.objecter ms_handle_connect 0x7fd794002860 2024-08-19T09:54:51.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b98a1640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd794002860 legacy=0x7fd794004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 client.27395.objecter resend_mon_ops 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484493210 0 0) 0x7fd7940065c0 con 0x7fd794002860 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient(hunting): my global_id is 27404 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b413a2b0 con 0x7fd794002860 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2595566438 0 0) 0x7fd7b413a2b0 con 0x7fd794002860 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7940065c0 con 0x7fd794002860 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7a8004eb0 con 0x7fd794002860 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient: dump: 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1447063965 0 0) 0x7fd7940065c0 con 0x7fd794002860 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd79400a160 con 0x7fd794002860 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 client.27395.objecter ms_handle_connect 0x7fd7a0041600 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd7a8006370 con 0x7fd794002860 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd794002860 legacy=0x7fd794004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.855+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd794002860 legacy=0x7fd794004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.862+0000 7fd7bc306640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.862+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd794009e50 con 0x7fd794009110 2024-08-19T09:54:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b98a1640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd794009110 legacy=0x7fd7940095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 client.27395.objecter ms_handle_connect 0x7fd794009110 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 client.27395.objecter resend_mon_ops 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229002858 0 0) 0x7fd794009e50 con 0x7fd794009110 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient(hunting): my global_id is 27413 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7940065c0 con 0x7fd794009110 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 448931803 0 0) 0x7fd7940065c0 con 0x7fd794009110 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd794009e50 con 0x7fd794009110 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7a80053e0 con 0x7fd794009110 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 10 monclient: dump: 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3984999902 0 0) 0x7fd794009e50 con 0x7fd794009110 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.864+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd79400c3a0 con 0x7fd794009110 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3885552942) 0x7fd7a8005890 con 0x7fd794009110 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7b2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd794009110 legacy=0x7fd7940095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7b2ffd640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd794009110 legacy=0x7fd7940095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 298138, 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:51.883 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.884 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.885 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.886 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:51.887 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:51.888 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:51.889 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:51.890 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.891 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:51.894 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.895 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.896 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.897 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.898 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.899 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.901 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:51.902 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 10 librados: watch_flush enter 2024-08-19T09:54:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 10 librados: watch_flush exit 2024-08-19T09:54:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 20 client.27395.objecter shutdown clearing up homeless session... 2024-08-19T09:54:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 10 client.27395.objecter successfully canceled tick 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd7a0041600 legacy=0x7fd7a0043ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 10 monclient: shutdown 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 >> v1:172.21.15.1:6793/0 conn(0x7fd7b40a67c0 legacy=0x7fd7b40a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7ba8a3640 1 -- 172.21.15.1:0/788454765 reap_dead start 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7ba8a3640 1 -- 172.21.15.1:0/788454765 reap_dead start 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 shutdown_connections 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 >> 172.21.15.1:0/788454765 conn(0x7fd7b40ac4b0 msgr2=0x7fd7b40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 shutdown_connections 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 -- 172.21.15.1:0/788454765 wait complete. 2024-08-19T09:54:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:51.865+0000 7fd7bc306640 1 librados: shutdown 2024-08-19T09:54:51.906 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:51.906 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:54:51.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.006+0000 7f724d248640 10 monclient: get_monmap_and_config 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.006+0000 7f724d248640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.006+0000 7f724d248640 10 monclient: monmap: 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:52.007114+0000 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:52.007114+0000 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.008 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.006+0000 7f724d248640 1 Processor -- start 2024-08-19T09:54:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 1 -- start start 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient: init 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient: _add_conns ranks=[6,0,2,5,7,1,8,4,3] 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient(hunting): picked mon.noname-d con 0x7f72480a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient(hunting): picked mon.noname-a con 0x7f72480694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient(hunting): picked mon.noname-b con 0x7f72480a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72480a3c70 con 0x7f72480694b0 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72480a33d0 con 0x7f72480a46a0 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72480a36b0 con 0x7f72480a6830 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f724d248640 10 monclient(hunting): authenticate will time out at 948.083593s 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f7246d9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f72480a6830 0x7f72480a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52806/0 (socket says 172.21.15.1:52806) 2024-08-19T09:54:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.007+0000 7f7246d9d640 1 -- 172.21.15.1:0/3596088572 learned_addr learned my addr 172.21.15.1:0/3596088572 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.009+0000 7f7246d9d640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6792/0 conn(0x7f72480a6830 legacy=0x7f72480a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.009+0000 7f724759e640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 legacy=0x7f7248141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.009+0000 7f723ffff640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6789/0 conn(0x7f72480694b0 legacy=0x7f72480a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.009+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1412921806 0 0) 0x7f72480a36b0 con 0x7f72480a6830 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): my global_id is 26988 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7234003520 con 0x7f72480a6830 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232573615 0 0) 0x7f72480a33d0 con 0x7f72480a46a0 2024-08-19T09:54:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): my global_id is 144137 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72480a36b0 con 0x7f72480a46a0 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897856566 0 0) 0x7f72480a3c70 con 0x7f72480694b0 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): my global_id is 134604 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72480a33d0 con 0x7f72480694b0 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724759e640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 legacy=0x7f7248141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-08-19T09:54:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724759e640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 legacy=0x7f7248141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724759e640 1 --1- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 0x7f7248141850 :-1 s=OPENED pgs=3147 cs=1 l=1).handle_message read tag failed 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724759e640 1 --1- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 0x7f7248141850 :-1 s=OPENED pgs=3147 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6790/0 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 480608212 0 0) 0x7f7234003520 con 0x7f72480a6830 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 legacy=0x7f7248141850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6789/0 conn(0x7f72480694b0 legacy=0x7f72480a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 monclient: found mon.noname-d 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724806dbb0 con 0x7f72480a6830 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7238001ae0 con 0x7f72480a6830 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 5 monclient: authenticate success, global_id 26988 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: dump: 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.018 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72380048a0 con 0x7f72480a6830 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 1 -- 172.21.15.1:0/3596088572 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7238004e10 con 0x7f72480a6830 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 4 set_mon_vals no callback set 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724659c640 10 monclient: dump: 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 10 monclient: shutdown 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 1 -- 172.21.15.1:0/3596088572 >> v1:172.21.15.1:6792/0 conn(0x7f72480a6830 legacy=0x7f72480a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 1 -- 172.21.15.1:0/3596088572 shutdown_connections 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.010+0000 7f724d248640 1 -- 172.21.15.1:0/3596088572 >> 172.21.15.1:0/3596088572 conn(0x7f72480a9020 msgr2=0x7f7248068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.011+0000 7f724d248640 1 -- 172.21.15.1:0/3596088572 shutdown_connections 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.011+0000 7f724d248640 1 -- 172.21.15.1:0/3596088572 wait complete. 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.011+0000 7f724d248640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.011+0000 7f724d248640 10 monclient: monmap: 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.011+0000 7f724d248640 1 librados: starting msgr at 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.011+0000 7f724d248640 1 librados: starting objecter 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 Processor -- start 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 -- start start 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 librados: setting wanted keys 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 librados: calling monclient init 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient: init 2024-08-19T09:54:52.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient: _add_conns ranks=[0,2,3,7,6,8,1,4,5] 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient(hunting): picked mon.noname-a con 0x7f72480694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient(hunting): picked mon.noname-b con 0x7f72480a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient(hunting): picked mon.noname-g con 0x7f72480a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72481cdf10 con 0x7f72480694b0 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7248142d30 con 0x7f72480a46a0 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72481ce4a0 con 0x7f72480a6830 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f724d248640 10 monclient(hunting): authenticate will time out at 948.088670s 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f7246d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f72480694b0 0x7f7248141f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37184/0 (socket says 172.21.15.1:37184) 2024-08-19T09:54:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.012+0000 7f7246d9d640 1 -- 172.21.15.1:0/92544456 learned_addr learned my addr 172.21.15.1:0/92544456 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f7246d9d640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.1:6789/0 conn(0x7f72480694b0 legacy=0x7f7248141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723ffff640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 legacy=0x7f72481d2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 client.?.objecter ms_handle_connect 0x7f72480694b0 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 client.?.objecter ms_handle_connect 0x7f72480a46a0 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120199993 0 0) 0x7f724806dbb0 con 0x7f72480694b0 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.014+0000 7f723f7fe640 10 monclient(hunting): my global_id is 134613 2024-08-19T09:54:52.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7228004ee0 con 0x7f72480694b0 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247349184 0 0) 0x7f7248142d30 con 0x7f72480a46a0 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): my global_id is 144146 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f724806dbb0 con 0x7f72480a46a0 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 client.?.objecter ms_handle_connect 0x7f72480a6830 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724759e640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6790/0 conn(0x7f72480a6830 legacy=0x7f72481d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3270513026 0 0) 0x7f7228004ee0 con 0x7f72480694b0 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7248142d30 con 0x7f72480694b0 2024-08-19T09:54:52.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515004369 0 0) 0x7f72481ce4a0 con 0x7f72480a6830 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): my global_id is 134616 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7228004ee0 con 0x7f72480a6830 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7238001d30 con 0x7f72480694b0 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): dump: 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.032 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3328507847 0 0) 0x7f724806dbb0 con 0x7f72480a46a0 2024-08-19T09:54:52.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72481ce4a0 con 0x7f72480a46a0 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7230003950 con 0x7f72480a46a0 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient(hunting): dump: 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.034 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 905715257 0 0) 0x7f7248142d30 con 0x7f72480694b0 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6790/0 conn(0x7f72480a6830 legacy=0x7f72481d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.1:6790/0 conn(0x7f72480a46a0 legacy=0x7f72481d2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 monclient: found mon.a 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72481386a0 con 0x7f72480694b0 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 5 monclient: authenticate success, global_id 134613 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 10 monclient: _renew_subs 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f724813a1c0 con 0x7f72480694b0 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 10 client.134613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:52.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 10 monclient: _renew_subs 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72481ce150 con 0x7f72480694b0 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f724d248640 1 librados: init done 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72380048a0 con 0x7f72480694b0 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7238005380 con 0x7f72480694b0 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f7245d9b640 4 set_mon_vals no callback set 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.015+0000 7f723f7fe640 10 monclient: dump: 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.037 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.038 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f724d248640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f722c0065c0 con 0x7f722c002860 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f72380117e0 con 0x7f72480694b0 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7f723804e2c0 con 0x7f72480694b0 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f723f7fe640 10 client.134613.objecter ms_dispatch 0x7f7248000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f723f7fe640 3 client.134613.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f723f7fe640 3 client.134613.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f723f7fe640 20 client.134613.objecter dump_active .. 0 homeless 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.016+0000 7f724759e640 10 monclient: get_auth_request con 0x7f72280416a0 auth_method 0 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 10 client.134613.objecter ms_handle_connect 0x7f722c002860 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 10 client.134613.objecter resend_mon_ops 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723ffff640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6789/0 conn(0x7f722c002860 legacy=0x7f722c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010998590 0 0) 0x7f722c0065c0 con 0x7f722c002860 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 10 monclient(hunting): my global_id is 28315 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.018+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7248142d30 con 0x7f722c002860 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3763265116 0 0) 0x7f7248142d30 con 0x7f722c002860 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f722c0065c0 con 0x7f722c002860 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7230003e30 con 0x7f722c002860 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 10 monclient: dump: 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.040 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.041 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 10 client.134613.objecter ms_handle_connect 0x7f72280416a0 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2827943272 0 0) 0x7f722c0065c0 con 0x7f722c002860 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f722c00a160 con 0x7f722c002860 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7230003ec0 con 0x7f722c002860 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6789/0 conn(0x7f722c002860 legacy=0x7f722c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.019+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6789/0 conn(0x7f722c002860 legacy=0x7f722c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.025+0000 7f724d248640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.025+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f722c009e70 con 0x7f722c009130 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 client.134613.objecter ms_handle_connect 0x7f722c009130 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723ffff640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6789/0 conn(0x7f722c009130 legacy=0x7f722c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 client.134613.objecter resend_mon_ops 2024-08-19T09:54:52.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483099376 0 0) 0x7f722c009e70 con 0x7f722c009130 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient(hunting): my global_id is 28324 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f722c0065c0 con 0x7f722c009130 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2195372278 0 0) 0x7f722c0065c0 con 0x7f722c009130 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f722c009e70 con 0x7f722c009130 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f72300043c0 con 0x7f722c009130 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.028+0000 7f723f7fe640 10 monclient: dump: 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.043 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.044 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3253050762 0 0) 0x7f722c009e70 con 0x7f722c009130 2024-08-19T09:54:52.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f722c00c410 con 0x7f722c009130 2024-08-19T09:54:52.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 2539546027) 0x7f7230005cf0 con 0x7f722c009130 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6789/0 conn(0x7f722c009130 legacy=0x7f722c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f723f7fe640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6789/0 conn(0x7f722c009130 legacy=0x7f722c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:52.046 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 298289, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:54:52.047 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.048 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.049 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:52.050 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.051 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.052 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.054 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:52.055 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.056 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.058 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.062 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.063 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.064 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:52.065 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 10 librados: watch_flush enter 2024-08-19T09:54:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 10 librados: watch_flush exit 2024-08-19T09:54:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 20 client.134613.objecter shutdown clearing up homeless session... 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 10 client.134613.objecter successfully canceled tick 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.82:6812/2549104415 conn(0x7f72280416a0 legacy=0x7f7228043b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 10 monclient: shutdown 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 >> v1:172.21.15.1:6789/0 conn(0x7f72480694b0 legacy=0x7f7248141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724759e640 1 -- 172.21.15.1:0/92544456 reap_dead start 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724759e640 1 -- 172.21.15.1:0/92544456 reap_dead start 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 shutdown_connections 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 >> 172.21.15.1:0/92544456 conn(0x7f72480a9020 msgr2=0x7f72480a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 shutdown_connections 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.029+0000 7f724d248640 1 -- 172.21.15.1:0/92544456 wait complete. 2024-08-19T09:54:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.030+0000 7f724d248640 1 librados: shutdown 2024-08-19T09:54:52.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:52.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:54:52.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:54:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.168+0000 7f18c3fff640 10 monclient: get_monmap_and_config 2024-08-19T09:54:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.168+0000 7f18c3fff640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.168+0000 7f18c3fff640 10 monclient: monmap: 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:52.169583+0000 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:52.169583+0000 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.171 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 1 Processor -- start 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 1 -- start start 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient: init 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient: _add_conns ranks=[8,4,3,2,7,5,6,1,0] 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient(hunting): picked mon.noname-e con 0x7f18c40a8ce0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f18c40a5720 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient(hunting): picked mon.noname-g con 0x7f18c40a5b00 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c4136660 con 0x7f18c40a5b00 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c409ded0 con 0x7f18c40a5720 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c409e1b0 con 0x7f18c40a8ce0 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c3fff640 10 monclient(hunting): authenticate will time out at 948.245903s 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c27fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f18c40a5720 0x7f18c413e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59372/0 (socket says 172.21.15.1:59372) 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c27fc640 1 -- 172.21.15.1:0/1173913572 learned_addr learned my addr 172.21.15.1:0/1173913572 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.169+0000 7f18c2ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f18c40a8ce0 0x7f18c4135f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40632/0 (socket says 172.21.15.1:40632) 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c2ffd640 1 -- 172.21.15.1:0/1173913572 >> v1:172.21.15.1:6793/0 conn(0x7f18c40a8ce0 legacy=0x7f18c4135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c27fc640 1 -- 172.21.15.1:0/1173913572 >> v1:172.21.15.1:6791/0 conn(0x7f18c40a5720 legacy=0x7f18c413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094845823 0 0) 0x7f18c409e1b0 con 0x7f18c40a8ce0 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient(hunting): my global_id is 27440 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac003520 con 0x7f18c40a8ce0 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3504564669 0 0) 0x7f18ac003520 con 0x7f18c40a8ce0 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c37fe640 1 -- 172.21.15.1:0/1173913572 >> v1:172.21.15.82:6790/0 conn(0x7f18c40a5b00 legacy=0x7f18c41418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 >> v1:172.21.15.1:6791/0 conn(0x7f18c40a5720 legacy=0x7f18c413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 >> v1:172.21.15.82:6790/0 conn(0x7f18c40a5b00 legacy=0x7f18c41418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 monclient: found mon.noname-e 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:54:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c40a5f50 con 0x7f18c40a8ce0 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18b4001ba0 con 0x7f18c40a8ce0 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c3fff640 5 monclient: authenticate success, global_id 27440 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: dump: 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18b40047e0 con 0x7f18c40a8ce0 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 1 -- 172.21.15.1:0/1173913572 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18b4004d10 con 0x7f18c40a8ce0 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c1ffb640 10 monclient: dump: 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c3fff640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.172+0000 7f18c3fff640 4 set_mon_vals no callback set 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 10 monclient: shutdown 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 -- 172.21.15.1:0/1173913572 >> v1:172.21.15.1:6793/0 conn(0x7f18c40a8ce0 legacy=0x7f18c4135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 -- 172.21.15.1:0/1173913572 shutdown_connections 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 -- 172.21.15.1:0/1173913572 >> 172.21.15.1:0/1173913572 conn(0x7f18c40ac100 msgr2=0x7f18c40a85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 -- 172.21.15.1:0/1173913572 shutdown_connections 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 -- 172.21.15.1:0/1173913572 wait complete. 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 10 monclient: monmap: 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.183 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 librados: starting msgr at 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.173+0000 7f18c3fff640 1 librados: starting objecter 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 Processor -- start 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 -- start start 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 librados: setting wanted keys 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 librados: calling monclient init 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient: init 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient: _add_conns ranks=[8,3,2,6,0,4,1,5,7] 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient(hunting): picked mon.noname-e con 0x7f18c40a5720 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient(hunting): picked mon.noname-g con 0x7f18c40a5b00 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f18c40a8ce0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c4131630 con 0x7f18c40a8ce0 2024-08-19T09:54:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c4131910 con 0x7f18c40a5b00 2024-08-19T09:54:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c412ea80 con 0x7f18c40a5720 2024-08-19T09:54:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c3fff640 10 monclient(hunting): authenticate will time out at 948.250733s 2024-08-19T09:54:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c2ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f18c40a5720 0x7f18c412dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40640/0 (socket says 172.21.15.1:40640) 2024-08-19T09:54:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.174+0000 7f18c2ffd640 1 -- 172.21.15.1:0/1145395564 learned_addr learned my addr 172.21.15.1:0/1145395564 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.176+0000 7f18c2ffd640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.1:6793/0 conn(0x7f18c40a5720 legacy=0x7f18c412dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.176+0000 7f18ab7fe640 10 client.?.objecter ms_handle_connect 0x7f18c40a5720 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.176+0000 7f18ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.176+0000 7f18ab7fe640 10 client.?.objecter ms_handle_connect 0x7f18c40a8ce0 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.176+0000 7f18ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.176+0000 7f18c37fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.1:6790/0 conn(0x7f18c40a8ce0 legacy=0x7f18c41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 197365060 0 0) 0x7f18c412ea80 con 0x7f18c40a5720 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): my global_id is 27449 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18b0004eb0 con 0x7f18c40a5720 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982307451 0 0) 0x7f18c4131630 con 0x7f18c40a8ce0 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): my global_id is 144155 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c412ea80 con 0x7f18c40a8ce0 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c27fc640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18c40a5b00 legacy=0x7f18c412e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 client.?.objecter ms_handle_connect 0x7f18c40a5b00 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 417389533 0 0) 0x7f18b0004eb0 con 0x7f18c40a5720 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18c4131630 con 0x7f18c40a5720 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18b4001b60 con 0x7f18c40a5720 2024-08-19T09:54:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): dump: 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3266797667 0 0) 0x7f18c412ea80 con 0x7f18c40a8ce0 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18b0004eb0 con 0x7f18c40a8ce0 2024-08-19T09:54:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18bc002050 con 0x7f18c40a8ce0 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): dump: 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435406894 0 0) 0x7f18c4131910 con 0x7f18c40a5b00 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient(hunting): my global_id is 134625 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c412ea80 con 0x7f18c40a5b00 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1642873286 0 0) 0x7f18c4131630 con 0x7f18c40a5720 2024-08-19T09:54:52.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18c40a5b00 legacy=0x7f18c412e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.1:6790/0 conn(0x7f18c40a8ce0 legacy=0x7f18c41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 monclient: found mon.e 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c4131e10 con 0x7f18c40a5720 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 5 monclient: authenticate success, global_id 27449 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 10 monclient: _renew_subs 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18c4130aa0 con 0x7f18c40a5720 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 10 client.27449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:52.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 10 monclient: _renew_subs 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f18c4131070 con 0x7f18c40a5720 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c3fff640 1 librados: init done 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18b40047e0 con 0x7f18c40a5720 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18b4001d00 con 0x7f18c40a5720 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18c17fa640 4 set_mon_vals no callback set 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.177+0000 7f18ab7fe640 10 monclient: dump: 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.197 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18c3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f18a40065c0 con 0x7f18a4002860 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f18b4011810 con 0x7f18c40a5720 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7f18b404e290 con 0x7f18c40a5720 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18ab7fe640 10 client.27449.objecter ms_dispatch 0x7f18c4000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18ab7fe640 3 client.27449.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18ab7fe640 3 client.27449.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18ab7fe640 20 client.27449.objecter dump_active .. 0 homeless 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.178+0000 7f18c37fe640 10 monclient: get_auth_request con 0x7f18b00416e0 auth_method 0 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.180+0000 7f18c27fc640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18a4002860 legacy=0x7f18a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.180+0000 7f18ab7fe640 10 client.27449.objecter ms_handle_connect 0x7f18a4002860 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.180+0000 7f18ab7fe640 10 client.27449.objecter resend_mon_ops 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609058937 0 0) 0x7f18a40065c0 con 0x7f18a4002860 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient(hunting): my global_id is 134634 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c4131630 con 0x7f18a4002860 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 client.27449.objecter ms_handle_connect 0x7f18b00416e0 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2212525855 0 0) 0x7f18c4131630 con 0x7f18a4002860 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18a40065c0 con 0x7f18a4002860 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18b8001c20 con 0x7f18a4002860 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 10 monclient: dump: 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2578833247 0 0) 0x7f18a40065c0 con 0x7f18a4002860 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.181+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18a400a160 con 0x7f18a4002860 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.182+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f18b8004840 con 0x7f18a4002860 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.182+0000 7f18ab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.182+0000 7f18ab7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.182+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18a4002860 legacy=0x7f18a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.182+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18a4002860 legacy=0x7f18a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.188+0000 7f18c3fff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.188+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f18a4009e50 con 0x7f18a4009110 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18c27fc640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18a4009110 legacy=0x7f18a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 10 client.27449.objecter ms_handle_connect 0x7f18a4009110 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 10 client.27449.objecter resend_mon_ops 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3001870116 0 0) 0x7f18a4009e50 con 0x7f18a4009110 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 10 monclient(hunting): my global_id is 134643 2024-08-19T09:54:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.190+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a40065c0 con 0x7f18a4009110 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3913557186 0 0) 0x7f18a40065c0 con 0x7f18a4009110 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18a4009e50 con 0x7f18a4009110 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f18b8005030 con 0x7f18a4009110 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 10 monclient: dump: 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3757255238 0 0) 0x7f18a4009e50 con 0x7f18a4009110 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.191+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18a400c3a0 con 0x7f18a4009110 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1660237218) 0x7f18b8007330 con 0x7f18a4009110 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18ab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18ab7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18a4009110 legacy=0x7f18a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18ab7fe640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6790/0 conn(0x7f18a4009110 legacy=0x7f18a40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 149252, 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:52.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.207 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:52.209 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.213 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:52.214 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.215 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.217 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.218 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.219 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:52.220 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.221 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.223 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.224 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.225 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:52.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.226 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.226 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:52.226 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:52.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 10 librados: watch_flush enter 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 10 librados: watch_flush exit 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 20 client.27449.objecter shutdown clearing up homeless session... 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 10 client.27449.objecter successfully canceled tick 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.82:6812/2549104415 conn(0x7f18b00416e0 legacy=0x7f18b0043ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 10 monclient: shutdown 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 >> v1:172.21.15.1:6793/0 conn(0x7f18c40a5720 legacy=0x7f18c412dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c37fe640 1 -- 172.21.15.1:0/1145395564 reap_dead start 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c37fe640 1 -- 172.21.15.1:0/1145395564 reap_dead start 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 shutdown_connections 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 >> 172.21.15.1:0/1145395564 conn(0x7f18c40ac100 msgr2=0x7f18c40aaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 shutdown_connections 2024-08-19T09:54:52.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 -- 172.21.15.1:0/1145395564 wait complete. 2024-08-19T09:54:52.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.192+0000 7f18c3fff640 1 librados: shutdown 2024-08-19T09:54:52.229 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:52.229 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:54:52.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:54:52.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.327+0000 7fe90acde640 10 monclient: get_monmap_and_config 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.327+0000 7fe90acde640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.327+0000 7fe90acde640 10 monclient: monmap: 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:52.328644+0000 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:52.328644+0000 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.329 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.330 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.330 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.330 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.330 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.330 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 1 Processor -- start 2024-08-19T09:54:52.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 1 -- start start 2024-08-19T09:54:52.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 10 monclient: init 2024-08-19T09:54:52.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 10 monclient: _add_conns ranks=[6,5,8,1,3,4,7,2,0] 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 10 monclient(hunting): picked mon.noname-d con 0x7fe9040a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 10 monclient(hunting): picked mon.noname-h con 0x7fe9040a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 10 monclient(hunting): picked mon.noname-e con 0x7fe90413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.328+0000 7fe90acde640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9040a3b40 con 0x7fe9040a6ba0 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe90acde640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9040a8130 con 0x7fe9040a67c0 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe90acde640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe90406ede0 con 0x7fe90413f5c0 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe90acde640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe90acde640 10 monclient(hunting): authenticate will time out at 948.405033s 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe908a7a640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe9040a67c0 0x7fe9040a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52814/0 (socket says 172.21.15.1:52814) 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe90927b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe90413f5c0 0x7fe9041419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40644/0 (socket says 172.21.15.1:40644) 2024-08-19T09:54:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.329+0000 7fe908a7a640 1 -- 172.21.15.1:0/1144502619 learned_addr learned my addr 172.21.15.1:0/1144502619 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90927b640 1 -- 172.21.15.1:0/1144502619 >> v1:172.21.15.1:6793/0 conn(0x7fe90413f5c0 legacy=0x7fe9041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe908a7a640 1 -- 172.21.15.1:0/1144502619 >> v1:172.21.15.1:6792/0 conn(0x7fe9040a67c0 legacy=0x7fe9040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491408295 0 0) 0x7fe90406ede0 con 0x7fe90413f5c0 2024-08-19T09:54:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): my global_id is 27458 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8ec003520 con 0x7fe90413f5c0 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264061860 0 0) 0x7fe9040a8130 con 0x7fe9040a67c0 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient(hunting): my global_id is 26997 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe90406ede0 con 0x7fe9040a67c0 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fbfff640 1 -- 172.21.15.1:0/1144502619 >> v1:172.21.15.82:6791/0 conn(0x7fe9040a6ba0 legacy=0x7fe9040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3773458008 0 0) 0x7fe8ec003520 con 0x7fe90413f5c0 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 >> v1:172.21.15.1:6792/0 conn(0x7fe9040a67c0 legacy=0x7fe9040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 >> v1:172.21.15.82:6791/0 conn(0x7fe9040a6ba0 legacy=0x7fe9040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 monclient: found mon.noname-e 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9040a92a0 con 0x7fe90413f5c0 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe8fc001e50 con 0x7fe90413f5c0 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90acde640 5 monclient: authenticate success, global_id 27458 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90acde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: dump: 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90acde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90927b640 1 --1- 172.21.15.1:0/1144502619 >> v1:172.21.15.1:6793/0 conn(0x7fe90413f5c0 0x7fe9041419b0 :-1 s=OPENED pgs=3310 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fe8fc0050d0 mon_map magic: 0 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe8fc004b60 con 0x7fe90413f5c0 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe8fb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90acde640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.331+0000 7fe90acde640 4 set_mon_vals no callback set 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 10 monclient: shutdown 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 1 -- 172.21.15.1:0/1144502619 >> v1:172.21.15.1:6793/0 conn(0x7fe90413f5c0 legacy=0x7fe9041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe8fb7fe640 1 -- 172.21.15.1:0/1144502619 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe8fc0050d0 con 0x7fe90413f5c0 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe8fb7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 1 -- 172.21.15.1:0/1144502619 shutdown_connections 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 1 -- 172.21.15.1:0/1144502619 >> 172.21.15.1:0/1144502619 conn(0x7fe9040ac4b0 msgr2=0x7fe9040a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 1 -- 172.21.15.1:0/1144502619 shutdown_connections 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 1 -- 172.21.15.1:0/1144502619 wait complete. 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.332+0000 7fe90acde640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient: monmap: 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 librados: starting msgr at 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 librados: starting objecter 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 Processor -- start 2024-08-19T09:54:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 -- start start 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 librados: setting wanted keys 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 librados: calling monclient init 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient: init 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient: _add_conns ranks=[6,0,5,4,1,2,8,7,3] 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient(hunting): picked mon.noname-d con 0x7fe9040a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient(hunting): picked mon.noname-a con 0x7fe9040a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 10 monclient(hunting): picked mon.noname-h con 0x7fe90413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe904136360 con 0x7fe9040a6ba0 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9040a1c10 con 0x7fe90413f5c0 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.333+0000 7fe90acde640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9040a9e50 con 0x7fe9040a67c0 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.334+0000 7fe90acde640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.334+0000 7fe90acde640 10 monclient(hunting): authenticate will time out at 948.410016s 2024-08-19T09:54:52.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.334+0000 7fe908a7a640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe9040a67c0 0x7fe904135540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52816/0 (socket says 172.21.15.1:52816) 2024-08-19T09:54:52.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.334+0000 7fe908a7a640 1 -- 172.21.15.1:0/3878896200 learned_addr learned my addr 172.21.15.1:0/3878896200 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe908a7a640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.1:6792/0 conn(0x7fe9040a67c0 legacy=0x7fe904135540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 client.?.objecter ms_handle_connect 0x7fe9040a67c0 2024-08-19T09:54:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 client.?.objecter ms_handle_connect 0x7fe9040a6ba0 2024-08-19T09:54:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8fbfff640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.1:6789/0 conn(0x7fe9040a6ba0 legacy=0x7fe904135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8fbfff640 1 --1- 172.21.15.1:0/3878896200 >> v1:172.21.15.1:6789/0 conn(0x7fe9040a6ba0 0x7fe904135c50 :-1 s=OPENED pgs=312 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fe904136360 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3521501090 0 0) 0x7fe9040a9e50 con 0x7fe9040a67c0 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): my global_id is 27006 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90927b640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe90413f5c0 legacy=0x7fe9041d6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8f0004eb0 con 0x7fe9040a67c0 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 client.?.objecter ms_handle_connect 0x7fe90413f5c0 2024-08-19T09:54:52.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757723447 0 0) 0x7fe9040a1c10 con 0x7fe90413f5c0 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): my global_id is 26627 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9040a9e50 con 0x7fe90413f5c0 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2253542272 0 0) 0x7fe8f0004eb0 con 0x7fe9040a67c0 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9040a1c10 con 0x7fe9040a67c0 2024-08-19T09:54:52.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe900001ce0 con 0x7fe9040a67c0 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): dump: 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 366858164 0 0) 0x7fe9040a1c10 con 0x7fe9040a67c0 2024-08-19T09:54:52.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe90413f5c0 legacy=0x7fe9041d6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.1:6789/0 conn(0x7fe9040a6ba0 legacy=0x7fe904135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 monclient: found mon.d 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9040aa360 con 0x7fe9040a67c0 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 5 monclient: authenticate success, global_id 27006 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261786351 0 0) 0x7fe904136360 con 0x7fe9040a6ba0 2024-08-19T09:54:52.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe8f97fa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 10 monclient: _renew_subs 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe904142cd0 con 0x7fe9040a67c0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 10 client.27006.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 10 monclient: _renew_subs 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe90413aab0 con 0x7fe9040a67c0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.336+0000 7fe90acde640 1 librados: init done 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe900004ac0 con 0x7fe9040a67c0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe900004fb0 con 0x7fe9040a67c0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8faffd640 4 set_mon_vals no callback set 2024-08-19T09:54:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 10 monclient: dump: 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.356 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe90acde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe8e40065c0 con 0x7fe8e4002860 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.337+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe900005250 con 0x7fe9040a67c0 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.338+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(194..194 src has 1..194) ==== 5780+0+0 (unknown 337065095 0 0) 0x7fe90004e280 con 0x7fe9040a67c0 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.338+0000 7fe8f97fa640 10 client.27006.objecter ms_dispatch 0x7fe904000b90 osd_map(194..194 src has 1..194) 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.338+0000 7fe8f97fa640 3 client.27006.objecter handle_osd_map got epochs [194,194] > 0 2024-08-19T09:54:52.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.338+0000 7fe8f97fa640 3 client.27006.objecter handle_osd_map decoding full epoch 194 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.338+0000 7fe8f97fa640 20 client.27006.objecter dump_active .. 0 homeless 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.338+0000 7fe90927b640 10 monclient: get_auth_request con 0x7fe8f0040fb0 auth_method 0 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 client.27006.objecter ms_handle_connect 0x7fe8e4002860 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8fbfff640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe8e4002860 legacy=0x7fe8e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 client.27006.objecter resend_mon_ops 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524590479 0 0) 0x7fe8e40065c0 con 0x7fe8e4002860 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient(hunting): my global_id is 26636 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe904136360 con 0x7fe8e4002860 2024-08-19T09:54:52.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2469968441 0 0) 0x7fe904136360 con 0x7fe8e4002860 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8e40065c0 con 0x7fe8e4002860 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe8f4001cf0 con 0x7fe8e4002860 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 monclient: dump: 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.359 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 10 client.27006.objecter ms_handle_connect 0x7fe8f0040fb0 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3775557061 0 0) 0x7fe8e40065c0 con 0x7fe8e4002860 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.340+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8e400a160 con 0x7fe8e4002860 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.341+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe8f40057c0 con 0x7fe8e4002860 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.341+0000 7fe8f97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.341+0000 7fe8f97fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:52.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.341+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe8e4002860 legacy=0x7fe8e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.341+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe8e4002860 legacy=0x7fe8e4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.347+0000 7fe90acde640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:52.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.347+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe8e4009e20 con 0x7fe8e40090e0 2024-08-19T09:54:52.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.349+0000 7fe8fbfff640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe8e40090e0 legacy=0x7fe8e4009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.349+0000 7fe8f97fa640 10 client.27006.objecter ms_handle_connect 0x7fe8e40090e0 2024-08-19T09:54:52.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.349+0000 7fe8f97fa640 10 client.27006.objecter resend_mon_ops 2024-08-19T09:54:52.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3695877512 0 0) 0x7fe8e4009e20 con 0x7fe8e40090e0 2024-08-19T09:54:52.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient(hunting): my global_id is 26645 2024-08-19T09:54:52.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8e40065c0 con 0x7fe8e40090e0 2024-08-19T09:54:52.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3941070544 0 0) 0x7fe8e40065c0 con 0x7fe8e40090e0 2024-08-19T09:54:52.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8e4009e20 con 0x7fe8e40090e0 2024-08-19T09:54:52.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe8f4003e30 con 0x7fe8e40090e0 2024-08-19T09:54:52.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:52.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient: dump: 2024-08-19T09:54:52.366 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.366 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.366 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.368 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.368 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.368 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.368 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.368 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.369 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3502244304 0 0) 0x7fe8e4009e20 con 0x7fe8e40090e0 2024-08-19T09:54:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe8e400c3f0 con 0x7fe8e40090e0 2024-08-19T09:54:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 51501099) 0x7fe8f4004120 con 0x7fe8e40090e0 2024-08-19T09:54:52.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:52.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:52.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe8e40090e0 legacy=0x7fe8e4009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.350+0000 7fe8f97fa640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6791/0 conn(0x7fe8e40090e0 legacy=0x7fe8e4009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.371 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:52.371 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:52.371 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:52.372 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:52.372 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:52.372 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 298616, 2024-08-19T09:54:52.372 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:52.372 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:52.373 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:52.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:52.373 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:52.373 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:52.373 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:52.374 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:52.374 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:52.374 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:52.374 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.374 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:54:52.374 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.375 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:52.375 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:52.375 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.375 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.375 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.376 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.376 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.376 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.376 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.376 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.377 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:52.378 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:52.379 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.380 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:52.385 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:52.388 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.389 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.391 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:52.392 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.393 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.396 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.396 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:52.396 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 10 librados: watch_flush enter 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 10 librados: watch_flush exit 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 20 client.27006.objecter shutdown clearing up homeless session... 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 10 client.27006.objecter successfully canceled tick 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe8f0040fb0 legacy=0x7fe8f0043470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 10 monclient: shutdown 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 >> v1:172.21.15.1:6792/0 conn(0x7fe9040a67c0 legacy=0x7fe904135540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90927b640 1 -- 172.21.15.1:0/3878896200 reap_dead start 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90927b640 1 -- 172.21.15.1:0/3878896200 reap_dead start 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 shutdown_connections 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 >> 172.21.15.1:0/3878896200 conn(0x7fe9040ac4b0 msgr2=0x7fe9040a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 shutdown_connections 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 -- 172.21.15.1:0/3878896200 wait complete. 2024-08-19T09:54:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.351+0000 7fe90acde640 1 librados: shutdown 2024-08-19T09:54:52.399 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:52.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:54:52.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:54:52.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.499+0000 7fd17ea84640 10 monclient: get_monmap_and_config 2024-08-19T09:54:52.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.499+0000 7fd17ea84640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.499+0000 7fd17ea84640 10 monclient: monmap: 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:52.500469+0000 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:52.500469+0000 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.501 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.502 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.502 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.502 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.502 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.502 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 1 Processor -- start 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 1 -- start start 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient: init 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient: _add_conns ranks=[8,0,6,3,7,5,2,1,4] 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient(hunting): picked mon.noname-e con 0x7fd1780a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient(hunting): picked mon.noname-a con 0x7fd1780a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient(hunting): picked mon.noname-d con 0x7fd17813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1780a3b40 con 0x7fd1780a6ba0 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1780a8130 con 0x7fd17813f5c0 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17806ede0 con 0x7fd1780a67c0 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17ea84640 10 monclient(hunting): authenticate will time out at 948.576841s 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17c820640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd1780a67c0 0x7fd1780a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40646/0 (socket says 172.21.15.1:40646) 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17d021640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fd17813f5c0 0x7fd1781419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52822/0 (socket says 172.21.15.1:52822) 2024-08-19T09:54:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.500+0000 7fd17c820640 1 -- 172.21.15.1:0/2797224283 learned_addr learned my addr 172.21.15.1:0/2797224283 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd17d021640 1 -- 172.21.15.1:0/2797224283 >> v1:172.21.15.1:6792/0 conn(0x7fd17813f5c0 legacy=0x7fd1781419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd17c820640 1 -- 172.21.15.1:0/2797224283 >> v1:172.21.15.1:6793/0 conn(0x7fd1780a67c0 legacy=0x7fd1780a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd177fff640 1 -- 172.21.15.1:0/2797224283 >> v1:172.21.15.1:6789/0 conn(0x7fd1780a6ba0 legacy=0x7fd1780a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 973223492 0 0) 0x7fd1780a8130 con 0x7fd17813f5c0 2024-08-19T09:54:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): my global_id is 27015 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd168003520 con 0x7fd17813f5c0 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675605945 0 0) 0x7fd17806ede0 con 0x7fd1780a67c0 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): my global_id is 27467 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1780a8130 con 0x7fd1780a67c0 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471304721 0 0) 0x7fd1780a3b40 con 0x7fd1780a6ba0 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient(hunting): my global_id is 134631 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd17806ede0 con 0x7fd1780a6ba0 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 753463077 0 0) 0x7fd168003520 con 0x7fd17813f5c0 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 >> v1:172.21.15.1:6793/0 conn(0x7fd1780a67c0 legacy=0x7fd1780a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 >> v1:172.21.15.1:6789/0 conn(0x7fd1780a6ba0 legacy=0x7fd1780a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 monclient: found mon.noname-d 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1780a92a0 con 0x7fd17813f5c0 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd170001e60 con 0x7fd17813f5c0 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: dump: 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd17ea84640 5 monclient: authenticate success, global_id 27015 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd164001ad0 con 0x7fd1780a6ba0 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd17ea84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd170004b60 con 0x7fd17813f5c0 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd17ea84640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 1 -- 172.21.15.1:0/2797224283 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd170005050 con 0x7fd17813f5c0 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd17ea84640 4 set_mon_vals no callback set 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.503+0000 7fd1777fe640 10 monclient: dump: 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.516 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 10 monclient: shutdown 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 1 -- 172.21.15.1:0/2797224283 >> v1:172.21.15.1:6792/0 conn(0x7fd17813f5c0 legacy=0x7fd1781419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 1 -- 172.21.15.1:0/2797224283 shutdown_connections 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 1 -- 172.21.15.1:0/2797224283 >> 172.21.15.1:0/2797224283 conn(0x7fd1780ac4b0 msgr2=0x7fd1780a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 1 -- 172.21.15.1:0/2797224283 shutdown_connections 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 1 -- 172.21.15.1:0/2797224283 wait complete. 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.504+0000 7fd17ea84640 10 monclient: monmap: 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:52.517 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.518 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 librados: starting msgr at 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 librados: starting objecter 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 Processor -- start 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 -- start start 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 librados: setting wanted keys 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 librados: calling monclient init 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient: init 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient: _add_conns ranks=[4,1,6,8,7,3,2,5,0] 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient(hunting): picked mon.noname-c con 0x7fd1780a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient(hunting): picked mon.noname-f con 0x7fd1780a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient(hunting): picked mon.noname-d con 0x7fd17813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17813b6d0 con 0x7fd1780a6ba0 2024-08-19T09:54:52.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17813c7b0 con 0x7fd1780a67c0 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17813a760 con 0x7fd17813f5c0 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17ea84640 10 monclient(hunting): authenticate will time out at 948.581929s 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.505+0000 7fd17c820640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 0x7fd178139940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59384/0 (socket says 172.21.15.1:59384) 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.506+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 learned_addr learned my addr 172.21.15.1:0/207823088 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 legacy=0x7fd178139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 client.?.objecter ms_handle_connect 0x7fd1780a67c0 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 client.?.objecter ms_handle_connect 0x7fd17813f5c0 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17d021640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6792/0 conn(0x7fd17813f5c0 legacy=0x7fd1781d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122278367 0 0) 0x7fd17813c7b0 con 0x7fd1780a67c0 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): my global_id is 26905 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd15c004eb0 con 0x7fd1780a67c0 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3346290892 0 0) 0x7fd17813a760 con 0x7fd17813f5c0 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): my global_id is 27024 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd17813c7b0 con 0x7fd17813f5c0 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd177fff640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6789/0 conn(0x7fd1780a6ba0 legacy=0x7fd17813a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 client.?.objecter ms_handle_connect 0x7fd1780a6ba0 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204322716 0 0) 0x7fd17813b6d0 con 0x7fd1780a6ba0 2024-08-19T09:54:52.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): my global_id is 28333 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd17813a760 con 0x7fd1780a6ba0 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3500015094 0 0) 0x7fd15c004eb0 con 0x7fd1780a67c0 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd17813b6d0 con 0x7fd1780a67c0 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd16c0019f0 con 0x7fd1780a67c0 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): dump: 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.522 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.523 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3928720650 0 0) 0x7fd17813c7b0 con 0x7fd17813f5c0 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd15c004eb0 con 0x7fd17813f5c0 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd1700020b0 con 0x7fd17813f5c0 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): dump: 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.524 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1933827642 0 0) 0x7fd17813b6d0 con 0x7fd1780a67c0 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6792/0 conn(0x7fd17813f5c0 legacy=0x7fd1781d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6789/0 conn(0x7fd1780a6ba0 legacy=0x7fd17813a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 monclient: found mon.c 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17813ca90 con 0x7fd1780a67c0 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 5 monclient: authenticate success, global_id 26905 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 10 monclient: _renew_subs 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1780aa080 con 0x7fd1780a67c0 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 legacy=0x7fd178139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T09:54:52.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 10 client.26905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:52.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 legacy=0x7fd178139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:54:52.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 10 monclient: _renew_subs 2024-08-19T09:54:52.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17c820640 1 --1- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 0x7fd178139940 :-1 s=OPENED pgs=3251 cs=1 l=1).handle_message read tag failed 2024-08-19T09:54:52.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:52.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17c820640 1 --1- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 0x7fd178139940 :-1 s=OPENED pgs=3251 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:54:52.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1780a2280 con 0x7fd1780a67c0 2024-08-19T09:54:52.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd17ea84640 1 librados: init done 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.1:6791/0 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd1780a67c0 legacy=0x7fd178139940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient: _add_conns ranks=[7,4,1,3,6,5,8,2,0] 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): picked mon.i con 0x7fd15c007530 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): picked mon.c con 0x7fd15c00b040 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 10 monclient(hunting): picked mon.f con 0x7fd1780a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.508+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd17813b6d0 con 0x7fd1780a6ba0 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.509+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd17813a760 con 0x7fd15c00b040 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.509+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd17813c7b0 con 0x7fd15c007530 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.509+0000 7fd1757fa640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.509+0000 7fd17ea84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.509+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd1781421c0 con 0x7fd178139940 2024-08-19T09:54:52.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter ms_handle_connect 0x7fd15c00b040 2024-08-19T09:54:52.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter resend_mon_ops 2024-08-19T09:54:52.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd177fff640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd15c00b040 legacy=0x7fd15c00d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd15c007530 legacy=0x7fd15c009920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter ms_handle_connect 0x7fd15c007530 2024-08-19T09:54:52.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter resend_mon_ops 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter ms_handle_connect 0x7fd1780a6ba0 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd17d021640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6789/0 conn(0x7fd1780a6ba0 legacy=0x7fd15c010fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter resend_mon_ops 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386933276 0 0) 0x7fd17813a760 con 0x7fd15c00b040 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd15c014b10 con 0x7fd15c00b040 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395659797 0 0) 0x7fd17813b6d0 con 0x7fd1780a6ba0 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:54:52.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd17813a760 con 0x7fd1780a6ba0 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587402924 0 0) 0x7fd17813c7b0 con 0x7fd15c007530 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd17813b6d0 con 0x7fd15c007530 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2846735568 0 0) 0x7fd15c014b10 con 0x7fd15c00b040 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd15c007530 legacy=0x7fd15c009920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6789/0 conn(0x7fd1780a6ba0 legacy=0x7fd15c010fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 monclient: found mon.c 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:52.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fd15c015d60 con 0x7fd15c00b040 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd164003c30 con 0x7fd15c00b040 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: dump: 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.538 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd164003e40 con 0x7fd15c00b040 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd176ffd640 4 set_mon_vals no callback set 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter ms_handle_connect 0x7fd178139940 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.511+0000 7fd1757fa640 10 client.26905.objecter resend_mon_ops 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd178139940 legacy=0x7fd178139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd1640040f0 con 0x7fd15c00b040 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.4 v1:172.21.15.1:6791/0 6 ==== osd_map(195..195 src has 1..195) ==== 6480+0+0 (unknown 3429768955 0 0) 0x7fd16404e590 con 0x7fd15c00b040 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 10 client.26905.objecter ms_dispatch 0x7fd178000b90 osd_map(195..195 src has 1..195) 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 3 client.26905.objecter handle_osd_map got epochs [195,195] > 0 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 3 client.26905.objecter handle_osd_map decoding full epoch 195 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 20 client.26905.objecter dump_active .. 0 homeless 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3267611520 0 0) 0x7fd17813b6d0 con 0x7fd178139940 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 10 monclient(hunting): my global_id is 144142 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd15c014b10 con 0x7fd178139940 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.512+0000 7fd17d021640 10 monclient: get_auth_request con 0x7fd1780a67c0 auth_method 0 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2638218872 0 0) 0x7fd15c014b10 con 0x7fd178139940 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd17813b6d0 con 0x7fd178139940 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd16c005450 con 0x7fd178139940 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 10 monclient: dump: 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2355920822 0 0) 0x7fd17813b6d0 con 0x7fd178139940 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd178142960 con 0x7fd178139940 2024-08-19T09:54:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd16c005870 con 0x7fd178139940 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd178139940 legacy=0x7fd178139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.513+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd178139940 legacy=0x7fd178139d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.515+0000 7fd1757fa640 10 client.26905.objecter ms_handle_connect 0x7fd1780a67c0 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.519+0000 7fd17ea84640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.519+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd1780a2250 con 0x7fd17809da50 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd17c820640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd17809da50 legacy=0x7fd17809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 client.26905.objecter ms_handle_connect 0x7fd17809da50 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 client.26905.objecter resend_mon_ops 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218351149 0 0) 0x7fd1780a2250 con 0x7fd17809da50 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient(hunting): my global_id is 144151 2024-08-19T09:54:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd17813b6d0 con 0x7fd17809da50 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 176723903 0 0) 0x7fd17813b6d0 con 0x7fd17809da50 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1780a2250 con 0x7fd17809da50 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd16c006010 con 0x7fd17809da50 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 10 monclient: dump: 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.546 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.546 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.546 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.546 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2597176373 0 0) 0x7fd1780a2250 con 0x7fd17809da50 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.522+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1780a9ca0 con 0x7fd17809da50 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 510085032) 0x7fd16c008330 con 0x7fd17809da50 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd1757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd1757fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd17809da50 legacy=0x7fd17809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd1757fa640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6792/0 conn(0x7fd17809da50 legacy=0x7fd17809de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:52.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17d021640 1 -- 172.21.15.1:0/207823088 reap_dead start 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16902, 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:52.548 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.549 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.550 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.551 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.552 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.555 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:52.556 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.558 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.562 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:52.563 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.565 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.567 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:52.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.570 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:52.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:52.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:52.570 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:52.570 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:52.570 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:52.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:52.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:52.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:52.571 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:52.571 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 10 librados: watch_flush enter 2024-08-19T09:54:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 10 librados: watch_flush exit 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 20 client.26905.objecter shutdown clearing up homeless session... 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 10 client.26905.objecter successfully canceled tick 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd1780a67c0 legacy=0x7fd15c04f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 10 monclient: shutdown 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 >> v1:172.21.15.1:6791/0 conn(0x7fd15c00b040 legacy=0x7fd15c00d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 shutdown_connections 2024-08-19T09:54:52.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 >> 172.21.15.1:0/207823088 conn(0x7fd1780ac4b0 msgr2=0x7fd178141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 shutdown_connections 2024-08-19T09:54:52.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 -- 172.21.15.1:0/207823088 wait complete. 2024-08-19T09:54:52.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.523+0000 7fd17ea84640 1 librados: shutdown 2024-08-19T09:54:52.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:52.576 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T09:54:52.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T09:54:52.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.675+0000 7f5f7a99e640 10 monclient: get_monmap_and_config 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.675+0000 7f5f7a99e640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.675+0000 7f5f7a99e640 10 monclient: monmap: 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:52.676435+0000 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:52.676435+0000 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.678 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 1 Processor -- start 2024-08-19T09:54:52.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 1 -- start start 2024-08-19T09:54:52.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient: init 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient: _add_conns ranks=[1,5,3,6,0,4,2,8,7] 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient(hunting): picked mon.noname-f con 0x7f5f740a77d0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient(hunting): picked mon.noname-h con 0x7f5f740a1e60 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient(hunting): picked mon.noname-g con 0x7f5f740a82c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f74068e30 con 0x7f5f740a77d0 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f740692f0 con 0x7f5f740a82c0 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f740a2620 con 0x7f5f740a1e60 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.676+0000 7f5f7a99e640 10 monclient(hunting): authenticate will time out at 948.752953s 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.677+0000 7f5f7999c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5f740a77d0 0x7f5f740a7bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:52316/0 (socket says 172.21.15.1:52316) 2024-08-19T09:54:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.677+0000 7f5f7919b640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f5f740a1e60 0x7f5f7413e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:41254/0 (socket says 172.21.15.1:41254) 2024-08-19T09:54:52.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.677+0000 7f5f7999c640 1 -- 172.21.15.1:0/3971843634 learned_addr learned my addr 172.21.15.1:0/3971843634 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7a19d640 1 -- 172.21.15.1:0/3971843634 >> v1:172.21.15.82:6790/0 conn(0x7f5f740a82c0 legacy=0x7f5f74141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7919b640 1 -- 172.21.15.1:0/3971843634 >> v1:172.21.15.82:6791/0 conn(0x7f5f740a1e60 legacy=0x7f5f7413e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7999c640 1 -- 172.21.15.1:0/3971843634 >> v1:172.21.15.82:6789/0 conn(0x7f5f740a77d0 legacy=0x7f5f740a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916145865 0 0) 0x7f5f740692f0 con 0x7f5f740a82c0 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): my global_id is 134652 2024-08-19T09:54:52.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f60003520 con 0x7f5f740a82c0 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596425577 0 0) 0x7f5f74068e30 con 0x7f5f740a77d0 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): my global_id is 28342 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f740692f0 con 0x7f5f740a77d0 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3152305584 0 0) 0x7f5f740a2620 con 0x7f5f740a1e60 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient(hunting): my global_id is 26654 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f74068e30 con 0x7f5f740a1e60 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2417090419 0 0) 0x7f5f60003520 con 0x7f5f740a82c0 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 >> v1:172.21.15.82:6791/0 conn(0x7f5f740a1e60 legacy=0x7f5f7413e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 >> v1:172.21.15.82:6789/0 conn(0x7f5f740a77d0 legacy=0x7f5f740a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 monclient: found mon.noname-g 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f740a2900 con 0x7f5f740a82c0 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f6c001eb0 con 0x7f5f740a82c0 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: dump: 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7a99e640 5 monclient: authenticate success, global_id 134652 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7a99e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f6c003990 con 0x7f5f740a82c0 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 1 -- 172.21.15.1:0/3971843634 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f6c003f40 con 0x7f5f740a82c0 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7a99e640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7a99e640 4 set_mon_vals no callback set 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.679+0000 7f5f7899a640 10 monclient: dump: 2024-08-19T09:54:52.695 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 10 monclient: shutdown 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3971843634 >> v1:172.21.15.82:6790/0 conn(0x7f5f740a82c0 legacy=0x7f5f74141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3971843634 shutdown_connections 2024-08-19T09:54:52.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3971843634 >> 172.21.15.1:0/3971843634 conn(0x7f5f740a8ef0 msgr2=0x7f5f740a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3971843634 shutdown_connections 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.680+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3971843634 wait complete. 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient: build_initial_monmap 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient: monmap: 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:52.698 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 librados: starting msgr at 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 librados: starting objecter 2024-08-19T09:54:52.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 Processor -- start 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 -- start start 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 librados: setting wanted keys 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 librados: calling monclient init 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient: init 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient: _add_conns ranks=[1,6,3,7,5,4,8,0,2] 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient(hunting): picked mon.noname-f con 0x7f5f740a1e60 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient(hunting): picked mon.noname-d con 0x7f5f740a77d0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient(hunting): picked mon.noname-g con 0x7f5f740a82c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f74141bf0 con 0x7f5f740a1e60 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f740a54e0 con 0x7f5f740a82c0 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f740a5070 con 0x7f5f740a77d0 2024-08-19T09:54:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.681+0000 7f5f7a99e640 10 monclient(hunting): authenticate will time out at 948.757928s 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.682+0000 7f5f7919b640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5f740a77d0 0x7f5f7413f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52838/0 (socket says 172.21.15.1:52838) 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.682+0000 7f5f7919b640 1 -- 172.21.15.1:0/3623183897 learned_addr learned my addr 172.21.15.1:0/3623183897 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7919b640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.1:6792/0 conn(0x7f5f740a77d0 legacy=0x7f5f7413f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f740a77d0 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7999c640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.82:6789/0 conn(0x7f5f740a1e60 legacy=0x7f5f7413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029179367 0 0) 0x7f5f740a5070 con 0x7f5f740a77d0 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): my global_id is 27033 2024-08-19T09:54:52.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f58004eb0 con 0x7f5f740a77d0 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a19d640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.82:6790/0 conn(0x7f5f740a82c0 legacy=0x7f5f741d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f740a1e60 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f740a82c0 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411823501 0 0) 0x7f5f74141bf0 con 0x7f5f740a1e60 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): my global_id is 28351 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f740a5070 con 0x7f5f740a1e60 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608294288 0 0) 0x7f5f740a54e0 con 0x7f5f740a82c0 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): my global_id is 134661 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f74141bf0 con 0x7f5f740a82c0 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2163644183 0 0) 0x7f5f58004eb0 con 0x7f5f740a77d0 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f740a54e0 con 0x7f5f740a77d0 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f640047e0 con 0x7f5f740a77d0 2024-08-19T09:54:52.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): dump: 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.704 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1418130189 0 0) 0x7f5f740a5070 con 0x7f5f740a1e60 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f58004eb0 con 0x7f5f740a1e60 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f700047e0 con 0x7f5f740a1e60 2024-08-19T09:54:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient(hunting): dump: 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.706 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2862020189 0 0) 0x7f5f740a54e0 con 0x7f5f740a77d0 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.82:6790/0 conn(0x7f5f740a82c0 legacy=0x7f5f741d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.82:6789/0 conn(0x7f5f740a1e60 legacy=0x7f5f7413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 monclient: found mon.d 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f74138110 con 0x7f5f740a77d0 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 5 monclient: authenticate success, global_id 27033 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 10 monclient: _renew_subs 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f74140700 con 0x7f5f740a77d0 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 10 client.27033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 10 monclient: _renew_subs 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f7411f3d0 con 0x7f5f740a77d0 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f7a99e640 1 librados: init done 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f64004d60 con 0x7f5f740a77d0 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f64005250 con 0x7f5f740a77d0 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6bfff640 4 set_mon_vals no callback set 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.684+0000 7f5f6a7fc640 10 monclient: dump: 2024-08-19T09:54:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:52.710 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5f640116b0 con 0x7f5f740a77d0 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(195..195 src has 1..195) ==== 6480+0+0 (unknown 3429768955 0 0) 0x7f5f6404e6f0 con 0x7f5f740a77d0 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f6a7fc640 10 client.27033.objecter ms_dispatch 0x7f5f74000b90 osd_map(195..195 src has 1..195) 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f7a99e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f6a7fc640 3 client.27033.objecter handle_osd_map got epochs [195,195] > 0 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f6a7fc640 3 client.27033.objecter handle_osd_map decoding full epoch 195 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f7a99e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f7a99e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.685+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f5c005080 con 0x7f5f740a77d0 2024-08-19T09:54:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.686+0000 7f5f7999c640 10 monclient: get_auth_request con 0x7f5f580414d0 auth_method 0 2024-08-19T09:54:52.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.686+0000 7f5f6a7fc640 20 client.27033.objecter dump_active .. 0 homeless 2024-08-19T09:54:52.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.688+0000 7f5f6a7fc640 10 client.27033.objecter ms_handle_connect 0x7f5f580414d0 2024-08-19T09:54:52.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.690+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f5f64017690 con 0x7f5f740a77d0 2024-08-19T09:54:52.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.690+0000 7f5f6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:52.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.690+0000 7f5f6a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:52.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.943+0000 7f5f7a99e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T09:54:52.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.943+0000 7f5f7a99e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T09:54:52.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.943+0000 7f5f7a99e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:52.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.943+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5f5c005b80 con 0x7f5f740a77d0 2024-08-19T09:54:52.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.944+0000 7f5f6a7fc640 1 -- 172.21.15.1:0/3623183897 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f5f64016f90 con 0x7f5f740a77d0 2024-08-19T09:54:52.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.944+0000 7f5f6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T09:54:52.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.944+0000 7f5f6a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=988085373,config=2586702516,health=4031279277,kv=1455257465,logm=1164576476}) 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=3725811175,config=2586702516,health=3448887894,kv=1455257465,logm=1771873416}) 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=2325293702,config=2586702516,health=648049328,kv=1251488192,logm=516641318}) 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=20,config=1,health=12,kv=11,logm=56} crc {auth=3766476705,config=1747058506,health=3448887894,kv=2401233588,logm=1457202316}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=1882426177,config=2586702516,health=3448887894,kv=1455257465,logm=3715494091}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=21,config=2,health=12,kv=11,logm=54} crc {auth=3917517653,config=2586702516,health=3448887894,kv=1251488193,logm=168411503}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=296322130,config=2586702516,health=3448887894,kv=1455257465,logm=2938126141}) 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1613078803,config=2586702516,health=1571889810,kv=1455257465,logm=906572438}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.945+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,config=2,health=12,kv=10,logm=56} crc {auth=3275617686,config=2586702516,health=3448887894,kv=2751544881,logm=2587672264}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423536551}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=36754965}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3424932693}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2579441273}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3369362494}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=434784942}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3702313635}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822014818}) 2024-08-19T09:54:52.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3907565126}) 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f5f7a99e640 10 librados: watch_flush enter 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f5f7a99e640 10 librados: watch_flush exit 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f5f7a99e640 20 client.27033.objecter shutdown clearing up homeless session... 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.947+0000 7f5f7a99e640 10 client.27033.objecter successfully canceled tick 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5f580414d0 legacy=0x7f5f58043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 10 monclient: shutdown 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 >> v1:172.21.15.1:6792/0 conn(0x7f5f740a77d0 legacy=0x7f5f7413f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:52.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 shutdown_connections 2024-08-19T09:54:52.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 >> 172.21.15.1:0/3623183897 conn(0x7f5f740a8ef0 msgr2=0x7f5f74142be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:52.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 shutdown_connections 2024-08-19T09:54:52.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 -- 172.21.15.1:0/3623183897 wait complete. 2024-08-19T09:54:52.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f5f7a99e640 1 librados: shutdown 2024-08-19T09:54:52.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686193535}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4272655128}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2055006097}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3125491235}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3638889261}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=109123177}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=583041668}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434417014}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.948+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3540619991}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2602838283}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008211005}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1550018761}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4180755006}) 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2652381763}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=724036430}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1883702076}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665959630}) 2024-08-19T09:54:52.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:52.950+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4160318635}) 2024-08-19T09:54:52.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.048+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.048+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.048+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988507159}) 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.048+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.048+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.048+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1299465152}) 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2965448525}) 2024-08-19T09:54:53.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4143870344}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4104128598}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1890795846}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2926659354}) 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042213434}) 2024-08-19T09:54:53.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.049+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3361786657}) 2024-08-19T09:54:53.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1483080727}) 2024-08-19T09:54:53.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029337858}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887481732}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2531324678}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3831889873}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1405233044}) 2024-08-19T09:54:53.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=771422917}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909321669}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.326+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2888213048}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141709037}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857480634}) 2024-08-19T09:54:53.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1104679230}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2057562458}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1608952386}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1784157864}) 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1768370670}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382130900}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.328+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2195278017}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703657129}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.329+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2182616319}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=336179004}) 2024-08-19T09:54:53.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4005709497}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=613603258}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3380013615}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3286600690}) 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597679369}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.330+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2680105092}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399107717}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948885644}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2872031298}) 2024-08-19T09:54:53.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1883399747}) 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1982429644}) 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2146471726}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=848152389}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438575938}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.331+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2388200005}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.332+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223324581}) 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575608375}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1991728826}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504082803}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3920381097}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2446879225}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2162672907}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982153154}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.333+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395007768}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000872555}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871907995}) 2024-08-19T09:54:53.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2226444099}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=117085042}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2788235273}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=792770592}) 2024-08-19T09:54:53.367 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=838278957}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095585848}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.334+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=349506041}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803637234}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767798631}) 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.368 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3526983615}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3432342028}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3302089629}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1876020305}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.335+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.369 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2851229982}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682146537}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.336+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1096440482}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2151388457}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732010051}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2469953269}) 2024-08-19T09:54:53.370 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1336904245}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1254823418}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2237965982}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1609294569}) 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.371 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528723390}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.337+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1076298873}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136186986}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269135602}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2370053349}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.372 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1184940617}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3707959846}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1512380985}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1309913471}) 2024-08-19T09:54:53.373 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279775110}) 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.338+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3332544315}) 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3643485748}) 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3379303306}) 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.374 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=379522061}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3000645673}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=47264045}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4292832641}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.375 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454437937}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785729103}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.340+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1117336430}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170630933}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=507371679}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3393999438}) 2024-08-19T09:54:53.376 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4245402979}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1325433269}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3877777457}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3364331569}) 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.377 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801679554}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.341+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1279849021}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930417832}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648883448}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580112787}) 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.378 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3361831670}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=734643464}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=356104120}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.342+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.343+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1976143276}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.343+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.343+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084291873}) 2024-08-19T09:54:53.379 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.343+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=165751444}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456995970}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3427884250}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1001262378}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1277642400}) 2024-08-19T09:54:53.380 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1891522187}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1322408301}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4175684832}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76116877}) 2024-08-19T09:54:53.381 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.344+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1445588709}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.359+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.359+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.359+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3677617415}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2687736500}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246479677}) 2024-08-19T09:54:53.384 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=658411785}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3152061856}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1402679160}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1861939451}) 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.385 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826971664}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.360+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1237600007}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1690142811}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950511332}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1984543596}) 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.386 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=603146803}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1373929472}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1898943534}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=913589660}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957466974}) 2024-08-19T09:54:53.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.378+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=642980974}) 2024-08-19T09:54:53.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857295016}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976700280}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=631149850}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3381152711}) 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2399949041}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2620536913}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3694085985}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152082098}) 2024-08-19T09:54:53.396 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.393+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4169226163}) 2024-08-19T09:54:53.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401178085}) 2024-08-19T09:54:53.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=856583335}) 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1773576461}) 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3378284705}) 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4034110358}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1726426578}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2392087295}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97521958}) 2024-08-19T09:54:53.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.423+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=265419214}) 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168165929}) 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477089833}) 2024-08-19T09:54:53.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1452349518}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1109610280}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3712820911}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1737745319}) 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1736760493}) 2024-08-19T09:54:53.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:53.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450590036}) 2024-08-19T09:54:53.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.558+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2415215988}) 2024-08-19T09:54:53.966 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:54:53.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:54:54.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=946237002}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014446744}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4083743002}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2861728745}) 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3124487557}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1369096244}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=608818338}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474379834}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.997+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4118526252}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194602051}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726987935}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1529959973}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1748260409}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=859906552}) 2024-08-19T09:54:54.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2363869275}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=908667908}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968152258}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:53.999+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4075024209}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734195596}) 2024-08-19T09:54:54.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1497474708}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182706009}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.000+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2975030856}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1376603509}) 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1948965347}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=392338758}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632985996}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.001+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848336006}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3940824392}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2608599538}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3338563378}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2795212011}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2308567166}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2159867512}) 2024-08-19T09:54:54.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=895671314}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136098201}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.002+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4083762028}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=206018504}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709434037}) 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3646721519}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3703413015}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=619810077}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3133350992}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=785209096}) 2024-08-19T09:54:54.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827278811}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.004+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3807933656}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584657235}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000301731}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3670039982}) 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.042 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=576481149}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.005+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2518524356}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=578719026}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1337984511}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060957255}) 2024-08-19T09:54:54.043 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.006+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4274683667}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1215556364}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244032682}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4100253414}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=871723241}) 2024-08-19T09:54:54.044 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3368038240}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=409785394}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1677593311}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044401525}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.007+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4032747568}) 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893186650}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188510847}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1971404319}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=348098773}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3362403375}) 2024-08-19T09:54:54.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4008152676}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3637401338}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651085503}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.009+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2661585005}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827740557}) 2024-08-19T09:54:54.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4121909489}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=348528499}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4050777094}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=412803822}) 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1192246273}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2472251198}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277138075}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.010+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2523000526}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2539448407}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527846598}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2744960472}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3223927030}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3602729790}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.011+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2098265307}) 2024-08-19T09:54:54.050 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2162214769}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652431497}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2363743813}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835077930}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.012+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813875683}) 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1115324375}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1739406550}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2688676958}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2934623687}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.052 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1418188022}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888516632}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.013+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881701294}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065064243}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957771345}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=629211983}) 2024-08-19T09:54:54.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4002192323}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3287115298}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=375665048}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2621483987}) 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358093523}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.014+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=793581487}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=1} crc {logm=3296603431,mdsmap=3140514009,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=1905280745,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=26807816,osdmap=3300726320}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=27,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=5} crc {logm=2526400608,mdsmap=3690439819,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=263453377,mon_config_key=443309246,monmap=3096227638,osd_metadata=3497407357,osd_pg_creating=2555985650,osd_snap=1519312197,osdmap=3362486536}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=2069501892,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3966886632,mon_config_key=699569359,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=507614636}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=1} crc {logm=2459290884,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1624051546,mon_config_key=3785664663,monmap=3096227638,osd_metadata=1009524822,osd_pg_creating=2555985650,osd_snap=4097437126,osdmap=3300726320}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=27,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=1} crc {logm=3462937608,mdsmap=3140514009,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3664024120,mon_config_key=3936908447,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=3066264043,osdmap=3300726320}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=3} crc {logm=3258213958,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=778439930,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=505675056,osdmap=3511019582}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.016+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=1733734377,mdsmap=61829903,mgr=2021673060,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3750284595,mon_config_key=3255104584,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=3060578744}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.017+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.017+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=3} crc {logm=3446472598,mdsmap=61829903,mgr=2462459674,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2011472998,mon_config_key=3562702338,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2555985650,osd_snap=1836790677,osdmap=3511019582}) 2024-08-19T09:54:54.056 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.017+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=23,osdmap=4} crc {logm=3308109271,mdsmap=61829903,mgr=3591800611,mgr_metadata=3429325078,mgrstat=3903683537,mon_config_key=1905280744,monmap=3096227638,osd_metadata=3088495813,osd_pg_creating=2555985650,osd_snap=3931917472,osdmap=4291545177}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3175759994}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4203915097}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=955292691}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3751991349}) 2024-08-19T09:54:54.057 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3731127542}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1306617125}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.018+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.019+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.019+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=422489125}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.019+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.019+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505759134}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.019+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2738586805}) 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3191371212}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2917016988}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4271053109}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2548576160}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.059 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2690848483}) 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.029+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=243412725}) 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=191018075}) 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2653673901}) 2024-08-19T09:54:54.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.030+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2613426588}) 2024-08-19T09:54:54.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.086+0000 7f109f47e640 10 monclient: get_monmap_and_config 2024-08-19T09:54:54.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.086+0000 7f109f47e640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.086+0000 7f109f47e640 10 monclient: monmap: 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:54.087636+0000 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:54.087636+0000 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.088 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.089 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.087+0000 7f109f47e640 1 Processor -- start 2024-08-19T09:54:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.087+0000 7f109f47e640 1 -- start start 2024-08-19T09:54:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.087+0000 7f109f47e640 10 monclient: init 2024-08-19T09:54:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.087+0000 7f109f47e640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.087+0000 7f109f47e640 10 monclient: _add_conns ranks=[5,3,8,2,0,1,6,7,4] 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.087+0000 7f109f47e640 10 monclient(hunting): picked mon.noname-h con 0x7f10980a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 10 monclient(hunting): picked mon.noname-g con 0x7f10980a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 10 monclient(hunting): picked mon.noname-e con 0x7f109813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10980a3b40 con 0x7f10980a6ba0 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10980a8130 con 0x7f10980a67c0 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109806ede0 con 0x7f109813f5c0 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109f47e640 10 monclient(hunting): authenticate will time out at 950.164078s 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109da1b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f109813f5c0 0x7f10981419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40648/0 (socket says 172.21.15.1:40648) 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.088+0000 7f109da1b640 1 -- 172.21.15.1:0/1678212976 learned_addr learned my addr 172.21.15.1:0/1678212976 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f109da1b640 1 -- 172.21.15.1:0/1678212976 >> v1:172.21.15.1:6793/0 conn(0x7f109813f5c0 legacy=0x7f10981419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2074449271 0 0) 0x7f109806ede0 con 0x7f109813f5c0 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient(hunting): my global_id is 27476 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1084003520 con 0x7f109813f5c0 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f109d21a640 1 -- 172.21.15.1:0/1678212976 >> v1:172.21.15.82:6791/0 conn(0x7f10980a67c0 legacy=0x7f10980a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f109ca19640 1 -- 172.21.15.1:0/1678212976 >> v1:172.21.15.82:6790/0 conn(0x7f10980a6ba0 legacy=0x7f10980a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3909717426 0 0) 0x7f1084003520 con 0x7f109813f5c0 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 >> v1:172.21.15.82:6791/0 conn(0x7f10980a67c0 legacy=0x7f10980a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 >> v1:172.21.15.82:6790/0 conn(0x7f10980a6ba0 legacy=0x7f10980a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 monclient: found mon.noname-e 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10980a92a0 con 0x7f109813f5c0 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1090001ad0 con 0x7f109813f5c0 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f109f47e640 5 monclient: authenticate success, global_id 27476 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f108ffff640 10 monclient: dump: 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.090+0000 7f109f47e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10900048a0 con 0x7f109813f5c0 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f108ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f108ffff640 1 -- 172.21.15.1:0/1678212976 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1090004dd0 con 0x7f109813f5c0 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f108ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 4 set_mon_vals no callback set 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f108ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f108ffff640 10 monclient: dump: 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 10 monclient: shutdown 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 1 -- 172.21.15.1:0/1678212976 >> v1:172.21.15.1:6793/0 conn(0x7f109813f5c0 legacy=0x7f10981419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 1 -- 172.21.15.1:0/1678212976 shutdown_connections 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 1 -- 172.21.15.1:0/1678212976 >> 172.21.15.1:0/1678212976 conn(0x7f10980ac4b0 msgr2=0x7f10980a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 1 -- 172.21.15.1:0/1678212976 shutdown_connections 2024-08-19T09:54:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.091+0000 7f109f47e640 1 -- 172.21.15.1:0/1678212976 wait complete. 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient: monmap: 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 1 librados: starting msgr at 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 1 librados: starting objecter 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 1 Processor -- start 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 1 -- start start 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 1 librados: setting wanted keys 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 1 librados: calling monclient init 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient: init 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient: _add_conns ranks=[2,3,6,1,5,7,8,4,0] 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient(hunting): picked mon.noname-b con 0x7f10980a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.092+0000 7f109f47e640 10 monclient(hunting): picked mon.noname-g con 0x7f10980a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109f47e640 10 monclient(hunting): picked mon.noname-d con 0x7f109813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109f47e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10981cf020 con 0x7f10980a67c0 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109f47e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109813ae10 con 0x7f10980a6ba0 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109f47e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1098139920 con 0x7f109813f5c0 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109f47e640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109f47e640 10 monclient(hunting): authenticate will time out at 950.169062s 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109d21a640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f10980a67c0 0x7f10981ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34840/0 (socket says 172.21.15.1:34840) 2024-08-19T09:54:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.093+0000 7f109d21a640 1 -- 172.21.15.1:0/2428174851 learned_addr learned my addr 172.21.15.1:0/2428174851 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f109d21a640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.1:6790/0 conn(0x7f10980a67c0 legacy=0x7f10981ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 client.?.objecter ms_handle_connect 0x7f10980a67c0 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 client.?.objecter ms_handle_connect 0x7f109813f5c0 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f109da1b640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.1:6792/0 conn(0x7f109813f5c0 legacy=0x7f10981d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 client.?.objecter ms_handle_connect 0x7f10980a6ba0 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f109ca19640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.82:6790/0 conn(0x7f10980a6ba0 legacy=0x7f10981ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420942344 0 0) 0x7f10981cf020 con 0x7f10980a67c0 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): my global_id is 144164 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f107c004ee0 con 0x7f10980a67c0 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774269812 0 0) 0x7f1098139920 con 0x7f109813f5c0 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): my global_id is 27042 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10981cf020 con 0x7f109813f5c0 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148679701 0 0) 0x7f109813ae10 con 0x7f10980a6ba0 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): my global_id is 134679 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1098139920 con 0x7f10980a6ba0 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1546778763 0 0) 0x7f107c004ee0 con 0x7f10980a67c0 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f109813ae10 con 0x7f10980a67c0 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1094003990 con 0x7f10980a67c0 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): dump: 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 737823593 0 0) 0x7f10981cf020 con 0x7f109813f5c0 2024-08-19T09:54:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f107c004ee0 con 0x7f109813f5c0 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1090001d20 con 0x7f109813f5c0 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): dump: 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3472576949 0 0) 0x7f1098139920 con 0x7f10980a6ba0 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10981cf020 con 0x7f10980a6ba0 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1088001cd0 con 0x7f10980a6ba0 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:54.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient(hunting): dump: 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.115 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 202567685 0 0) 0x7f109813ae10 con 0x7f10980a67c0 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.1:6792/0 conn(0x7f109813f5c0 legacy=0x7f10981d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.82:6790/0 conn(0x7f10980a6ba0 legacy=0x7f10981ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 monclient: found mon.b 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109813bc80 con 0x7f10980a67c0 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f108dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f109f47e640 5 monclient: authenticate success, global_id 144164 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f109f47e640 10 monclient: _renew_subs 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.095+0000 7f109f47e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f109813b3e0 con 0x7f10980a67c0 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1094003f40 con 0x7f10980a67c0 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1094004430 con 0x7f10980a67c0 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 10 monclient: dump: 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.118 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f109f47e640 10 client.144164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f109f47e640 10 monclient: _renew_subs 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f109f47e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f109813a6f0 con 0x7f10980a67c0 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108f7fe640 4 set_mon_vals no callback set 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f109f47e640 1 librados: init done 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.096+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f10940045d0 con 0x7f10980a67c0 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f109f47e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f109f47e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f109f47e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10980008d0 con 0x7f10980a67c0 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(196..196 src has 1..196) ==== 6188+0+0 (unknown 1967371915 0 0) 0x7f109404e970 con 0x7f10980a67c0 2024-08-19T09:54:54.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f108dffb640 10 client.144164.objecter ms_dispatch 0x7f1098000b90 osd_map(196..196 src has 1..196) 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f108dffb640 3 client.144164.objecter handle_osd_map got epochs [196,196] > 0 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f108dffb640 3 client.144164.objecter handle_osd_map decoding full epoch 196 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f108dffb640 20 client.144164.objecter dump_active .. 0 homeless 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.097+0000 7f109ca19640 10 monclient: get_auth_request con 0x7f107c041980 auth_method 0 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.099+0000 7f108dffb640 10 client.144164.objecter ms_handle_connect 0x7f107c041980 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.101+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f10940179d0 con 0x7f10980a67c0 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.101+0000 7f108dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.101+0000 7f108dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.356+0000 7f109f47e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.356+0000 7f109f47e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.356+0000 7f109f47e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.356+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f109813e4f0 con 0x7f10980a67c0 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.357+0000 7f108dffb640 1 -- 172.21.15.1:0/2428174851 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 879112922) 0x7f10940172d0 con 0x7f10980a67c0 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.357+0000 7f108dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:54:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.357+0000 7f108dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:54.359 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:54:54.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 10 librados: watch_flush enter 2024-08-19T09:54:54.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 10 librados: watch_flush exit 2024-08-19T09:54:54.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 20 client.144164.objecter shutdown clearing up homeless session... 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 10 client.144164.objecter successfully canceled tick 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.82:6812/2549104415 conn(0x7f107c041980 legacy=0x7f107c043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 10 monclient: shutdown 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 >> v1:172.21.15.1:6790/0 conn(0x7f10980a67c0 legacy=0x7f10981ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 shutdown_connections 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 >> 172.21.15.1:0/2428174851 conn(0x7f10980ac4b0 msgr2=0x7f1098141e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 shutdown_connections 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 -- 172.21.15.1:0/2428174851 wait complete. 2024-08-19T09:54:54.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.359+0000 7f109f47e640 1 librados: shutdown 2024-08-19T09:54:54.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=90} crc {osdmap=2909088506}) 2024-08-19T09:54:54.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=95} crc {osdmap=3425833070}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=94} crc {osdmap=363061654}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=93} crc {osdmap=3161770082}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=92} crc {osdmap=3040862917}) 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=93} crc {osdmap=3347629746}) 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=94} crc {osdmap=3584522742}) 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=166973746}) 2024-08-19T09:54:54.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:54:54.362+0000 7f9efa358640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=88} crc {osdmap=1798162663}) 2024-08-19T09:54:54.371 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:54:54.371 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T09:54:54.371 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:54:54.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:54:54.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.509+0000 7f3860353640 10 monclient: get_monmap_and_config 2024-08-19T09:54:54.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.509+0000 7f3860353640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.509+0000 7f3860353640 10 monclient: monmap: 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:54.510572+0000 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:54.510572+0000 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.511 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.512 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.512 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 1 Processor -- start 2024-08-19T09:54:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 1 -- start start 2024-08-19T09:54:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient: init 2024-08-19T09:54:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient: _add_conns ranks=[2,8,7,1,5,3,4,6,0] 2024-08-19T09:54:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient(hunting): picked mon.noname-b con 0x7f38580a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient(hunting): picked mon.noname-e con 0x7f38580a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient(hunting): picked mon.noname-i con 0x7f385813f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38580a3b40 con 0x7f38580a67c0 2024-08-19T09:54:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38580a8130 con 0x7f385813f5c0 2024-08-19T09:54:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385806ede0 con 0x7f38580a6ba0 2024-08-19T09:54:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.510+0000 7f3860353640 10 monclient(hunting): authenticate will time out at 950.586928s 2024-08-19T09:54:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.511+0000 7f385d8ee640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f38580a6ba0 0x7f38580a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40650/0 (socket says 172.21.15.1:40650) 2024-08-19T09:54:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.511+0000 7f385d8ee640 1 -- 172.21.15.1:0/4019028679 learned_addr learned my addr 172.21.15.1:0/4019028679 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.511+0000 7f385e0ef640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f38580a67c0 0x7f38580a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34856/0 (socket says 172.21.15.1:34856) 2024-08-19T09:54:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.513+0000 7f385e0ef640 1 -- 172.21.15.1:0/4019028679 >> v1:172.21.15.1:6790/0 conn(0x7f38580a67c0 legacy=0x7f38580a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.513+0000 7f385d8ee640 1 -- 172.21.15.1:0/4019028679 >> v1:172.21.15.1:6793/0 conn(0x7f38580a6ba0 legacy=0x7f38580a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.513+0000 7f385e8f0640 1 -- 172.21.15.1:0/4019028679 >> v1:172.21.15.82:6792/0 conn(0x7f385813f5c0 legacy=0x7f38581419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430388160 0 0) 0x7f385806ede0 con 0x7f38580a6ba0 2024-08-19T09:54:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): my global_id is 27485 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3844003520 con 0x7f38580a6ba0 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075335570 0 0) 0x7f38580a3b40 con 0x7f38580a67c0 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient(hunting): my global_id is 144173 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f385806ede0 con 0x7f38580a67c0 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607478429 0 0) 0x7f3844003520 con 0x7f38580a6ba0 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 >> v1:172.21.15.82:6792/0 conn(0x7f385813f5c0 legacy=0x7f38581419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 >> v1:172.21.15.1:6790/0 conn(0x7f38580a67c0 legacy=0x7f38580a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 monclient: found mon.noname-e 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38580a92a0 con 0x7f38580a6ba0 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3848001a00 con 0x7f38580a6ba0 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f385d0ed640 10 monclient: dump: 2024-08-19T09:54:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f3860353640 5 monclient: authenticate success, global_id 27485 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.603+0000 7f3860353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3848001cd0 con 0x7f38580a6ba0 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f385d0ed640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f385d0ed640 1 -- 172.21.15.1:0/4019028679 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3848004be0 con 0x7f38580a6ba0 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f385d0ed640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f385d0ed640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f385d0ed640 10 monclient: dump: 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f3860353640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.604+0000 7f3860353640 4 set_mon_vals no callback set 2024-08-19T09:54:54.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.605+0000 7f3860353640 10 monclient: shutdown 2024-08-19T09:54:54.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.605+0000 7f3860353640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.605+0000 7f3860353640 1 -- 172.21.15.1:0/4019028679 >> v1:172.21.15.1:6793/0 conn(0x7f38580a6ba0 legacy=0x7f38580a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.607+0000 7f3860353640 1 -- 172.21.15.1:0/4019028679 shutdown_connections 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.607+0000 7f3860353640 1 -- 172.21.15.1:0/4019028679 >> 172.21.15.1:0/4019028679 conn(0x7f38580ac4b0 msgr2=0x7f38580a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.609+0000 7f3860353640 1 -- 172.21.15.1:0/4019028679 shutdown_connections 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.609+0000 7f3860353640 1 -- 172.21.15.1:0/4019028679 wait complete. 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 10 monclient: monmap: 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.614 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.615 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 1 librados: starting msgr at 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 1 librados: starting objecter 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 1 Processor -- start 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 1 -- start start 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 1 librados: setting wanted keys 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 1 librados: calling monclient init 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.610+0000 7f3860353640 10 monclient: init 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient: _add_conns ranks=[3,8,5,7,0,6,4,1,2] 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient(hunting): picked mon.noname-g con 0x7f38580a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient(hunting): picked mon.noname-e con 0x7f38580a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient(hunting): picked mon.noname-h con 0x7f385813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385813ad70 con 0x7f38580a67c0 2024-08-19T09:54:54.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3858139880 con 0x7f385813f5c0 2024-08-19T09:54:54.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385813bbe0 con 0x7f38580a6ba0 2024-08-19T09:54:54.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f3860353640 10 monclient(hunting): authenticate will time out at 950.687196s 2024-08-19T09:54:54.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f385e8f0640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f385813f5c0 0x7f38581d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:41280/0 (socket says 172.21.15.1:41280) 2024-08-19T09:54:54.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.611+0000 7f385e8f0640 1 -- 172.21.15.1:0/1421074068 learned_addr learned my addr 172.21.15.1:0/1421074068 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.614+0000 7f385e8f0640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.82:6791/0 conn(0x7f385813f5c0 legacy=0x7f38581d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.614+0000 7f384effd640 10 client.?.objecter ms_handle_connect 0x7f385813f5c0 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.614+0000 7f384effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.615+0000 7f385d8ee640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6793/0 conn(0x7f38580a6ba0 legacy=0x7f385813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.615+0000 7f384effd640 10 client.?.objecter ms_handle_connect 0x7f38580a6ba0 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.615+0000 7f384effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.616+0000 7f385e0ef640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.82:6790/0 conn(0x7f38580a67c0 legacy=0x7f385813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.616+0000 7f384effd640 10 client.?.objecter ms_handle_connect 0x7f38580a67c0 2024-08-19T09:54:54.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.616+0000 7f384effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3209081631 0 0) 0x7f3858139880 con 0x7f385813f5c0 2024-08-19T09:54:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): my global_id is 26672 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f383c004ee0 con 0x7f385813f5c0 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500357337 0 0) 0x7f385813ad70 con 0x7f38580a67c0 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): my global_id is 134688 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3858139880 con 0x7f38580a67c0 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512138223 0 0) 0x7f385813bbe0 con 0x7f38580a6ba0 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 10 monclient(hunting): my global_id is 27494 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.623+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f385813ad70 con 0x7f38580a6ba0 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1950564645 0 0) 0x7f3858139880 con 0x7f38580a67c0 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f385813bbe0 con 0x7f38580a67c0 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38540047e0 con 0x7f38580a67c0 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): dump: 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3172220844 0 0) 0x7f385813ad70 con 0x7f38580a6ba0 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3858139880 con 0x7f38580a6ba0 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3252455913 0 0) 0x7f383c004ee0 con 0x7f385813f5c0 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f385813ad70 con 0x7f385813f5c0 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3848004b20 con 0x7f38580a6ba0 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): dump: 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3850003990 con 0x7f385813f5c0 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient(hunting): dump: 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3383571011 0 0) 0x7f385813bbe0 con 0x7f38580a67c0 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6793/0 conn(0x7f38580a6ba0 legacy=0x7f385813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.82:6791/0 conn(0x7f385813f5c0 legacy=0x7f38581d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 monclient: found mon.g 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385813b340 con 0x7f38580a67c0 2024-08-19T09:54:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 5 monclient: authenticate success, global_id 134688 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 10 monclient: _renew_subs 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f385813a680 con 0x7f38580a67c0 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 10 client.134688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 10 monclient: _renew_subs 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f385813cf00 con 0x7f38580a67c0 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f3860353640 1 librados: init done 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3854004d60 con 0x7f38580a67c0 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3854005290 con 0x7f38580a67c0 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f384effd640 10 monclient: dump: 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.624+0000 7f385c8ec640 4 set_mon_vals no callback set 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3854005430 con 0x7f38580a67c0 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f3860353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f38580a9ca0 con 0x7f385809da50 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 6284+0+0 (unknown 2534302429 0 0) 0x7f385404e410 con 0x7f38580a67c0 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f384effd640 10 client.134688.objecter ms_dispatch 0x7f3858000b90 osd_map(197..197 src has 1..197) 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f384effd640 3 client.134688.objecter handle_osd_map got epochs [197,197] > 0 2024-08-19T09:54:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f384effd640 3 client.134688.objecter handle_osd_map decoding full epoch 197 2024-08-19T09:54:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f384effd640 20 client.134688.objecter dump_active .. 0 homeless 2024-08-19T09:54:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.625+0000 7f385e8f0640 10 monclient: get_auth_request con 0x7f383c041a70 auth_method 0 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.627+0000 7f384effd640 10 client.134688.objecter ms_handle_connect 0x7f385809da50 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.627+0000 7f385d8ee640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6789/0 conn(0x7f385809da50 legacy=0x7f385809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.627+0000 7f384effd640 10 client.134688.objecter resend_mon_ops 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704095781 0 0) 0x7f38580a9ca0 con 0x7f385809da50 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient(hunting): my global_id is 134640 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f385813bbe0 con 0x7f385809da50 2024-08-19T09:54:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 client.134688.objecter ms_handle_connect 0x7f383c041a70 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3666728492 0 0) 0x7f385813bbe0 con 0x7f385809da50 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38580a9ca0 con 0x7f385809da50 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3848004cc0 con 0x7f385809da50 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient: dump: 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1681923293 0 0) 0x7f38580a9ca0 con 0x7f385809da50 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f385813e630 con 0x7f385809da50 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3848001d20 con 0x7f385809da50 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6789/0 conn(0x7f385809da50 legacy=0x7f385809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.628+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6789/0 conn(0x7f385809da50 legacy=0x7f385809de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.635+0000 7f3860353640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.635+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3858068f60 con 0x7f385809e540 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 client.134688.objecter ms_handle_connect 0x7f385809e540 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 client.134688.objecter resend_mon_ops 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f385d8ee640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6789/0 conn(0x7f385809e540 legacy=0x7f38581db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946836320 0 0) 0x7f3858068f60 con 0x7f385809e540 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient(hunting): my global_id is 134649 2024-08-19T09:54:54.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38580a9ca0 con 0x7f385809e540 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 650699076 0 0) 0x7f38580a9ca0 con 0x7f385809e540 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3858068f60 con 0x7f385809e540 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f38480058e0 con 0x7f385809e540 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 10 monclient: dump: 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.637+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1824949449 0 0) 0x7f3858068f60 con 0x7f385809e540 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f385809e920 con 0x7f385809e540 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7686 (unknown 0 0 1953269995) 0x7f3848007e60 con 0x7f385809e540 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f384effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f384effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:54.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6789/0 conn(0x7f385809e540 legacy=0x7f38581db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f384effd640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.1:6789/0 conn(0x7f385809e540 legacy=0x7f38581db990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 151701, 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:54.647 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:54.648 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:54.649 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:54.651 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.654 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.655 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:54.658 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:54.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.664 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.665 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.668 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:54.668 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 10 librados: watch_flush enter 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 10 librados: watch_flush exit 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 20 client.134688.objecter shutdown clearing up homeless session... 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 10 client.134688.objecter successfully canceled tick 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.82:6812/2549104415 conn(0x7f383c041a70 legacy=0x7f383c043f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 10 monclient: shutdown 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 >> v1:172.21.15.82:6790/0 conn(0x7f38580a67c0 legacy=0x7f385813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f385e8f0640 1 -- 172.21.15.1:0/1421074068 reap_dead start 2024-08-19T09:54:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f385e8f0640 1 -- 172.21.15.1:0/1421074068 reap_dead start 2024-08-19T09:54:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 shutdown_connections 2024-08-19T09:54:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 >> 172.21.15.1:0/1421074068 conn(0x7f38580ac4b0 msgr2=0x7f385813ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 shutdown_connections 2024-08-19T09:54:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 -- 172.21.15.1:0/1421074068 wait complete. 2024-08-19T09:54:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.638+0000 7f3860353640 1 librados: shutdown 2024-08-19T09:54:54.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:54.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:54:54.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.770+0000 7efd1cd1b640 10 monclient: get_monmap_and_config 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.770+0000 7efd1cd1b640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.770+0000 7efd1cd1b640 10 monclient: monmap: 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:54.771184+0000 2024-08-19T09:54:54.771 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:54.771184+0000 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.772 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.770+0000 7efd1cd1b640 1 Processor -- start 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 1 -- start start 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient: init 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient: _add_conns ranks=[0,8,6,7,3,4,2,5,1] 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient(hunting): picked mon.noname-a con 0x7efd180a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient(hunting): picked mon.noname-e con 0x7efd180a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient(hunting): picked mon.noname-d con 0x7efd1813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:54.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd180a3b40 con 0x7efd180a67c0 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd180a8130 con 0x7efd1813f5c0 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd1806ede0 con 0x7efd180a6ba0 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1cd1b640 10 monclient(hunting): authenticate will time out at 950.847570s 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd16d9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7efd1813f5c0 0x7efd181419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52860/0 (socket says 172.21.15.1:52860) 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd0ffff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7efd180a6ba0 0x7efd180a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40662/0 (socket says 172.21.15.1:40662) 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd16d9d640 1 -- 172.21.15.1:0/1317363998 learned_addr learned my addr 172.21.15.1:0/1317363998 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.771+0000 7efd1659c640 1 --1- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6789/0 conn(0x7efd180a67c0 0x7efd180a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37244/0 (socket says 172.21.15.1:37244) 2024-08-19T09:54:54.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.773+0000 7efd16d9d640 1 -- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6792/0 conn(0x7efd1813f5c0 legacy=0x7efd181419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.773+0000 7efd0ffff640 1 -- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6793/0 conn(0x7efd180a6ba0 legacy=0x7efd180a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.773+0000 7efd1659c640 1 -- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6789/0 conn(0x7efd180a67c0 legacy=0x7efd180a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669133745 0 0) 0x7efd180a3b40 con 0x7efd180a67c0 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): my global_id is 134658 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd04003520 con 0x7efd180a67c0 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134224057 0 0) 0x7efd180a8130 con 0x7efd1813f5c0 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): my global_id is 27051 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd180a3b40 con 0x7efd1813f5c0 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137820236 0 0) 0x7efd1806ede0 con 0x7efd180a6ba0 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient(hunting): my global_id is 27503 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd180a8130 con 0x7efd180a6ba0 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd16d9d640 1 --1- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6792/0 conn(0x7efd1813f5c0 0x7efd181419b0 :-1 s=OPENED pgs=3327 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7efd10001dc0 mon_map magic: 0 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1607849327 0 0) 0x7efd04003520 con 0x7efd180a67c0 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6793/0 conn(0x7efd180a6ba0 legacy=0x7efd180a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6792/0 conn(0x7efd1813f5c0 legacy=0x7efd181419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 monclient: found mon.noname-a 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd180a92a0 con 0x7efd180a67c0 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd1cd1b640 5 monclient: authenticate success, global_id 134658 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd08001ae0 con 0x7efd180a67c0 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd1cd1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: dump: 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.791 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd10001dc0 con 0x7efd1813f5c0 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd1cd1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd080047e0 con 0x7efd180a67c0 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 1 -- 172.21.15.1:0/1317363998 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd08004cd0 con 0x7efd180a67c0 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd1cd1b640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd1cd1b640 4 set_mon_vals no callback set 2024-08-19T09:54:54.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.774+0000 7efd15d9b640 10 monclient: dump: 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.793 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 10 monclient: shutdown 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 1 -- 172.21.15.1:0/1317363998 >> v1:172.21.15.1:6789/0 conn(0x7efd180a67c0 legacy=0x7efd180a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 1 -- 172.21.15.1:0/1317363998 shutdown_connections 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 1 -- 172.21.15.1:0/1317363998 >> 172.21.15.1:0/1317363998 conn(0x7efd180ac4b0 msgr2=0x7efd180a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 1 -- 172.21.15.1:0/1317363998 shutdown_connections 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 1 -- 172.21.15.1:0/1317363998 wait complete. 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.775+0000 7efd1cd1b640 10 monclient: monmap: 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.795 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 librados: starting msgr at 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 librados: starting objecter 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 Processor -- start 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 -- start start 2024-08-19T09:54:54.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 librados: setting wanted keys 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 librados: calling monclient init 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient: init 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient: _add_conns ranks=[0,7,2,5,6,3,1,4,8] 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient(hunting): picked mon.noname-a con 0x7efd180a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient(hunting): picked mon.noname-i con 0x7efd180a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient(hunting): picked mon.noname-b con 0x7efd1813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd180aa1f0 con 0x7efd180a67c0 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd18142b60 con 0x7efd1813f5c0 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd1813a230 con 0x7efd180a6ba0 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1cd1b640 10 monclient(hunting): authenticate will time out at 950.852671s 2024-08-19T09:54:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efd180a67c0 0x7efd1809fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37258/0 (socket says 172.21.15.1:37258) 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.776+0000 7efd1659c640 1 -- 172.21.15.1:0/906171731 learned_addr learned my addr 172.21.15.1:0/906171731 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.778+0000 7efd1659c640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6789/0 conn(0x7efd180a67c0 legacy=0x7efd1809fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.778+0000 7efd0effd640 10 client.?.objecter ms_handle_connect 0x7efd180a67c0 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.778+0000 7efd0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.778+0000 7efd0effd640 10 client.?.objecter ms_handle_connect 0x7efd1813f5c0 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.778+0000 7efd0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.778+0000 7efd16d9d640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efd1813f5c0 legacy=0x7efd181d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3725197999 0 0) 0x7efd10002010 con 0x7efd1813f5c0 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): my global_id is 144182 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcf8004eb0 con 0x7efd1813f5c0 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024230602 0 0) 0x7efd180a92a0 con 0x7efd180a67c0 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): my global_id is 134667 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd10002010 con 0x7efd180a67c0 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 client.?.objecter ms_handle_connect 0x7efd180a6ba0 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0ffff640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.82:6792/0 conn(0x7efd180a6ba0 legacy=0x7efd180a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365647000 0 0) 0x7efd1813a230 con 0x7efd180a6ba0 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): my global_id is 144160 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd180a92a0 con 0x7efd180a6ba0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 604932638 0 0) 0x7efcf8004eb0 con 0x7efd1813f5c0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd1813a230 con 0x7efd1813f5c0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 109032400 0 0) 0x7efd10002010 con 0x7efd180a67c0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcf8004eb0 con 0x7efd180a67c0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd10001ca0 con 0x7efd1813f5c0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): dump: 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.801 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd08001ae0 con 0x7efd180a67c0 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): dump: 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.803 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 820935992 0 0) 0x7efd180a92a0 con 0x7efd180a6ba0 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd10002010 con 0x7efd180a6ba0 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd00001c10 con 0x7efd180a6ba0 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient(hunting): dump: 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2268644386 0 0) 0x7efd1813a230 con 0x7efd1813f5c0 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.82:6792/0 conn(0x7efd180a6ba0 legacy=0x7efd180a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6789/0 conn(0x7efd180a67c0 legacy=0x7efd1809fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 monclient: found mon.b 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1813a7f0 con 0x7efd1813f5c0 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 5 monclient: authenticate success, global_id 144182 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 10 monclient: _renew_subs 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd180a08d0 con 0x7efd1813f5c0 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 10 client.144182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 10 monclient: _renew_subs 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efd1813f1b0 con 0x7efd1813f5c0 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1cd1b640 1 librados: init done 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd10005150 con 0x7efd1813f5c0 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd10005640 con 0x7efd1813f5c0 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd1559a640 4 set_mon_vals no callback set 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.779+0000 7efd0effd640 10 monclient: dump: 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd1cd1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd1cd1b640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efcfc0065c0 con 0x7efcfc002860 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7efd10011aa0 con 0x7efd1813f5c0 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 6284+0+0 (unknown 2534302429 0 0) 0x7efd1004e960 con 0x7efd1813f5c0 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd0effd640 10 client.144182.objecter ms_dispatch 0x7efd18000b90 osd_map(197..197 src has 1..197) 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd0effd640 3 client.144182.objecter handle_osd_map got epochs [197,197] > 0 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd0effd640 3 client.144182.objecter handle_osd_map decoding full epoch 197 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.780+0000 7efd0effd640 20 client.144182.objecter dump_active .. 0 homeless 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.781+0000 7efd0ffff640 10 monclient: get_auth_request con 0x7efcf8041910 auth_method 0 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 10 client.144182.objecter ms_handle_connect 0x7efcfc002860 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 10 client.144182.objecter resend_mon_ops 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd1659c640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efcfc002860 legacy=0x7efcfc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243647732 0 0) 0x7efcfc0065c0 con 0x7efcfc002860 2024-08-19T09:54:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 10 monclient(hunting): my global_id is 144191 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.782+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd1813a230 con 0x7efcfc002860 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 328170631 0 0) 0x7efd1813a230 con 0x7efcfc002860 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcfc0065c0 con 0x7efcfc002860 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd080047e0 con 0x7efcfc002860 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 10 monclient: dump: 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1329018556 0 0) 0x7efcfc0065c0 con 0x7efcfc002860 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efcfc00a160 con 0x7efcfc002860 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 10 client.144182.objecter ms_handle_connect 0x7efcf8041910 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efd08005770 con 0x7efcfc002860 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efcfc002860 legacy=0x7efcfc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.783+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efcfc002860 legacy=0x7efcfc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.789+0000 7efd1cd1b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.789+0000 7efd1cd1b640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efcfc009e50 con 0x7efcfc009110 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 client.144182.objecter ms_handle_connect 0x7efcfc009110 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 client.144182.objecter resend_mon_ops 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd1659c640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efcfc009110 legacy=0x7efcfc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866081887 0 0) 0x7efcfc009e50 con 0x7efcfc009110 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient(hunting): my global_id is 144200 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcfc0065c0 con 0x7efcfc009110 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3498542780 0 0) 0x7efcfc0065c0 con 0x7efcfc009110 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcfc009e50 con 0x7efcfc009110 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd08005ed0 con 0x7efcfc009110 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient: dump: 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 445489193 0 0) 0x7efcfc009e50 con 0x7efcfc009110 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efcfc00c3a0 con 0x7efcfc009110 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 484509180) 0x7efd080063c0 con 0x7efcfc009110 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efcfc009110 legacy=0x7efcfc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.792+0000 7efd0effd640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efcfc009110 legacy=0x7efcfc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19174, 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:54.818 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.819 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:54.820 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.821 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:54.822 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.823 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:54.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.825 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.826 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.827 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.828 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:54.829 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:54.831 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:54.832 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.833 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.834 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.836 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.838 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:54.839 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 10 librados: watch_flush enter 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 10 librados: watch_flush exit 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 20 client.144182.objecter shutdown clearing up homeless session... 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 10 client.144182.objecter successfully canceled tick 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.82:6812/2549104415 conn(0x7efcf8041910 legacy=0x7efcf8043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 10 monclient: shutdown 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 -- 172.21.15.1:0/906171731 >> v1:172.21.15.1:6790/0 conn(0x7efd1813f5c0 legacy=0x7efd181d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd16d9d640 1 -- 172.21.15.1:0/906171731 reap_dead start 2024-08-19T09:54:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd16d9d640 1 -- 172.21.15.1:0/906171731 reap_dead start 2024-08-19T09:54:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 -- 172.21.15.1:0/906171731 shutdown_connections 2024-08-19T09:54:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 -- 172.21.15.1:0/906171731 >> 172.21.15.1:0/906171731 conn(0x7efd180ac4b0 msgr2=0x7efd180a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 -- 172.21.15.1:0/906171731 shutdown_connections 2024-08-19T09:54:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 -- 172.21.15.1:0/906171731 wait complete. 2024-08-19T09:54:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.793+0000 7efd0cff9640 1 librados: shutdown 2024-08-19T09:54:54.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:54.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:54:54.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.941+0000 7fd82634c640 10 monclient: get_monmap_and_config 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.941+0000 7fd82634c640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.941+0000 7fd82634c640 10 monclient: monmap: 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:54.942279+0000 2024-08-19T09:54:54.942 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:54.942279+0000 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.943 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 1 Processor -- start 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 1 -- start start 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient: init 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient: _add_conns ranks=[4,1,5,0,3,6,2,8,7] 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient(hunting): picked mon.noname-c con 0x7fd8200a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient(hunting): picked mon.noname-f con 0x7fd8200a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient(hunting): picked mon.noname-h con 0x7fd82013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8200a3b40 con 0x7fd8200a6ba0 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8200a8130 con 0x7fd8200a67c0 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd82006ede0 con 0x7fd82013f5c0 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd82634c640 10 monclient(hunting): authenticate will time out at 951.018734s 2024-08-19T09:54:54.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd81ffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd8200a67c0 0x7fd8200a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59412/0 (socket says 172.21.15.1:59412) 2024-08-19T09:54:54.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.942+0000 7fd81ffff640 1 -- 172.21.15.1:0/852228609 learned_addr learned my addr 172.21.15.1:0/852228609 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81ffff640 1 -- 172.21.15.1:0/852228609 >> v1:172.21.15.1:6791/0 conn(0x7fd8200a67c0 legacy=0x7fd8200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd8248e9640 1 -- 172.21.15.1:0/852228609 >> v1:172.21.15.82:6791/0 conn(0x7fd82013f5c0 legacy=0x7fd8201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81f7fe640 1 -- 172.21.15.1:0/852228609 >> v1:172.21.15.82:6789/0 conn(0x7fd8200a6ba0 legacy=0x7fd8200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1701213991 0 0) 0x7fd82006ede0 con 0x7fd82013f5c0 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): my global_id is 26681 2024-08-19T09:54:54.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd808003520 con 0x7fd82013f5c0 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130793844 0 0) 0x7fd8200a8130 con 0x7fd8200a67c0 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): my global_id is 26914 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd82006ede0 con 0x7fd8200a67c0 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 32559029 0 0) 0x7fd8200a3b40 con 0x7fd8200a6ba0 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient(hunting): my global_id is 28360 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8200a8130 con 0x7fd8200a6ba0 2024-08-19T09:54:54.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 762423926 0 0) 0x7fd808003520 con 0x7fd82013f5c0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 >> v1:172.21.15.1:6791/0 conn(0x7fd8200a67c0 legacy=0x7fd8200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 >> v1:172.21.15.82:6789/0 conn(0x7fd8200a6ba0 legacy=0x7fd8200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 monclient: found mon.noname-h 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8200a92a0 con 0x7fd82013f5c0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd818001dd0 con 0x7fd82013f5c0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:54:54.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: dump: 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd82634c640 5 monclient: authenticate success, global_id 26681 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd82634c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8180020a0 con 0x7fd82013f5c0 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 1 -- 172.21.15.1:0/852228609 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd818003d90 con 0x7fd82013f5c0 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd82634c640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd82634c640 4 set_mon_vals no callback set 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.945+0000 7fd81effd640 10 monclient: dump: 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.955 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 10 monclient: shutdown 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 1 -- 172.21.15.1:0/852228609 >> v1:172.21.15.82:6791/0 conn(0x7fd82013f5c0 legacy=0x7fd8201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 1 -- 172.21.15.1:0/852228609 shutdown_connections 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 1 -- 172.21.15.1:0/852228609 >> 172.21.15.1:0/852228609 conn(0x7fd8200ac4b0 msgr2=0x7fd8200a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 1 -- 172.21.15.1:0/852228609 shutdown_connections 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 1 -- 172.21.15.1:0/852228609 wait complete. 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 10 monclient: build_initial_monmap 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.946+0000 7fd82634c640 10 monclient: monmap: 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:54.957 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 librados: starting msgr at 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 librados: starting objecter 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 Processor -- start 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 -- start start 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 librados: setting wanted keys 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 librados: calling monclient init 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient: init 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient: _add_conns ranks=[7,3,0,8,1,2,5,6,4] 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient(hunting): picked mon.noname-i con 0x7fd8200a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient(hunting): picked mon.noname-g con 0x7fd8200a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient(hunting): picked mon.noname-a con 0x7fd82013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd82013bc60 con 0x7fd82013f5c0 2024-08-19T09:54:54.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8201395a0 con 0x7fd8200a6ba0 2024-08-19T09:54:54.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8200a9ff0 con 0x7fd8200a67c0 2024-08-19T09:54:54.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:54.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.947+0000 7fd82634c640 10 monclient(hunting): authenticate will time out at 951.023919s 2024-08-19T09:54:54.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.948+0000 7fd8248e9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd82013f5c0 0x7fd8201d64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37272/0 (socket says 172.21.15.1:37272) 2024-08-19T09:54:54.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.948+0000 7fd8248e9640 1 -- 172.21.15.1:0/2905953757 learned_addr learned my addr 172.21.15.1:0/2905953757 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd8248e9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6789/0 conn(0x7fd82013f5c0 legacy=0x7fd8201d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 client.?.objecter ms_handle_connect 0x7fd82013f5c0 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 client.?.objecter ms_handle_connect 0x7fd8200a67c0 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81ffff640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6792/0 conn(0x7fd8200a67c0 legacy=0x7fd820138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 client.?.objecter ms_handle_connect 0x7fd8200a6ba0 2024-08-19T09:54:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81f7fe640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6790/0 conn(0x7fd8200a6ba0 legacy=0x7fd820138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958779081 0 0) 0x7fd82013bc60 con 0x7fd82013f5c0 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): my global_id is 134676 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd80c004eb0 con 0x7fd82013f5c0 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792694322 0 0) 0x7fd8200a9ff0 con 0x7fd8200a67c0 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): my global_id is 144169 2024-08-19T09:54:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd82013bc60 con 0x7fd8200a67c0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992728513 0 0) 0x7fd8201395a0 con 0x7fd8200a6ba0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): my global_id is 134697 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8200a9ff0 con 0x7fd8200a6ba0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 62411126 0 0) 0x7fd80c004eb0 con 0x7fd82013f5c0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8201395a0 con 0x7fd82013f5c0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1707230720 0 0) 0x7fd82013bc60 con 0x7fd8200a67c0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd80c004eb0 con 0x7fd8200a67c0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd818003d90 con 0x7fd82013f5c0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): dump: 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd8100047e0 con 0x7fd8200a67c0 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): dump: 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2305386717 0 0) 0x7fd8200a9ff0 con 0x7fd8200a6ba0 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd82013bc60 con 0x7fd8200a6ba0 2024-08-19T09:54:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd814003610 con 0x7fd8200a6ba0 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient(hunting): dump: 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2046491388 0 0) 0x7fd8201395a0 con 0x7fd82013f5c0 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6792/0 conn(0x7fd8200a67c0 legacy=0x7fd820138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6790/0 conn(0x7fd8200a6ba0 legacy=0x7fd820138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81f7fe640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6790/0 conn(0x7fd8200a6ba0 legacy=0x7fd820138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81f7fe640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6790/0 conn(0x7fd8200a6ba0 legacy=0x7fd820138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81f7fe640 1 --1- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6790/0 conn(0x7fd8200a6ba0 0x7fd820138e90 :-1 s=OPENED pgs=209 cs=1 l=1).handle_message read tag failed 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81f7fe640 1 --1- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6790/0 conn(0x7fd8200a6ba0 0x7fd820138e90 :-1 s=OPENED pgs=209 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:54:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: _finish_hunting 0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 monclient: found mon.a 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8200a9bc0 con 0x7fd82013f5c0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: _finish_auth 0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: _check_auth_tickets 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: ms_handle_reset stray mon v1:172.21.15.82:6790/0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 5 monclient: authenticate success, global_id 134676 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 10 monclient: _renew_subs 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd820139900 con 0x7fd82013f5c0 2024-08-19T09:54:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 10 client.134676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 10 monclient: _renew_subs 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd82013a8a0 con 0x7fd82013f5c0 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd82634c640 1 librados: init done 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8180042e0 con 0x7fd82013f5c0 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd8180047d0 con 0x7fd82013f5c0 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81e7fc640 4 set_mon_vals no callback set 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.950+0000 7fd81cff9640 10 monclient: dump: 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.951+0000 7fd82634c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.951+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd8000065c0 con 0x7fd800002860 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.951+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd818011e00 con 0x7fd82013f5c0 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.952+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(197..197 src has 1..197) ==== 6284+0+0 (unknown 2534302429 0 0) 0x7fd81804e790 con 0x7fd82013f5c0 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.952+0000 7fd81cff9640 10 client.134676.objecter ms_dispatch 0x7fd820000b90 osd_map(197..197 src has 1..197) 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.952+0000 7fd81cff9640 3 client.134676.objecter handle_osd_map got epochs [197,197] > 0 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.952+0000 7fd81cff9640 3 client.134676.objecter handle_osd_map decoding full epoch 197 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.952+0000 7fd81cff9640 20 client.134676.objecter dump_active .. 0 homeless 2024-08-19T09:54:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.952+0000 7fd81f7fe640 10 monclient: get_auth_request con 0x7fd80c041930 auth_method 0 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.953+0000 7fd81ffff640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6791/0 conn(0x7fd800002860 legacy=0x7fd800004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.953+0000 7fd81cff9640 10 client.134676.objecter ms_handle_connect 0x7fd800002860 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.953+0000 7fd81cff9640 10 client.134676.objecter resend_mon_ops 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721406799 0 0) 0x7fd8000065c0 con 0x7fd800002860 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient(hunting): my global_id is 26923 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8201395a0 con 0x7fd800002860 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1239539706 0 0) 0x7fd8201395a0 con 0x7fd800002860 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8000065c0 con 0x7fd800002860 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd810004eb0 con 0x7fd800002860 2024-08-19T09:54:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient: dump: 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1088545540 0 0) 0x7fd8000065c0 con 0x7fd800002860 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd80000a160 con 0x7fd800002860 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 client.134676.objecter ms_handle_connect 0x7fd80c041930 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd8100063f0 con 0x7fd800002860 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6791/0 conn(0x7fd800002860 legacy=0x7fd800004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.954+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6791/0 conn(0x7fd800002860 legacy=0x7fd800004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.960+0000 7fd82634c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.960+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd800009e50 con 0x7fd800009110 2024-08-19T09:54:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81ffff640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6791/0 conn(0x7fd800009110 legacy=0x7fd8000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 client.134676.objecter ms_handle_connect 0x7fd800009110 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 client.134676.objecter resend_mon_ops 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864901449 0 0) 0x7fd800009e50 con 0x7fd800009110 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient(hunting): my global_id is 26932 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8000065c0 con 0x7fd800009110 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1945580776 0 0) 0x7fd8000065c0 con 0x7fd800009110 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd800009e50 con 0x7fd800009110 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd810005460 con 0x7fd800009110 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient: dump: 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:54.985 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2635270639 0 0) 0x7fd800009e50 con 0x7fd800009110 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd80000c3a0 con 0x7fd800009110 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1198879357) 0x7fd810005830 con 0x7fd800009110 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6791/0 conn(0x7fd800009110 legacy=0x7fd8000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.963+0000 7fd81cff9640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6791/0 conn(0x7fd800009110 legacy=0x7fd8000095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 301235, 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:54.987 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.989 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.990 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:54.991 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:54.992 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.994 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:54.995 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.996 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.997 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:54.999 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.005 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:55.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:55.007 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 10 librados: watch_flush enter 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 10 librados: watch_flush exit 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 20 client.134676.objecter shutdown clearing up homeless session... 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 10 client.134676.objecter successfully canceled tick 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd80c041930 legacy=0x7fd80c043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 10 monclient: shutdown 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 >> v1:172.21.15.1:6789/0 conn(0x7fd82013f5c0 legacy=0x7fd8201d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd8248e9640 1 -- 172.21.15.1:0/2905953757 reap_dead start 2024-08-19T09:54:55.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd8248e9640 1 -- 172.21.15.1:0/2905953757 reap_dead start 2024-08-19T09:54:55.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 shutdown_connections 2024-08-19T09:54:55.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 >> 172.21.15.1:0/2905953757 conn(0x7fd8200ac4b0 msgr2=0x7fd82013eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 shutdown_connections 2024-08-19T09:54:55.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 -- 172.21.15.1:0/2905953757 wait complete. 2024-08-19T09:54:55.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:54.964+0000 7fd82634c640 1 librados: shutdown 2024-08-19T09:54:55.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:55.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:54:55.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:54:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.109+0000 7f8846056640 10 monclient: get_monmap_and_config 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.109+0000 7f8846056640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.109+0000 7f8846056640 10 monclient: monmap: 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:55.110655+0000 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:55.110655+0000 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.112 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.110+0000 7f8846056640 1 Processor -- start 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.110+0000 7f8846056640 1 -- start start 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.110+0000 7f8846056640 10 monclient: init 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.110+0000 7f8846056640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.110+0000 7f8846056640 10 monclient: _add_conns ranks=[5,1,2,0,7,8,6,3,4] 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 10 monclient(hunting): picked mon.noname-h con 0x7f88400a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 10 monclient(hunting): picked mon.noname-f con 0x7f88400a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 10 monclient(hunting): picked mon.noname-b con 0x7f884013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88400a3b40 con 0x7f88400a6ba0 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88400a8130 con 0x7f884013f5c0 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f884006ede0 con 0x7f88400a67c0 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f8846056640 10 monclient(hunting): authenticate will time out at 951.187088s 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f883ffff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f884013f5c0 0x7f88401419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34882/0 (socket says 172.21.15.1:34882) 2024-08-19T09:54:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.111+0000 7f883ffff640 1 -- 172.21.15.1:0/3840322138 learned_addr learned my addr 172.21.15.1:0/3840322138 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883ffff640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.1:6790/0 conn(0x7f884013f5c0 legacy=0x7f88401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883f7fe640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6791/0 conn(0x7f88400a67c0 legacy=0x7f88400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883effd640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6789/0 conn(0x7f88400a6ba0 legacy=0x7f88400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958282221 0 0) 0x7f88400a8130 con 0x7f884013f5c0 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): my global_id is 144209 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8828003520 con 0x7f884013f5c0 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883effd640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6789/0 conn(0x7f88400a6ba0 legacy=0x7f88400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883effd640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6789/0 conn(0x7f88400a6ba0 legacy=0x7f88400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883effd640 1 --1- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6789/0 conn(0x7f88400a6ba0 0x7f88400a6fb0 :-1 s=OPENED pgs=3361 cs=1 l=1).handle_message read tag failed 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883effd640 1 --1- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6789/0 conn(0x7f88400a6ba0 0x7f88400a6fb0 :-1 s=OPENED pgs=3361 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:54:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6789/0 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2875347025 0 0) 0x7f884006ede0 con 0x7f88400a67c0 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient(hunting): my global_id is 26690 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88400a8130 con 0x7f88400a67c0 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3376738337 0 0) 0x7f8828003520 con 0x7f884013f5c0 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6791/0 conn(0x7f88400a67c0 legacy=0x7f88400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.82:6789/0 conn(0x7f88400a6ba0 legacy=0x7f88400a6fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 monclient: found mon.noname-b 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88400a92a0 con 0x7f884013f5c0 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8838001e60 con 0x7f884013f5c0 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f8846056640 5 monclient: authenticate success, global_id 144209 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f8846056640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f883e7fc640 10 monclient: dump: 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.122 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.113+0000 7f8846056640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8838004c20 con 0x7f884013f5c0 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f883e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f883e7fc640 1 -- 172.21.15.1:0/3840322138 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8838005190 con 0x7f884013f5c0 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f883e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f883e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f883e7fc640 10 monclient: dump: 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 4 set_mon_vals no callback set 2024-08-19T09:54:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 10 monclient: shutdown 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 1 -- 172.21.15.1:0/3840322138 >> v1:172.21.15.1:6790/0 conn(0x7f884013f5c0 legacy=0x7f88401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 1 -- 172.21.15.1:0/3840322138 shutdown_connections 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 1 -- 172.21.15.1:0/3840322138 >> 172.21.15.1:0/3840322138 conn(0x7f88400ac4b0 msgr2=0x7f88400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 1 -- 172.21.15.1:0/3840322138 shutdown_connections 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.114+0000 7f8846056640 1 -- 172.21.15.1:0/3840322138 wait complete. 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 10 monclient: monmap: 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 1 librados: starting msgr at 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 1 librados: starting objecter 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 1 Processor -- start 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 1 -- start start 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 1 librados: setting wanted keys 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 1 librados: calling monclient init 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.115+0000 7f8846056640 10 monclient: init 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient: _add_conns ranks=[6,4,2,5,1,3,7,8,0] 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient(hunting): picked mon.noname-d con 0x7f88400a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient(hunting): picked mon.noname-c con 0x7f88400a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient(hunting): picked mon.noname-b con 0x7f884013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88400a1ab0 con 0x7f884013f5c0 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88400a2640 con 0x7f88400a6ba0 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88401cf020 con 0x7f88400a67c0 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f8846056640 10 monclient(hunting): authenticate will time out at 951.192362s 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f883f7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f88400a67c0 0x7f88401ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52870/0 (socket says 172.21.15.1:52870) 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f883effd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f88400a6ba0 0x7f88401ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59436/0 (socket says 172.21.15.1:59436) 2024-08-19T09:54:55.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.116+0000 7f883f7fe640 1 -- 172.21.15.1:0/2468302449 learned_addr learned my addr 172.21.15.1:0/2468302449 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f883effd640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6791/0 conn(0x7f88400a6ba0 legacy=0x7f88401ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f883ffff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6790/0 conn(0x7f884013f5c0 legacy=0x7f88401d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 client.?.objecter ms_handle_connect 0x7f88400a6ba0 2024-08-19T09:54:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 client.?.objecter ms_handle_connect 0x7f884013f5c0 2024-08-19T09:54:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f883f7fe640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f88400a67c0 legacy=0x7f88401ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 client.?.objecter ms_handle_connect 0x7f88400a67c0 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876855203 0 0) 0x7f88400a1ab0 con 0x7f884013f5c0 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): my global_id is 144218 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882c004ee0 con 0x7f884013f5c0 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685982557 0 0) 0x7f88400a2640 con 0x7f88400a6ba0 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): my global_id is 26941 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88400a1ab0 con 0x7f88400a6ba0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716581261 0 0) 0x7f88401cf020 con 0x7f88400a67c0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 10 monclient(hunting): my global_id is 27060 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.118+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88400a2640 con 0x7f88400a67c0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2314521465 0 0) 0x7f88400a2640 con 0x7f88400a67c0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88401cf020 con 0x7f88400a67c0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1519639261 0 0) 0x7f88400a1ab0 con 0x7f88400a6ba0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88400a2640 con 0x7f88400a6ba0 2024-08-19T09:54:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4020603357 0 0) 0x7f882c004ee0 con 0x7f884013f5c0 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88400a1ab0 con 0x7f884013f5c0 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8830003610 con 0x7f88400a67c0 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): dump: 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.138 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8834003610 con 0x7f88400a6ba0 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): dump: 2024-08-19T09:54:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f88380020b0 con 0x7f884013f5c0 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient(hunting): dump: 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1584507327 0 0) 0x7f88401cf020 con 0x7f88400a67c0 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6791/0 conn(0x7f88400a6ba0 legacy=0x7f88401ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6790/0 conn(0x7f884013f5c0 legacy=0x7f88401d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 monclient: found mon.d 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88400a9bc0 con 0x7f88400a67c0 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 5 monclient: authenticate success, global_id 27060 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 10 monclient: _renew_subs 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88400a2360 con 0x7f88400a67c0 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 10 client.27060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:55.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 10 monclient: _renew_subs 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f88400a1120 con 0x7f88400a67c0 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8846056640 1 librados: init done 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8830003b00 con 0x7f88400a67c0 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8830003ff0 con 0x7f88400a67c0 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f883dffb640 4 set_mon_vals no callback set 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.119+0000 7f8827fff640 10 monclient: dump: 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.144 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.145 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8830004190 con 0x7f88400a67c0 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8846056640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88200065c0 con 0x7f8820002860 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(197..197 src has 1..197) ==== 6284+0+0 (unknown 2534302429 0 0) 0x7f883004e4d0 con 0x7f88400a67c0 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8827fff640 10 client.27060.objecter ms_dispatch 0x7f8840000b90 osd_map(197..197 src has 1..197) 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8827fff640 3 client.27060.objecter handle_osd_map got epochs [197,197] > 0 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8827fff640 3 client.27060.objecter handle_osd_map decoding full epoch 197 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f8827fff640 20 client.27060.objecter dump_active .. 0 homeless 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.120+0000 7f883ffff640 10 monclient: get_auth_request con 0x7f882c041a70 auth_method 0 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f883effd640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f8820002860 legacy=0x7f8820004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 client.27060.objecter ms_handle_connect 0x7f8820002860 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 client.27060.objecter resend_mon_ops 2024-08-19T09:54:55.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583071929 0 0) 0x7f88200065c0 con 0x7f8820002860 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 monclient(hunting): my global_id is 27069 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88401cf020 con 0x7f8820002860 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2627807073 0 0) 0x7f88401cf020 con 0x7f8820002860 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88200065c0 con 0x7f8820002860 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8834004150 con 0x7f8820002860 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.122+0000 7f8827fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 10 monclient: dump: 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.147 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 10 client.27060.objecter ms_handle_connect 0x7f882c041a70 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1873850219 0 0) 0x7f88200065c0 con 0x7f8820002860 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f882000a160 con 0x7f8820002860 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f88340043f0 con 0x7f8820002860 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f8820002860 legacy=0x7f8820004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.123+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f8820002860 legacy=0x7f8820004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.129+0000 7f8846056640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.129+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8820009e50 con 0x7f8820009110 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.131+0000 7f883effd640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f8820009110 legacy=0x7f88200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.131+0000 7f8827fff640 10 client.27060.objecter ms_handle_connect 0x7f8820009110 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.131+0000 7f8827fff640 10 client.27060.objecter resend_mon_ops 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062866180 0 0) 0x7f8820009e50 con 0x7f8820009110 2024-08-19T09:54:55.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient(hunting): my global_id is 27078 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88200065c0 con 0x7f8820009110 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1766078301 0 0) 0x7f88200065c0 con 0x7f8820009110 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8820009e50 con 0x7f8820009110 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8834005d30 con 0x7f8820009110 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient: dump: 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.150 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 488172720 0 0) 0x7f8820009e50 con 0x7f8820009110 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f882000c3f0 con 0x7f8820009110 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 53778444) 0x7f8834008090 con 0x7f8820009110 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f8820009110 legacy=0x7f88200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8827fff640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f8820009110 legacy=0x7f88200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:55.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 301389, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:55.153 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.154 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.155 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:55.156 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.157 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.159 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:55.160 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.167 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.171 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:55.172 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:55.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8846056640 10 librados: watch_flush enter 2024-08-19T09:54:55.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.132+0000 7f8846056640 10 librados: watch_flush exit 2024-08-19T09:54:55.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 20 client.27060.objecter shutdown clearing up homeless session... 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 10 client.27060.objecter successfully canceled tick 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.82:6812/2549104415 conn(0x7f882c041a70 legacy=0x7f882c043f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 10 monclient: shutdown 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 >> v1:172.21.15.1:6792/0 conn(0x7f88400a67c0 legacy=0x7f88401ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f883ffff640 1 -- 172.21.15.1:0/2468302449 reap_dead start 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f883ffff640 1 -- 172.21.15.1:0/2468302449 reap_dead start 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 shutdown_connections 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 >> 172.21.15.1:0/2468302449 conn(0x7f88400ac4b0 msgr2=0x7f88400a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 shutdown_connections 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 -- 172.21.15.1:0/2468302449 wait complete. 2024-08-19T09:54:55.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.133+0000 7f8846056640 1 librados: shutdown 2024-08-19T09:54:55.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:55.176 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:54:55.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:54:55.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.274+0000 7ff73173a640 10 monclient: get_monmap_and_config 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.274+0000 7ff73173a640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.274+0000 7ff73173a640 10 monclient: monmap: 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:55.275748+0000 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:55.275748+0000 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.277 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.277 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.277 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.277 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.277 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.277 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.275+0000 7ff73173a640 1 Processor -- start 2024-08-19T09:54:55.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.275+0000 7ff73173a640 1 -- start start 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.275+0000 7ff73173a640 10 monclient: init 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.275+0000 7ff73173a640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.275+0000 7ff73173a640 10 monclient: _add_conns ranks=[0,6,1,8,7,3,4,5,2] 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 10 monclient(hunting): picked mon.noname-a con 0x7ff72c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 10 monclient(hunting): picked mon.noname-d con 0x7ff72c0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 10 monclient(hunting): picked mon.noname-f con 0x7ff72c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72c0a3b40 con 0x7ff72c0a67c0 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72c0a8130 con 0x7ff72c13f5c0 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72c06ede0 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff73173a640 10 monclient(hunting): authenticate will time out at 951.352106s 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff72affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff72c0a67c0 0x7ff72c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37284/0 (socket says 172.21.15.1:37284) 2024-08-19T09:54:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff72affd640 1 -- 172.21.15.1:0/2043721812 learned_addr learned my addr 172.21.15.1:0/2043721812 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.276+0000 7ff72a7fc640 1 --1- 172.21.15.1:0/2043721812 >> v1:172.21.15.1:6792/0 conn(0x7ff72c0a6ba0 0x7ff72c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52894/0 (socket says 172.21.15.1:52894) 2024-08-19T09:54:55.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff72affd640 1 -- 172.21.15.1:0/2043721812 >> v1:172.21.15.1:6789/0 conn(0x7ff72c0a67c0 legacy=0x7ff72c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff72a7fc640 1 -- 172.21.15.1:0/2043721812 >> v1:172.21.15.1:6792/0 conn(0x7ff72c0a6ba0 legacy=0x7ff72c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff72b7fe640 1 -- 172.21.15.1:0/2043721812 >> v1:172.21.15.82:6789/0 conn(0x7ff72c13f5c0 legacy=0x7ff72c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449538360 0 0) 0x7ff72c0a3b40 con 0x7ff72c0a67c0 2024-08-19T09:54:55.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): my global_id is 134685 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff718003520 con 0x7ff72c0a67c0 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244159843 0 0) 0x7ff72c06ede0 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): my global_id is 27087 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff72c0a3b40 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509757648 0 0) 0x7ff72c0a8130 con 0x7ff72c13f5c0 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient(hunting): my global_id is 28369 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff72c06ede0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1444249220 0 0) 0x7ff72c0a3b40 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 >> v1:172.21.15.82:6789/0 conn(0x7ff72c13f5c0 legacy=0x7ff72c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 >> v1:172.21.15.1:6789/0 conn(0x7ff72c0a67c0 legacy=0x7ff72c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 monclient: found mon.noname-d 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72c0a92a0 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff73173a640 5 monclient: authenticate success, global_id 27087 2024-08-19T09:54:55.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff714001a50 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff73173a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: dump: 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.290 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff73173a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff714001d20 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.278+0000 7ff729ffb640 1 -- 172.21.15.1:0/2043721812 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff714004b20 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 4 set_mon_vals no callback set 2024-08-19T09:54:55.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff729ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff729ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff729ffb640 10 monclient: dump: 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.292 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 10 monclient: shutdown 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 1 -- 172.21.15.1:0/2043721812 >> v1:172.21.15.1:6792/0 conn(0x7ff72c0a6ba0 legacy=0x7ff72c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 1 -- 172.21.15.1:0/2043721812 shutdown_connections 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 1 -- 172.21.15.1:0/2043721812 >> 172.21.15.1:0/2043721812 conn(0x7ff72c0ac4b0 msgr2=0x7ff72c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 1 -- 172.21.15.1:0/2043721812 shutdown_connections 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.279+0000 7ff73173a640 1 -- 172.21.15.1:0/2043721812 wait complete. 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient: monmap: 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 1 librados: starting msgr at 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 1 librados: starting objecter 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 1 Processor -- start 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 1 -- start start 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 1 librados: setting wanted keys 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 1 librados: calling monclient init 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient: init 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient: _add_conns ranks=[4,6,0,3,2,1,8,5,7] 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient(hunting): picked mon.noname-c con 0x7ff72c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:55.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.280+0000 7ff73173a640 10 monclient(hunting): picked mon.noname-d con 0x7ff72c0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff73173a640 10 monclient(hunting): picked mon.noname-a con 0x7ff72c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff73173a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72c139ee0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff73173a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72c13ace0 con 0x7ff72c0a67c0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff73173a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72c0a0f40 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff73173a640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff73173a640 10 monclient(hunting): authenticate will time out at 951.357042s 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff72a7fc640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ff72c0a6ba0 0x7ff72c0a0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52896/0 (socket says 172.21.15.1:52896) 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.281+0000 7ff72a7fc640 1 -- 172.21.15.1:0/1365991892 learned_addr learned my addr 172.21.15.1:0/1365991892 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff72affd640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6791/0 conn(0x7ff72c0a67c0 legacy=0x7ff72c0a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 client.?.objecter ms_handle_connect 0x7ff72c0a67c0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 client.?.objecter ms_handle_connect 0x7ff72c0a6ba0 2024-08-19T09:54:55.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 client.?.objecter ms_handle_connect 0x7ff72c13f5c0 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff72b7fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6789/0 conn(0x7ff72c13f5c0 legacy=0x7ff72c1d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff72a7fc640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6792/0 conn(0x7ff72c0a6ba0 legacy=0x7ff72c0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 707866865 0 0) 0x7ff72c139ee0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): my global_id is 134694 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff70c004eb0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1425368227 0 0) 0x7ff72c0a0f40 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): my global_id is 27096 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff72c139ee0 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728908173 0 0) 0x7ff72c13ace0 con 0x7ff72c0a67c0 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): my global_id is 26950 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff72c0a0f40 con 0x7ff72c0a67c0 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4152302364 0 0) 0x7ff70c004eb0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff72c13ace0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff724002070 con 0x7ff72c13f5c0 2024-08-19T09:54:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): dump: 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.299 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3011330881 0 0) 0x7ff72c139ee0 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff70c004eb0 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2450030707 0 0) 0x7ff72c0a0f40 con 0x7ff72c0a67c0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff72c139ee0 con 0x7ff72c0a67c0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff714004b20 con 0x7ff72c0a6ba0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): dump: 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff71c0019e0 con 0x7ff72c0a67c0 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient(hunting): dump: 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1631408916 0 0) 0x7ff72c13ace0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6792/0 conn(0x7ff72c0a6ba0 legacy=0x7ff72c0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6791/0 conn(0x7ff72c0a67c0 legacy=0x7ff72c0a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.283+0000 7ff7237fe640 1 monclient: found mon.a 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72c142bf0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 5 monclient: authenticate success, global_id 134694 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 10 monclient: _renew_subs 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff72c0a1ba0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 10 client.134694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 10 monclient: _renew_subs 2024-08-19T09:54:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff72c0a2a00 con 0x7ff72c13f5c0 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 1 librados: init done 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff724003db0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff7240042a0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7297fa640 4 set_mon_vals no callback set 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff7237fe640 10 monclient: dump: 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.305 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.284+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff7100065c0 con 0x7ff710002860 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff724004440 con 0x7ff72c13f5c0 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(197..197 src has 1..197) ==== 6284+0+0 (unknown 2534302429 0 0) 0x7ff72404e7b0 con 0x7ff72c13f5c0 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff7237fe640 10 client.134694.objecter ms_dispatch 0x7ff72c000b90 osd_map(197..197 src has 1..197) 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff7237fe640 3 client.134694.objecter handle_osd_map got epochs [197,197] > 0 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff7237fe640 3 client.134694.objecter handle_osd_map decoding full epoch 197 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff7237fe640 20 client.134694.objecter dump_active .. 0 homeless 2024-08-19T09:54:55.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.285+0000 7ff72a7fc640 10 monclient: get_auth_request con 0x7ff70c0419a0 auth_method 0 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff72affd640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6793/0 conn(0x7ff710002860 legacy=0x7ff710004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 client.134694.objecter ms_handle_connect 0x7ff710002860 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 client.134694.objecter resend_mon_ops 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922245614 0 0) 0x7ff7100065c0 con 0x7ff710002860 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient(hunting): my global_id is 27512 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff72c13ace0 con 0x7ff710002860 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2265810240 0 0) 0x7ff72c13ace0 con 0x7ff710002860 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7100065c0 con 0x7ff710002860 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff71c005050 con 0x7ff710002860 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient: dump: 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1449932383 0 0) 0x7ff7100065c0 con 0x7ff710002860 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff71000a160 con 0x7ff710002860 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 client.134694.objecter ms_handle_connect 0x7ff70c0419a0 2024-08-19T09:54:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff71c005050 con 0x7ff710002860 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6793/0 conn(0x7ff710002860 legacy=0x7ff710004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.287+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6793/0 conn(0x7ff710002860 legacy=0x7ff710004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.294+0000 7ff73173a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.294+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff710009e70 con 0x7ff710009130 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff72affd640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6793/0 conn(0x7ff710009130 legacy=0x7ff7100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 client.134694.objecter ms_handle_connect 0x7ff710009130 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 client.134694.objecter resend_mon_ops 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632170599 0 0) 0x7ff710009e70 con 0x7ff710009130 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient(hunting): my global_id is 27521 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7100065c0 con 0x7ff710009130 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1821663829 0 0) 0x7ff7100065c0 con 0x7ff710009130 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff710009e70 con 0x7ff710009130 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff71c005840 con 0x7ff710009130 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 10 monclient: dump: 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4193755240 0 0) 0x7ff710009e70 con 0x7ff710009130 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.296+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff71000c410 con 0x7ff710009130 2024-08-19T09:54:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3777492669) 0x7ff71c007da0 con 0x7ff710009130 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff7237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff7237fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6793/0 conn(0x7ff710009130 legacy=0x7ff7100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff7237fe640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6793/0 conn(0x7ff710009130 legacy=0x7ff7100095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 301570, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:55.317 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.318 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.319 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.320 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.321 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:55.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:55.324 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:55.325 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.326 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:55.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 10 librados: watch_flush enter 2024-08-19T09:54:55.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 10 librados: watch_flush exit 2024-08-19T09:54:55.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 20 client.134694.objecter shutdown clearing up homeless session... 2024-08-19T09:54:55.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 10 client.134694.objecter successfully canceled tick 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff70c0419a0 legacy=0x7ff70c043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 10 monclient: shutdown 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 >> v1:172.21.15.1:6789/0 conn(0x7ff72c13f5c0 legacy=0x7ff72c1d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff72b7fe640 1 -- 172.21.15.1:0/1365991892 reap_dead start 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff72b7fe640 1 -- 172.21.15.1:0/1365991892 reap_dead start 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 shutdown_connections 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 >> 172.21.15.1:0/1365991892 conn(0x7ff72c0ac4b0 msgr2=0x7ff72c0a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 shutdown_connections 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 -- 172.21.15.1:0/1365991892 wait complete. 2024-08-19T09:54:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.297+0000 7ff73173a640 1 librados: shutdown 2024-08-19T09:54:55.340 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:55.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:54:55.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.438+0000 7f80010b3640 10 monclient: get_monmap_and_config 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.438+0000 7f80010b3640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.438+0000 7f80010b3640 10 monclient: monmap: 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:55.439150+0000 2024-08-19T09:54:55.439 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:55.439150+0000 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.440 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.438+0000 7f80010b3640 1 Processor -- start 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 1 -- start start 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient: init 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient: _add_conns ranks=[6,0,1,4,2,5,3,8,7] 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient(hunting): picked mon.noname-d con 0x7f7ffc0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient(hunting): picked mon.noname-a con 0x7f7ffc0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient(hunting): picked mon.noname-f con 0x7f7ffc13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ffc0a3b40 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ffc0a8130 con 0x7f7ffc13f5c0 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ffc06ede0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f80010b3640 10 monclient(hunting): authenticate will time out at 951.515574s 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f7ff3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7ffc0a6ba0 0x7f7ffc0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37292/0 (socket says 172.21.15.1:37292) 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f7ff3fff640 1 -- 172.21.15.1:0/1742471331 learned_addr learned my addr 172.21.15.1:0/1742471331 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.439+0000 7f7ffad9d640 1 --1- 172.21.15.1:0/1742471331 >> v1:172.21.15.1:6792/0 conn(0x7f7ffc0a67c0 0x7f7ffc0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52912/0 (socket says 172.21.15.1:52912) 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.441+0000 7f7ffad9d640 1 -- 172.21.15.1:0/1742471331 >> v1:172.21.15.1:6792/0 conn(0x7f7ffc0a67c0 legacy=0x7f7ffc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.441+0000 7f7ff3fff640 1 -- 172.21.15.1:0/1742471331 >> v1:172.21.15.1:6789/0 conn(0x7f7ffc0a6ba0 legacy=0x7f7ffc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1296416440 0 0) 0x7f7ffc06ede0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): my global_id is 27105 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffb59e640 1 -- 172.21.15.1:0/1742471331 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc13f5c0 legacy=0x7f7ffc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fe8003520 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243415326 0 0) 0x7f7ffc0a3b40 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient(hunting): my global_id is 134703 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ffc06ede0 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 398018951 0 0) 0x7f7fe8003520 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc13f5c0 legacy=0x7f7ffc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 >> v1:172.21.15.1:6789/0 conn(0x7f7ffc0a6ba0 legacy=0x7f7ffc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 monclient: found mon.noname-d 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ffc0a92a0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fec001ad0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f80010b3640 5 monclient: authenticate success, global_id 27105 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f80010b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: dump: 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.449 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f80010b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fec0047e0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 1 -- 172.21.15.1:0/1742471331 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fec004d10 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f80010b3640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f80010b3640 4 set_mon_vals no callback set 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.442+0000 7f7ffa59c640 10 monclient: dump: 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 10 monclient: shutdown 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 1 -- 172.21.15.1:0/1742471331 >> v1:172.21.15.1:6792/0 conn(0x7f7ffc0a67c0 legacy=0x7f7ffc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 1 -- 172.21.15.1:0/1742471331 shutdown_connections 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 1 -- 172.21.15.1:0/1742471331 >> 172.21.15.1:0/1742471331 conn(0x7f7ffc0ac4b0 msgr2=0x7f7ffc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 1 -- 172.21.15.1:0/1742471331 shutdown_connections 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 1 -- 172.21.15.1:0/1742471331 wait complete. 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.443+0000 7f80010b3640 10 monclient: monmap: 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 librados: starting msgr at 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 librados: starting objecter 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 Processor -- start 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 -- start start 2024-08-19T09:54:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 librados: setting wanted keys 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 librados: calling monclient init 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient: init 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient: _add_conns ranks=[8,2,7,6,3,4,1,0,5] 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient(hunting): picked mon.noname-e con 0x7f7ffc0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient(hunting): picked mon.noname-b con 0x7f7ffc0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient(hunting): picked mon.noname-i con 0x7f7ffc13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ffc139ce0 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ffc13a830 con 0x7f7ffc13f5c0 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ffc13f310 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f80010b3640 10 monclient(hunting): authenticate will time out at 951.520652s 2024-08-19T09:54:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f7ffad9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f7ffc0a67c0 0x7f7ffc1ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:40686/0 (socket says 172.21.15.1:40686) 2024-08-19T09:54:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.444+0000 7f7ffad9d640 1 -- 172.21.15.1:0/400776583 learned_addr learned my addr 172.21.15.1:0/400776583 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.446+0000 7f7ffad9d640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.1:6793/0 conn(0x7f7ffc0a67c0 legacy=0x7f7ffc1ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.446+0000 7f7ff3fff640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.1:6790/0 conn(0x7f7ffc0a6ba0 legacy=0x7f7ffc1d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.446+0000 7f7ff37fe640 10 client.?.objecter ms_handle_connect 0x7f7ffc0a67c0 2024-08-19T09:54:55.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.446+0000 7f7ff37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.446+0000 7f7ff37fe640 10 client.?.objecter ms_handle_connect 0x7f7ffc0a6ba0 2024-08-19T09:54:55.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.446+0000 7f7ff37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616550097 0 0) 0x7f7ffc13f310 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): my global_id is 27530 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fdc004eb0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 client.?.objecter ms_handle_connect 0x7f7ffc13f5c0 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ffb59e640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6792/0 conn(0x7f7ffc13f5c0 legacy=0x7f7ffc1d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336378885 0 0) 0x7f7ffc139ce0 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): my global_id is 144227 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ffc13f310 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2129592919 0 0) 0x7f7fdc004eb0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ffc139ce0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fec004cd0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): dump: 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:55.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2023603952 0 0) 0x7f7ffc13f310 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fdc004eb0 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe4001ce0 con 0x7f7ffc0a6ba0 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): dump: 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476960624 0 0) 0x7f7ffc13a830 con 0x7f7ffc13f5c0 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient(hunting): my global_id is 144178 2024-08-19T09:54:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ffc13f310 con 0x7f7ffc13f5c0 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1658939233 0 0) 0x7f7ffc139ce0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6792/0 conn(0x7f7ffc13f5c0 legacy=0x7f7ffc1d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.1:6790/0 conn(0x7f7ffc0a6ba0 legacy=0x7f7ffc1d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 monclient: found mon.e 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ffc0a9d30 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 5 monclient: authenticate success, global_id 27530 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 10 monclient: _renew_subs 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ffc142bf0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 10 client.27530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 10 monclient: _renew_subs 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ffc0a0d30 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f80010b3640 1 librados: init done 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fec001d20 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fec005640 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff37fe640 10 monclient: dump: 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.447+0000 7f7ff9d9b640 4 set_mon_vals no callback set 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f80010b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ffc139fb0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7fec011aa0 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(197..197 src has 1..197) ==== 6284+0+0 (unknown 2534302429 0 0) 0x7f7fec04e680 con 0x7f7ffc0a67c0 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ff37fe640 10 client.27530.objecter ms_dispatch 0x7f7ffc000b90 osd_map(197..197 src has 1..197) 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ff37fe640 3 client.27530.objecter handle_osd_map got epochs [197,197] > 0 2024-08-19T09:54:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ff37fe640 3 client.27530.objecter handle_osd_map decoding full epoch 197 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ff37fe640 20 client.27530.objecter dump_active .. 0 homeless 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.448+0000 7f7ffb59e640 10 monclient: get_auth_request con 0x7f7fdc041760 auth_method 0 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 10 client.27530.objecter ms_handle_connect 0x7f7ffc1cebe0 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 10 client.27530.objecter resend_mon_ops 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff3fff640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc1cebe0 legacy=0x7f7ffc13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207730344 0 0) 0x7f7ffc139fb0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 10 monclient(hunting): my global_id is 28387 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.450+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ffc139ce0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4218245479 0 0) 0x7f7ffc139ce0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ffc139fb0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe4004da0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 10 monclient: dump: 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 10 client.27530.objecter ms_handle_connect 0x7f7fdc041760 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2403055251 0 0) 0x7f7ffc139fb0 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ffc0a1740 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7fe4006260 con 0x7f7ffc1cebe0 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:55.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc1cebe0 legacy=0x7f7ffc13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.451+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc1cebe0 legacy=0x7f7ffc13fd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.457+0000 7f80010b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.457+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ffc1cefc0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff3fff640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc02e1d0 legacy=0x7f7ffc1404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 client.27530.objecter ms_handle_connect 0x7f7ffc02e1d0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 client.27530.objecter resend_mon_ops 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408041209 0 0) 0x7f7ffc1cefc0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient(hunting): my global_id is 28396 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ffc139fb0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2651161495 0 0) 0x7f7ffc139fb0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ffc1cefc0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe4006a00 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.460+0000 7f7ff37fe640 10 monclient: dump: 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1805875692 0 0) 0x7f7ffc1cefc0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ffc068f60 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 1235382317) 0x7f7fe4006bb0 con 0x7f7ffc02e1d0 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc02e1d0 legacy=0x7f7ffc1404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ff37fe640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6789/0 conn(0x7f7ffc02e1d0 legacy=0x7f7ffc1404a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 301721, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:55.497 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:55.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 10 librados: watch_flush enter 2024-08-19T09:54:55.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 10 librados: watch_flush exit 2024-08-19T09:54:55.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 20 client.27530.objecter shutdown clearing up homeless session... 2024-08-19T09:54:55.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 10 client.27530.objecter successfully canceled tick 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7fdc041760 legacy=0x7f7fdc043c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 10 monclient: shutdown 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 >> v1:172.21.15.1:6793/0 conn(0x7f7ffc0a67c0 legacy=0x7f7ffc1ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ffb59e640 1 -- 172.21.15.1:0/400776583 reap_dead start 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f7ffb59e640 1 -- 172.21.15.1:0/400776583 reap_dead start 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 shutdown_connections 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.461+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 >> 172.21.15.1:0/400776583 conn(0x7f7ffc0ac4b0 msgr2=0x7f7ffc0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.462+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 shutdown_connections 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.462+0000 7f80010b3640 1 -- 172.21.15.1:0/400776583 wait complete. 2024-08-19T09:54:55.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.462+0000 7f80010b3640 1 librados: shutdown 2024-08-19T09:54:55.501 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:55.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:54:55.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:54:55.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.601+0000 7f5c42bb8640 10 monclient: get_monmap_and_config 2024-08-19T09:54:55.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.601+0000 7f5c42bb8640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.601+0000 7f5c42bb8640 10 monclient: monmap: 2024-08-19T09:54:55.602 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:55.602238+0000 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:55.602238+0000 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.603 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.602+0000 7f5c42bb8640 1 Processor -- start 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.602+0000 7f5c42bb8640 1 -- start start 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.602+0000 7f5c42bb8640 10 monclient: init 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient: _add_conns ranks=[3,1,6,0,8,7,5,4,2] 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient(hunting): picked mon.noname-g con 0x7f5c3c0a8fd0 addr v1:172.21.15.82:6790/0 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient(hunting): picked mon.noname-f con 0x7f5c3c06f020 addr v1:172.21.15.82:6789/0 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient(hunting): picked mon.noname-d con 0x7f5c3c0a6040 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a7d70 con 0x7f5c3c06f020 2024-08-19T09:54:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a75e0 con 0x7f5c3c0a8fd0 2024-08-19T09:54:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a4570 con 0x7f5c3c0a6040 2024-08-19T09:54:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c42bb8640 10 monclient(hunting): authenticate will time out at 951.679800s 2024-08-19T09:54:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c41155640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5c3c0a6040 0x7f5c3c145da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52914/0 (socket says 172.21.15.1:52914) 2024-08-19T09:54:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.603+0000 7f5c41155640 1 -- 172.21.15.1:0/1735329362 learned_addr learned my addr 172.21.15.1:0/1735329362 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c41155640 1 -- 172.21.15.1:0/1735329362 >> v1:172.21.15.1:6792/0 conn(0x7f5c3c0a6040 legacy=0x7f5c3c145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c40954640 1 -- 172.21.15.1:0/1735329362 >> v1:172.21.15.82:6790/0 conn(0x7f5c3c0a8fd0 legacy=0x7f5c3c0a67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3370320197 0 0) 0x7f5c3c0a4570 con 0x7f5c3c0a6040 2024-08-19T09:54:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c3b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.606+0000 7f5c3b7fe640 10 monclient(hunting): my global_id is 27114 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c28003520 con 0x7f5c3c0a6040 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4119180675 0 0) 0x7f5c28003520 con 0x7f5c3c0a6040 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 >> v1:172.21.15.82:6790/0 conn(0x7f5c3c0a8fd0 legacy=0x7f5c3c0a67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 >> v1:172.21.15.82:6789/0 conn(0x7f5c3c06f020 legacy=0x7f5c3c0a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 monclient: found mon.noname-d 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c14a4f0 con 0x7f5c3c0a6040 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c34001e60 con 0x7f5c3c0a6040 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: dump: 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c42bb8640 5 monclient: authenticate success, global_id 27114 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c42bb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c34004c20 con 0x7f5c3c0a6040 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 1 -- 172.21.15.1:0/1735329362 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c34005190 con 0x7f5c3c0a6040 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c3b7fe640 10 monclient: dump: 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c42bb8640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.607+0000 7f5c42bb8640 4 set_mon_vals no callback set 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.608+0000 7f5c42bb8640 10 monclient: shutdown 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.608+0000 7f5c42bb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.608+0000 7f5c42bb8640 1 -- 172.21.15.1:0/1735329362 >> v1:172.21.15.1:6792/0 conn(0x7f5c3c0a6040 legacy=0x7f5c3c145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.608+0000 7f5c42bb8640 1 -- 172.21.15.1:0/1735329362 shutdown_connections 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.608+0000 7f5c42bb8640 1 -- 172.21.15.1:0/1735329362 >> 172.21.15.1:0/1735329362 conn(0x7f5c3c0ac4b0 msgr2=0x7f5c3c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.610+0000 7f5c42bb8640 1 -- 172.21.15.1:0/1735329362 shutdown_connections 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.611+0000 7f5c42bb8640 1 -- 172.21.15.1:0/1735329362 wait complete. 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.611+0000 7f5c42bb8640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.611+0000 7f5c42bb8640 10 monclient: monmap: 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.617 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 1 librados: starting msgr at 2024-08-19T09:54:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 1 librados: starting objecter 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 1 Processor -- start 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 1 -- start start 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 1 librados: setting wanted keys 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 1 librados: calling monclient init 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.612+0000 7f5c42bb8640 10 monclient: init 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient: _add_conns ranks=[4,6,5,2,8,1,7,3,0] 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3c06f020 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient(hunting): picked mon.noname-d con 0x7f5c3c0a6040 addr v1:172.21.15.1:6792/0 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c3bfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5c3c0a6040 0x7f5c3c1d7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:52916/0 (socket says 172.21.15.1:52916) 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c3bfff640 1 -- 172.21.15.1:0/2580101925 learned_addr learned my addr 172.21.15.1:0/2580101925 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient(hunting): picked mon.noname-h con 0x7f5c3c0a8fd0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c149d20 con 0x7f5c3c06f020 2024-08-19T09:54:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c13f510 con 0x7f5c3c0a8fd0 2024-08-19T09:54:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c13f7f0 con 0x7f5c3c0a6040 2024-08-19T09:54:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.613+0000 7f5c42bb8640 10 monclient(hunting): authenticate will time out at 951.689898s 2024-08-19T09:54:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c3bfff640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.1:6792/0 conn(0x7f5c3c0a6040 legacy=0x7f5c3c1d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c41155640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6791/0 conn(0x7f5c3c0a8fd0 legacy=0x7f5c3c1dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c397fa640 10 client.?.objecter ms_handle_connect 0x7f5c3c0a6040 2024-08-19T09:54:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c397fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c397fa640 10 client.?.objecter ms_handle_connect 0x7f5c3c0a8fd0 2024-08-19T09:54:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c397fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c40954640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.1:6791/0 conn(0x7f5c3c06f020 legacy=0x7f5c3c1d2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c397fa640 10 client.?.objecter ms_handle_connect 0x7f5c3c06f020 2024-08-19T09:54:55.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.616+0000 7f5c397fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214392155 0 0) 0x7f5c3c13f7f0 con 0x7f5c3c0a6040 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): my global_id is 27123 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c1c004eb0 con 0x7f5c3c0a6040 2024-08-19T09:54:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1466776663 0 0) 0x7f5c3c0a75e0 con 0x7f5c3c06f020 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.629+0000 7f5c397fa640 10 monclient(hunting): my global_id is 26959 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c13f7f0 con 0x7f5c3c06f020 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3983141275 0 0) 0x7f5c3c13f7f0 con 0x7f5c3c06f020 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c3c0a75e0 con 0x7f5c3c06f020 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3655724436 0 0) 0x7f5c1c004eb0 con 0x7f5c3c0a6040 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c3c13f7f0 con 0x7f5c3c0a6040 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c2c001c80 con 0x7f5c3c06f020 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:54:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): dump: 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c24004470 con 0x7f5c3c0a6040 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient(hunting): dump: 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.636 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3290342317 0 0) 0x7f5c3c0a75e0 con 0x7f5c3c06f020 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.1:6792/0 conn(0x7f5c3c0a6040 legacy=0x7f5c3c1d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6791/0 conn(0x7f5c3c0a8fd0 legacy=0x7f5c3c1dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 monclient: found mon.c 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c0ab850 con 0x7f5c3c06f020 2024-08-19T09:54:55.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 5 monclient: authenticate success, global_id 26959 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 10 monclient: _renew_subs 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c13f230 con 0x7f5c3c06f020 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 10 client.26959.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 10 monclient: _renew_subs 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c3c0a09a0 con 0x7f5c3c06f020 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c42bb8640 1 librados: init done 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c2c0038c0 con 0x7f5c3c06f020 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c2c003e30 con 0x7f5c3c06f020 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c3a7fc640 4 set_mon_vals no callback set 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.630+0000 7f5c397fa640 10 monclient: dump: 2024-08-19T09:54:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.641 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.641 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.641 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.631+0000 7f5c42bb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.631+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c200065c0 con 0x7f5c20002860 2024-08-19T09:54:55.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.631+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5c2c004100 con 0x7f5c3c06f020 2024-08-19T09:54:55.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.632+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(198..198 src has 1..198) ==== 6177+0+0 (unknown 120415132 0 0) 0x7f5c2c04e390 con 0x7f5c3c06f020 2024-08-19T09:54:55.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.632+0000 7f5c397fa640 10 client.26959.objecter ms_dispatch 0x7f5c3c000b90 osd_map(198..198 src has 1..198) 2024-08-19T09:54:55.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.632+0000 7f5c397fa640 3 client.26959.objecter handle_osd_map got epochs [198,198] > 0 2024-08-19T09:54:55.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.632+0000 7f5c397fa640 3 client.26959.objecter handle_osd_map decoding full epoch 198 2024-08-19T09:54:55.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.632+0000 7f5c397fa640 20 client.26959.objecter dump_active .. 0 homeless 2024-08-19T09:54:55.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.632+0000 7f5c41155640 10 monclient: get_auth_request con 0x7f5c1c041750 auth_method 0 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c3bfff640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6790/0 conn(0x7f5c20002860 legacy=0x7f5c20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 client.26959.objecter ms_handle_connect 0x7f5c20002860 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 client.26959.objecter resend_mon_ops 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993118145 0 0) 0x7f5c200065c0 con 0x7f5c20002860 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient(hunting): my global_id is 134715 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c0a75e0 con 0x7f5c20002860 2024-08-19T09:54:55.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 client.26959.objecter ms_handle_connect 0x7f5c1c041750 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4108274342 0 0) 0x7f5c3c0a75e0 con 0x7f5c20002860 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c200065c0 con 0x7f5c20002860 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c24001640 con 0x7f5c20002860 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 10 monclient: dump: 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.645 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2010380561 0 0) 0x7f5c200065c0 con 0x7f5c20002860 2024-08-19T09:54:55.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.634+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c2000a160 con 0x7f5c20002860 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.635+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5c240018d0 con 0x7f5c20002860 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.635+0000 7f5c397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.635+0000 7f5c397fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.635+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6790/0 conn(0x7f5c20002860 legacy=0x7f5c20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.635+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6790/0 conn(0x7f5c20002860 legacy=0x7f5c20004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.641+0000 7f5c42bb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:55.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.641+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c2000c4f0 con 0x7f5c20009110 2024-08-19T09:54:55.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 client.26959.objecter ms_handle_connect 0x7f5c20009110 2024-08-19T09:54:55.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 client.26959.objecter resend_mon_ops 2024-08-19T09:54:55.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c3bfff640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6790/0 conn(0x7f5c20009110 legacy=0x7f5c20009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628193606 0 0) 0x7f5c2000c4f0 con 0x7f5c20009110 2024-08-19T09:54:55.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient(hunting): my global_id is 134724 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c200065c0 con 0x7f5c20009110 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1699651578 0 0) 0x7f5c200065c0 con 0x7f5c20009110 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c2000c4f0 con 0x7f5c20009110 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c24005b00 con 0x7f5c20009110 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient: dump: 2024-08-19T09:54:55.651 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.652 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2353617380 0 0) 0x7f5c2000c4f0 con 0x7f5c20009110 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c20009ce0 con 0x7f5c20009110 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1159139518) 0x7f5c24006f70 con 0x7f5c20009110 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:55.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6790/0 conn(0x7f5c20009110 legacy=0x7f5c20009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.644+0000 7f5c397fa640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6790/0 conn(0x7f5c20009110 legacy=0x7f5c20009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 152705, 2024-08-19T09:54:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:55.679 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:55.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 10 librados: watch_flush enter 2024-08-19T09:54:55.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 10 librados: watch_flush exit 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 20 client.26959.objecter shutdown clearing up homeless session... 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 10 client.26959.objecter successfully canceled tick 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5c1c041750 legacy=0x7f5c1c043c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 10 monclient: shutdown 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 >> v1:172.21.15.1:6791/0 conn(0x7f5c3c06f020 legacy=0x7f5c3c1d2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c41155640 1 -- 172.21.15.1:0/2580101925 reap_dead start 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c41155640 1 -- 172.21.15.1:0/2580101925 reap_dead start 2024-08-19T09:54:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 shutdown_connections 2024-08-19T09:54:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 >> 172.21.15.1:0/2580101925 conn(0x7f5c3c0ac4b0 msgr2=0x7f5c3c0a66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 shutdown_connections 2024-08-19T09:54:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 -- 172.21.15.1:0/2580101925 wait complete. 2024-08-19T09:54:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.645+0000 7f5c42bb8640 1 librados: shutdown 2024-08-19T09:54:55.683 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:55.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:54:55.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:54:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.782+0000 7fb9baeb3640 10 monclient: get_monmap_and_config 2024-08-19T09:54:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.782+0000 7fb9baeb3640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.782+0000 7fb9baeb3640 10 monclient: monmap: 2024-08-19T09:54:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:55.783283+0000 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:55.783283+0000 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.785 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 1 Processor -- start 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 1 -- start start 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient: init 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient: _add_conns ranks=[5,4,2,1,6,7,3,8,0] 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient(hunting): picked mon.noname-h con 0x7fb9b40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b40a3b40 con 0x7fb9b413f5c0 2024-08-19T09:54:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b40a8130 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b406ede0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9baeb3640 10 monclient(hunting): authenticate will time out at 951.859697s 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9b3fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb9b40a6ba0 0x7fb9b40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59450/0 (socket says 172.21.15.1:59450) 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.783+0000 7fb9b3fff640 1 -- 172.21.15.1:0/2832550504 learned_addr learned my addr 172.21.15.1:0/2832550504 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.785+0000 7fb9b3fff640 1 -- 172.21.15.1:0/2832550504 >> v1:172.21.15.1:6791/0 conn(0x7fb9b40a6ba0 legacy=0x7fb9b40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.785+0000 7fb9b9450640 1 -- 172.21.15.1:0/2832550504 >> v1:172.21.15.1:6790/0 conn(0x7fb9b413f5c0 legacy=0x7fb9b41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.785+0000 7fb9b8c4f640 1 -- 172.21.15.1:0/2832550504 >> v1:172.21.15.82:6791/0 conn(0x7fb9b40a67c0 legacy=0x7fb9b40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717396357 0 0) 0x7fb9b40a8130 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): my global_id is 26977 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a0003520 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046396531 0 0) 0x7fb9b40a3b40 con 0x7fb9b413f5c0 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): my global_id is 144236 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b40a8130 con 0x7fb9b413f5c0 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 170222227 0 0) 0x7fb9b406ede0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient(hunting): my global_id is 26699 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b40a3b40 con 0x7fb9b40a67c0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 672752480 0 0) 0x7fb9a0003520 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 >> v1:172.21.15.82:6791/0 conn(0x7fb9b40a67c0 legacy=0x7fb9b40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 >> v1:172.21.15.1:6790/0 conn(0x7fb9b413f5c0 legacy=0x7fb9b41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 monclient: found mon.noname-c 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b40a92a0 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb99c001ae0 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9baeb3640 5 monclient: authenticate success, global_id 26977 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9baeb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: dump: 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9baeb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb99c0048a0 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 1 -- 172.21.15.1:0/2832550504 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb99c004e10 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9baeb3640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9baeb3640 4 set_mon_vals no callback set 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.786+0000 7fb9b37fe640 10 monclient: dump: 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 10 monclient: shutdown 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 -- 172.21.15.1:0/2832550504 >> v1:172.21.15.1:6791/0 conn(0x7fb9b40a6ba0 legacy=0x7fb9b40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 -- 172.21.15.1:0/2832550504 shutdown_connections 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 -- 172.21.15.1:0/2832550504 >> 172.21.15.1:0/2832550504 conn(0x7fb9b40ac4b0 msgr2=0x7fb9b40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 -- 172.21.15.1:0/2832550504 shutdown_connections 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 -- 172.21.15.1:0/2832550504 wait complete. 2024-08-19T09:54:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 10 monclient: monmap: 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 librados: starting msgr at 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.787+0000 7fb9baeb3640 1 librados: starting objecter 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 Processor -- start 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 -- start start 2024-08-19T09:54:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 librados: setting wanted keys 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 librados: calling monclient init 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient: init 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient: _add_conns ranks=[4,8,7,3,5,1,2,6,0] 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient(hunting): picked mon.noname-e con 0x7fb9b40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient(hunting): picked mon.noname-i con 0x7fb9b413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b413d150 con 0x7fb9b40a67c0 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b413d430 con 0x7fb9b413f5c0 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b41cb0c0 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9baeb3640 10 monclient(hunting): authenticate will time out at 951.864778s 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9b8c4f640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb9b40a67c0 0x7fb9b41ca2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:59454/0 (socket says 172.21.15.1:59454) 2024-08-19T09:54:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.788+0000 7fb9b8c4f640 1 -- 172.21.15.1:0/1974367486 learned_addr learned my addr 172.21.15.1:0/1974367486 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.790+0000 7fb9b8c4f640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.1:6791/0 conn(0x7fb9b40a67c0 legacy=0x7fb9b41ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.790+0000 7fb9b3fff640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.1:6793/0 conn(0x7fb9b40a6ba0 legacy=0x7fb9b41ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.790+0000 7fb9b17fa640 10 client.?.objecter ms_handle_connect 0x7fb9b40a67c0 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.790+0000 7fb9b17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.790+0000 7fb9b17fa640 10 client.?.objecter ms_handle_connect 0x7fb9b40a6ba0 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.790+0000 7fb9b17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561692545 0 0) 0x7fb9b413d150 con 0x7fb9b40a67c0 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): my global_id is 26986 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b9450640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6792/0 conn(0x7fb9b413f5c0 legacy=0x7fb9b41d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb994004eb0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 client.?.objecter ms_handle_connect 0x7fb9b413f5c0 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2899879603 0 0) 0x7fb9b41cb0c0 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): my global_id is 27539 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b413d150 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1379592393 0 0) 0x7fb9b413d430 con 0x7fb9b413f5c0 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): my global_id is 144187 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b41cb0c0 con 0x7fb9b413f5c0 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3985345104 0 0) 0x7fb994004eb0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9b413d430 con 0x7fb9b40a67c0 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb9a4003990 con 0x7fb9b40a67c0 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): dump: 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.809 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 925005000 0 0) 0x7fb9b413d150 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb994004eb0 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb99c004d90 con 0x7fb9b40a6ba0 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): dump: 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.811 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.812 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 413620331 0 0) 0x7fb9b41cb0c0 con 0x7fb9b413f5c0 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9b413d150 con 0x7fb9b413f5c0 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb9ac003610 con 0x7fb9b413f5c0 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient(hunting): dump: 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.813 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1025652638 0 0) 0x7fb9b413d430 con 0x7fb9b40a67c0 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.1:6793/0 conn(0x7fb9b40a6ba0 legacy=0x7fb9b41ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6792/0 conn(0x7fb9b413f5c0 legacy=0x7fb9b41d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 monclient: found mon.c 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b413c130 con 0x7fb9b40a67c0 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 5 monclient: authenticate success, global_id 26986 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 10 monclient: _renew_subs 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b40a9fc0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 10 client.26986.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 10 monclient: _renew_subs 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fb9b41d72a0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9baeb3640 1 librados: init done 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9a4003f10 con 0x7fb9b40a67c0 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb9a4004480 con 0x7fb9b40a67c0 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b2ffd640 4 set_mon_vals no callback set 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.791+0000 7fb9b17fa640 10 monclient: dump: 2024-08-19T09:54:55.816 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.792+0000 7fb9baeb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.792+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb9980065c0 con 0x7fb998002860 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.792+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb9a4004620 con 0x7fb9b40a67c0 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.793+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(198..198 src has 1..198) ==== 6177+0+0 (unknown 120415132 0 0) 0x7fb9a404e8a0 con 0x7fb9b40a67c0 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.793+0000 7fb9b17fa640 10 client.26986.objecter ms_dispatch 0x7fb9b4000b90 osd_map(198..198 src has 1..198) 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.793+0000 7fb9b17fa640 3 client.26986.objecter handle_osd_map got epochs [198,198] > 0 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.793+0000 7fb9b17fa640 3 client.26986.objecter handle_osd_map decoding full epoch 198 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.793+0000 7fb9b17fa640 20 client.26986.objecter dump_active .. 0 homeless 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.793+0000 7fb9b9450640 10 monclient: get_auth_request con 0x7fb9940419a0 auth_method 0 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.794+0000 7fb9b3fff640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6791/0 conn(0x7fb998002860 legacy=0x7fb998004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.794+0000 7fb9b17fa640 10 client.26986.objecter ms_handle_connect 0x7fb998002860 2024-08-19T09:54:55.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.794+0000 7fb9b17fa640 10 client.26986.objecter resend_mon_ops 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228847969 0 0) 0x7fb9980065c0 con 0x7fb998002860 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient(hunting): my global_id is 26708 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b413d430 con 0x7fb998002860 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4079185809 0 0) 0x7fb9b413d430 con 0x7fb998002860 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9980065c0 con 0x7fb998002860 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb99c001580 con 0x7fb998002860 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient: dump: 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.820 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 client.26986.objecter ms_handle_connect 0x7fb9940419a0 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 495676276 0 0) 0x7fb9980065c0 con 0x7fb998002860 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb99800a160 con 0x7fb998002860 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb99c0048a0 con 0x7fb998002860 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6791/0 conn(0x7fb998002860 legacy=0x7fb998004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.795+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6791/0 conn(0x7fb998002860 legacy=0x7fb998004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.801+0000 7fb9baeb3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:55.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.802+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb998009e70 con 0x7fb998009130 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b3fff640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6791/0 conn(0x7fb998009130 legacy=0x7fb9980095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 client.26986.objecter ms_handle_connect 0x7fb998009130 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 client.26986.objecter resend_mon_ops 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512449448 0 0) 0x7fb998009e70 con 0x7fb998009130 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient(hunting): my global_id is 26717 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9980065c0 con 0x7fb998009130 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3508650951 0 0) 0x7fb9980065c0 con 0x7fb998009130 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb998009e70 con 0x7fb998009130 2024-08-19T09:54:55.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb99c005ac0 con 0x7fb998009130 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.804+0000 7fb9b17fa640 10 monclient: dump: 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.823 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2586697188 0 0) 0x7fb998009e70 con 0x7fb998009130 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb99800c410 con 0x7fb998009130 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3103412526) 0x7fb99c005dd0 con 0x7fb998009130 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6791/0 conn(0x7fb998009130 legacy=0x7fb9980095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b17fa640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6791/0 conn(0x7fb998009130 legacy=0x7fb9980095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 302071, 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:54:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 10 librados: watch_flush enter 2024-08-19T09:54:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 10 librados: watch_flush exit 2024-08-19T09:54:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 20 client.26986.objecter shutdown clearing up homeless session... 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 10 client.26986.objecter successfully canceled tick 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb9940419a0 legacy=0x7fb994043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 10 monclient: shutdown 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 >> v1:172.21.15.1:6791/0 conn(0x7fb9b40a67c0 legacy=0x7fb9b41ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b9450640 1 -- 172.21.15.1:0/1974367486 reap_dead start 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9b9450640 1 -- 172.21.15.1:0/1974367486 reap_dead start 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 shutdown_connections 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.805+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 >> 172.21.15.1:0/1974367486 conn(0x7fb9b40ac4b0 msgr2=0x7fb9b41419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.806+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 shutdown_connections 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.806+0000 7fb9baeb3640 1 -- 172.21.15.1:0/1974367486 wait complete. 2024-08-19T09:54:55.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.806+0000 7fb9baeb3640 1 librados: shutdown 2024-08-19T09:54:55.849 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:55.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:54:55.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.948+0000 7fabaf88b640 10 monclient: get_monmap_and_config 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.948+0000 7fabaf88b640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.948+0000 7fabaf88b640 10 monclient: monmap: 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:54:55.949894+0000 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:54:55.949894+0000 2024-08-19T09:54:55.950 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.949+0000 7fabaf88b640 1 Processor -- start 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.949+0000 7fabaf88b640 1 -- start start 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.949+0000 7fabaf88b640 10 monclient: init 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient: _add_conns ranks=[0,7,5,1,8,4,6,3,2] 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient(hunting): picked mon.noname-a con 0x7faba80a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient(hunting): picked mon.noname-i con 0x7faba80a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient(hunting): picked mon.noname-h con 0x7faba813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba80a3b40 con 0x7faba80a67c0 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba80a8130 con 0x7faba813f5c0 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba806ede0 con 0x7faba80a6ba0 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabaf88b640 10 monclient(hunting): authenticate will time out at 952.026310s 2024-08-19T09:54:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabad627640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faba80a67c0 0x7faba80a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37314/0 (socket says 172.21.15.1:37314) 2024-08-19T09:54:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.950+0000 7fabad627640 1 -- 172.21.15.1:0/1064290688 learned_addr learned my addr 172.21.15.1:0/1064290688 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7fabad627640 1 -- 172.21.15.1:0/1064290688 >> v1:172.21.15.1:6789/0 conn(0x7faba80a67c0 legacy=0x7faba80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7fabace26640 1 -- 172.21.15.1:0/1064290688 >> v1:172.21.15.82:6792/0 conn(0x7faba80a6ba0 legacy=0x7faba80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7fabade28640 1 -- 172.21.15.1:0/1064290688 >> v1:172.21.15.82:6791/0 conn(0x7faba813f5c0 legacy=0x7faba81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2264111973 0 0) 0x7faba80a3b40 con 0x7faba80a67c0 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): my global_id is 134721 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab90003520 con 0x7faba80a67c0 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096552126 0 0) 0x7faba806ede0 con 0x7faba80a6ba0 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): my global_id is 144196 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba80a3b40 con 0x7faba80a6ba0 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285930092 0 0) 0x7faba80a8130 con 0x7faba813f5c0 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 10 monclient(hunting): my global_id is 26726 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.952+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba806ede0 con 0x7faba813f5c0 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2030314400 0 0) 0x7fab90003520 con 0x7faba80a67c0 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 >> v1:172.21.15.82:6792/0 conn(0x7faba80a6ba0 legacy=0x7faba80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 >> v1:172.21.15.82:6791/0 conn(0x7faba813f5c0 legacy=0x7faba81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 monclient: found mon.noname-a 2024-08-19T09:54:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba80a92a0 con 0x7faba80a67c0 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab98001e60 con 0x7faba80a67c0 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 5 monclient: authenticate success, global_id 134721 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: dump: 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.961 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab98004c20 con 0x7faba80a67c0 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 1 -- 172.21.15.1:0/1064290688 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab98005190 con 0x7faba80a67c0 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7faba7fff640 10 monclient: dump: 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 10 monclient: get_monmap_and_config success 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 4 set_mon_vals no callback set 2024-08-19T09:54:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 10 monclient: shutdown 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 1 -- 172.21.15.1:0/1064290688 >> v1:172.21.15.1:6789/0 conn(0x7faba80a67c0 legacy=0x7faba80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 1 -- 172.21.15.1:0/1064290688 shutdown_connections 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 1 -- 172.21.15.1:0/1064290688 >> 172.21.15.1:0/1064290688 conn(0x7faba80ac4b0 msgr2=0x7faba80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 1 -- 172.21.15.1:0/1064290688 shutdown_connections 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.953+0000 7fabaf88b640 1 -- 172.21.15.1:0/1064290688 wait complete. 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 10 monclient: build_initial_monmap 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 10 monclient: monmap: 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:54:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 1 librados: starting msgr at 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 1 librados: starting objecter 2024-08-19T09:54:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 1 Processor -- start 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 1 -- start start 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 1 librados: setting wanted keys 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 1 librados: calling monclient init 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.954+0000 7fabaf88b640 10 monclient: init 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient: _reopen_session rank -1 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient: _add_conns ranks=[7,5,0,8,4,6,2,1,3] 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient(hunting): picked mon.noname-i con 0x7faba80a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient(hunting): picked mon.noname-h con 0x7faba80a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient(hunting): picked mon.noname-a con 0x7faba813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba80aa140 con 0x7faba813f5c0 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba813be30 con 0x7faba80a6ba0 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba81cf020 con 0x7faba80a67c0 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient(hunting): _renew_subs 2024-08-19T09:54:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabaf88b640 10 monclient(hunting): authenticate will time out at 952.031367s 2024-08-19T09:54:55.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabade28640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faba813f5c0 0x7faba81d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37318/0 (socket says 172.21.15.1:37318) 2024-08-19T09:54:55.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.955+0000 7fabade28640 1 -- 172.21.15.1:0/3038615674 learned_addr learned my addr 172.21.15.1:0/3038615674 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7fabade28640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.1:6789/0 conn(0x7faba813f5c0 legacy=0x7faba81d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 client.?.objecter ms_handle_connect 0x7faba813f5c0 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 client.?.objecter ms_handle_connect 0x7faba80a6ba0 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 client.?.objecter ms_handle_connect 0x7faba80a67c0 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7fabad627640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba80a67c0 legacy=0x7faba81ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7fabace26640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6791/0 conn(0x7faba80a6ba0 legacy=0x7faba81ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352573177 0 0) 0x7faba806ede0 con 0x7faba813f5c0 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): my global_id is 134730 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab94004eb0 con 0x7faba813f5c0 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3552143238 0 0) 0x7faba813be30 con 0x7faba80a6ba0 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): my global_id is 26735 2024-08-19T09:54:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba806ede0 con 0x7faba80a6ba0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4271266060 0 0) 0x7faba81cf020 con 0x7faba80a67c0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): my global_id is 144205 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba813be30 con 0x7faba80a67c0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2353907841 0 0) 0x7fab94004eb0 con 0x7faba813f5c0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faba81cf020 con 0x7faba813f5c0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faba0003610 con 0x7faba813f5c0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): dump: 2024-08-19T09:54:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.957+0000 7faba5ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 386631244 0 0) 0x7faba806ede0 con 0x7faba80a6ba0 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab94004eb0 con 0x7faba80a6ba0 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab9c003610 con 0x7faba80a6ba0 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient(hunting): dump: 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 758949436 0 0) 0x7faba81cf020 con 0x7faba813f5c0 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba80a67c0 legacy=0x7faba81ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6791/0 conn(0x7faba80a6ba0 legacy=0x7faba81ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 monclient: found mon.a 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba813bb50 con 0x7faba813f5c0 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: _finish_auth 0 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 5 monclient: authenticate success, global_id 134730 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 10 monclient: _renew_subs 2024-08-19T09:54:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faba813a8d0 con 0x7faba813f5c0 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faba0003b10 con 0x7faba813f5c0 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faba0004080 con 0x7faba813f5c0 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba5ffb640 10 monclient: dump: 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.980 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.981 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 10 client.134730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 10 monclient: _renew_subs 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7faba77fe640 4 set_mon_vals no callback set 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faba813d520 con 0x7faba813f5c0 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.958+0000 7fabaf88b640 1 librados: init done 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7fabaf88b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faba80008d0 con 0x7faba809da50 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7faba0004220 con 0x7faba813f5c0 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(198..198 src has 1..198) ==== 6177+0+0 (unknown 120415132 0 0) 0x7faba004e3e0 con 0x7faba813f5c0 2024-08-19T09:54:55.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7faba5ffb640 10 client.134730.objecter ms_dispatch 0x7faba8000b90 osd_map(198..198 src has 1..198) 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7faba5ffb640 3 client.134730.objecter handle_osd_map got epochs [198,198] > 0 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7faba5ffb640 3 client.134730.objecter handle_osd_map decoding full epoch 198 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7faba5ffb640 20 client.134730.objecter dump_active .. 0 homeless 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.959+0000 7fabace26640 10 monclient: get_auth_request con 0x7fab94041510 auth_method 0 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7fabad627640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba809da50 legacy=0x7faba809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 client.134730.objecter ms_handle_connect 0x7faba809da50 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 client.134730.objecter resend_mon_ops 2024-08-19T09:54:55.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583377536 0 0) 0x7faba80008d0 con 0x7faba809da50 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient(hunting): my global_id is 144214 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba81cf020 con 0x7faba809da50 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 905373704 0 0) 0x7faba81cf020 con 0x7faba809da50 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faba80008d0 con 0x7faba809da50 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab980020b0 con 0x7faba809da50 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 monclient: dump: 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.985 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 10 client.134730.objecter ms_handle_connect 0x7fab94041510 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 724749496 0 0) 0x7faba80008d0 con 0x7faba809da50 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.961+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faba80a9ca0 con 0x7faba809da50 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.962+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fab980016a0 con 0x7faba809da50 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.962+0000 7faba5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.962+0000 7faba5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.962+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba809da50 legacy=0x7faba809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.962+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba809da50 legacy=0x7faba809de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.968+0000 7fabaf88b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:54:55.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.968+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faba809e920 con 0x7faba809e540 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7fabad627640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba809e540 legacy=0x7faba81dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 client.134730.objecter ms_handle_connect 0x7faba809e540 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 client.134730.objecter resend_mon_ops 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1140473858 0 0) 0x7faba809e920 con 0x7faba809e540 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient(hunting): my global_id is 144223 2024-08-19T09:54:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba80008d0 con 0x7faba809e540 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2776937001 0 0) 0x7faba80008d0 con 0x7faba809e540 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faba809e920 con 0x7faba809e540 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab98005dc0 con 0x7faba809e540 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.971+0000 7faba5ffb640 10 monclient: dump: 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:54:55.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 243138789 0 0) 0x7faba809e920 con 0x7faba809e540 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faba81df450 con 0x7faba809e540 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 2778720284) 0x7fab980072a0 con 0x7faba809e540 2024-08-19T09:54:55.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:54:55.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:54:55.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba809e540 legacy=0x7faba81dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:55.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7faba5ffb640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6792/0 conn(0x7faba809e540 legacy=0x7faba81dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:54:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:54:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 20352, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:54:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:54:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:54:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:54:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:54:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:54:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:54:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:54:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:54:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:54:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:54:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:54:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:54:56.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 10 librados: watch_flush enter 2024-08-19T09:54:56.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 10 librados: watch_flush exit 2024-08-19T09:54:56.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 20 client.134730.objecter shutdown clearing up homeless session... 2024-08-19T09:54:56.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 10 client.134730.objecter successfully canceled tick 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.82:6812/2549104415 conn(0x7fab94041510 legacy=0x7fab940439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 10 monclient: shutdown 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 >> v1:172.21.15.1:6789/0 conn(0x7faba813f5c0 legacy=0x7faba81d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabade28640 1 -- 172.21.15.1:0/3038615674 reap_dead start 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabade28640 1 -- 172.21.15.1:0/3038615674 reap_dead start 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 shutdown_connections 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 >> 172.21.15.1:0/3038615674 conn(0x7faba80ac4b0 msgr2=0x7faba813edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 shutdown_connections 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.972+0000 7fabaf88b640 1 -- 172.21.15.1:0/3038615674 wait complete. 2024-08-19T09:54:56.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:54:55.973+0000 7fabaf88b640 1 librados: shutdown 2024-08-19T09:54:56.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:54:56.020 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'd', 'b', 'f'] 2024-08-19T09:54:56.020 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['g'] 2024-08-19T09:54:56.020 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T09:54:56.020 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T09:54:56.020 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T09:54:56.020 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:54:56.032 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T09:54:56.032 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T09:54:56.033 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T09:54:56.033 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T09:54:56.033 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:54:56.085 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T09:54:56.085 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-19T09:54:56.085 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-19T09:54:56.086 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-19T09:54:56.086 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:54:56.136 INFO:tasks.ceph.mon.b:Stopped 2024-08-19T09:54:56.136 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-08-19T09:54:56.137 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-08-19T09:54:56.137 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T09:54:56.137 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:54:56.154 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T09:54:56.154 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-08-19T09:54:56.154 INFO:tasks.ceph.mon.g:Sent signal 19 2024-08-19T09:54:56.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:55:11.155 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-08-19T09:55:11.157 INFO:tasks.ceph.mon.g:Sent signal 18 2024-08-19T09:55:11.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-08-19T09:55:11.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.266+0000 7f8fce660640 10 monclient: get_monmap_and_config 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.266+0000 7f8fce660640 10 monclient: build_initial_monmap 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.266+0000 7f8fce660640 10 monclient: monmap: 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:55:11.267874+0000 2024-08-19T09:55:11.268 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:55:11.267874+0000 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:11.269 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.267+0000 7f8fce660640 1 Processor -- start 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.267+0000 7f8fce660640 1 -- start start 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.267+0000 7f8fce660640 10 monclient: init 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient: _add_conns ranks=[2,1,6,0,3,4,7,8,5] 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient(hunting): picked mon.noname-b con 0x7f8fc809da50 addr v1:172.21.15.1:6790/0 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient(hunting): picked mon.noname-f con 0x7f8fc80a3fe0 addr v1:172.21.15.82:6789/0 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient(hunting): picked mon.noname-d con 0x7f8fc809e540 addr v1:172.21.15.1:6792/0 2024-08-19T09:55:11.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a3d60 con 0x7f8fc80a3fe0 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a1930 con 0x7f8fc809da50 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a1c10 con 0x7f8fc809e540 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fce660640 10 monclient(hunting): authenticate will time out at 967.344462s 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fc7fff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f8fc809da50 legacy=0x7f8fc809de30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fccbfd640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8fc809e540 legacy=0x7f8fc813fbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:55:11.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.268+0000 7f8fc77fe640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f8fc80a3fe0 legacy=0x7f8fc813c4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:11.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.469+0000 7f8fc7fff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f8fc809da50 legacy=0x7f8fc809de30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:11.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.469+0000 7f8fccbfd640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8fc809e540 legacy=0x7f8fc813fbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:55:11.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.469+0000 7f8fc77fe640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f8fc80a3fe0 legacy=0x7f8fc813c4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:11.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.870+0000 7f8fc77fe640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f8fc80a3fe0 legacy=0x7f8fc813c4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:11.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.870+0000 7f8fc7fff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f8fc809da50 legacy=0x7f8fc809de30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:11.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:11.870+0000 7f8fccbfd640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8fc809e540 legacy=0x7f8fc813fbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:55:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:12.671+0000 7f8fc77fe640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f8fc80a3fe0 legacy=0x7f8fc813c4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:12.671+0000 7f8fc7fff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f8fc809da50 legacy=0x7f8fc809de30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:12.671+0000 7f8fccbfd640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8fc809e540 legacy=0x7f8fc813fbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:55:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:12.671+0000 7f8fc6ffd640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:55:14.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): tick 2024-08-19T09:55:14.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:55:14.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 monclient(hunting): continuing hunt 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8fc809e540 legacy=0x7f8fc813fbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f8fc809da50 legacy=0x7f8fc809de30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f8fc80a3fe0 legacy=0x7f8fc813c4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient: _add_conns ranks=[8,4,5,3,7,0] 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-e con 0x7f8fb4003780 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb4007410 addr v1:172.21.15.1:6791/0 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-h con 0x7f8fb400b1a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a0f90 con 0x7f8fb4007410 2024-08-19T09:55:14.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a3d60 con 0x7f8fb400b1a0 2024-08-19T09:55:14.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a1930 con 0x7f8fb4003780 2024-08-19T09:55:14.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc5ffb640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:14.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc7fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f8fb4003780 0x7f8fb4005bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:53514/0 (socket says 172.21.15.1:53514) 2024-08-19T09:55:14.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc7fff640 1 -- 172.21.15.1:0/4270861902 learned_addr learned my addr 172.21.15.1:0/4270861902 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:14.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.268+0000 7f8fc77fe640 1 --1- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6791/0 conn(0x7f8fb4007410 0x7f8fb40098d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:47220/0 (socket says 172.21.15.1:47220) 2024-08-19T09:55:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.271+0000 7f8fc7fff640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6793/0 conn(0x7f8fb4003780 legacy=0x7f8fb4005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.271+0000 7f8fc77fe640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6791/0 conn(0x7f8fb4007410 legacy=0x7f8fb40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:14.271+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6791/0 conn(0x7f8fb400b1a0 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:17.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.268+0000 7f8fc5ffb640 10 monclient(hunting): tick 2024-08-19T09:55:17.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.268+0000 7f8fc5ffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.268+0000 7f8fc5ffb640 1 monclient(hunting): continuing hunt 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.268+0000 7f8fc5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.268+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6793/0 conn(0x7f8fb4003780 legacy=0x7f8fb4005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6791/0 conn(0x7f8fb400b1a0 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6791/0 conn(0x7f8fb4007410 legacy=0x7f8fb40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 10 monclient: _add_conns ranks=[3,7,0] 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 reap_dead start 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-g con 0x7f8fb400f210 addr v1:172.21.15.82:6790/0 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 reap_dead start 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-i con 0x7f8fb400a0a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb400a520 addr v1:172.21.15.1:6789/0 2024-08-19T09:55:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a1c10 con 0x7f8fb400a520 2024-08-19T09:55:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb400aa70 con 0x7f8fb400f210 2024-08-19T09:55:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb400ace0 con 0x7f8fb400a0a0 2024-08-19T09:55:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fc5ffb640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.269+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.271+0000 7f8fc7fff640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6790/0 conn(0x7f8fb400f210 legacy=0x7f8fb400f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.271+0000 7f8fc77fe640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6792/0 conn(0x7f8fb400a0a0 legacy=0x7f8fb4005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:17.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.470+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:17.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:17.871+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:18.672+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): tick 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 monclient(hunting): continuing hunt 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6792/0 conn(0x7f8fb400a0a0 legacy=0x7f8fb4005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6790/0 conn(0x7f8fb400f210 legacy=0x7f8fb400f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient: _add_conns ranks=[4,0,5,8,3,1,7,6,2] 2024-08-19T09:55:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb4009b30 addr v1:172.21.15.1:6791/0 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb400a520 addr v1:172.21.15.1:6789/0 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): picked mon.noname-h con 0x7f8fb4011680 addr v1:172.21.15.82:6791/0 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb40098e0 con 0x7f8fb400a520 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a1c10 con 0x7f8fb4009b30 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb4011be0 con 0x7f8fb4011680 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc5ffb640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.269+0000 7f8fc77fe640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.271+0000 7f8fc7fff640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6791/0 conn(0x7f8fb4009b30 legacy=0x7f8fb4010f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.272+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6791/0 conn(0x7f8fb4011680 legacy=0x7f8fb4017130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:20.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.470+0000 7f8fc77fe640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.644+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072333472 0 0) 0x7f8fc80a1c10 con 0x7f8fb4009b30 2024-08-19T09:55:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.644+0000 7f8fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:55:20.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.644+0000 7f8fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:20.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.644+0000 7f8fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:20.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.644+0000 7f8fc6ffd640 10 monclient(hunting): my global_id is 26995 2024-08-19T09:55:20.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.644+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fb0003520 con 0x7f8fb4009b30 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463161693 0 0) 0x7f8fb4011be0 con 0x7f8fb4011680 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient(hunting): my global_id is 26744 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fc80a1c10 con 0x7f8fb4011680 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1878213653 0 0) 0x7f8fb0003520 con 0x7f8fb4009b30 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.82:6791/0 conn(0x7f8fb4011680 legacy=0x7f8fb4017130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6789/0 conn(0x7f8fb400a520 legacy=0x7f8fb400d640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:20.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 monclient: found mon.noname-c 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb401b880 con 0x7f8fb4009b30 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: _finish_auth 0 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fb8004570 con 0x7f8fb4009b30 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fce660640 5 monclient: authenticate success, global_id 26995 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fce660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:55:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fc6ffd640 10 monclient: dump: 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:20.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:20.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.645+0000 7f8fce660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8fb80047c0 con 0x7f8fb4009b30 2024-08-19T09:55:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fc6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fc6ffd640 1 -- 172.21.15.1:0/4270861902 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fb8004d30 con 0x7f8fb4009b30 2024-08-19T09:55:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:55:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fc6ffd640 10 monclient: dump: 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fce660640 10 monclient: get_monmap_and_config success 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.647+0000 7f8fce660640 4 set_mon_vals no callback set 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 10 monclient: shutdown 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 1 -- 172.21.15.1:0/4270861902 >> v1:172.21.15.1:6791/0 conn(0x7f8fb4009b30 legacy=0x7f8fb4010f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fccbfd640 1 -- 172.21.15.1:0/4270861902 reap_dead start 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 1 -- 172.21.15.1:0/4270861902 shutdown_connections 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 1 -- 172.21.15.1:0/4270861902 >> 172.21.15.1:0/4270861902 conn(0x7f8fc8068650 msgr2=0x7f8fc8068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 1 -- 172.21.15.1:0/4270861902 shutdown_connections 2024-08-19T09:55:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.648+0000 7f8fce660640 1 -- 172.21.15.1:0/4270861902 wait complete. 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.649+0000 7f8fce660640 10 monclient: build_initial_monmap 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.649+0000 7f8fce660640 10 monclient: monmap: 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 1 librados: starting msgr at 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 1 librados: starting objecter 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 1 Processor -- start 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 1 -- start start 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 1 librados: setting wanted keys 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 1 librados: calling monclient init 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.650+0000 7f8fce660640 10 monclient: init 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient: _add_conns ranks=[6,3,2,8,4,1,7,0,5] 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient(hunting): picked mon.noname-d con 0x7f8fb4009b30 addr v1:172.21.15.1:6792/0 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient(hunting): picked mon.noname-g con 0x7f8fc809f540 addr v1:172.21.15.82:6790/0 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient(hunting): picked mon.noname-b con 0x7f8fc81d42f0 addr v1:172.21.15.1:6790/0 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a04e0 con 0x7f8fc81d42f0 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fc7fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8fb4009b30 legacy=0x7f8fc809ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc8128790 con 0x7f8fc809f540 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc81284b0 con 0x7f8fb4009b30 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fccbfd640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f8fc81d42f0 legacy=0x7f8fc81d66e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fce660640 10 monclient(hunting): authenticate will time out at 976.727350s 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fc77fe640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f8fc809f540 0x7f8fc809f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:48642/0 (socket says 172.21.15.1:48642) 2024-08-19T09:55:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.651+0000 7f8fc77fe640 1 -- 172.21.15.1:0/524815925 learned_addr learned my addr 172.21.15.1:0/524815925 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.653+0000 7f8fc77fe640 1 -- 172.21.15.1:0/524815925 >> v1:172.21.15.82:6790/0 conn(0x7f8fc809f540 legacy=0x7f8fc809f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.653+0000 7f8fc4ff9640 10 client.?.objecter ms_handle_connect 0x7f8fc809f540 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.653+0000 7f8fc4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.654+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3221581466 0 0) 0x7f8fc8128790 con 0x7f8fc809f540 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.654+0000 7f8fc4ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.655+0000 7f8fc4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.655+0000 7f8fc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.655+0000 7f8fc4ff9640 10 monclient(hunting): my global_id is 134733 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.655+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fb4011be0 con 0x7f8fc809f540 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1245186466 0 0) 0x7f8fb4011be0 con 0x7f8fc809f540 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fc8128790 con 0x7f8fc809f540 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fbc004730 con 0x7f8fc809f540 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient(hunting): dump: 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2842554498 0 0) 0x7f8fc8128790 con 0x7f8fc809f540 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 >> v1:172.21.15.1:6792/0 conn(0x7f8fb4009b30 legacy=0x7f8fc809ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 >> v1:172.21.15.1:6790/0 conn(0x7f8fc81d42f0 legacy=0x7f8fc81d66e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 monclient: found mon.g 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc81272f0 con 0x7f8fc809f540 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: _finish_auth 0 2024-08-19T09:55:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 5 monclient: authenticate success, global_id 134733 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 10 monclient: _renew_subs 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc8126f90 con 0x7f8fc809f540 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 10 client.134733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 10 monclient: _renew_subs 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8fc81d7fe0 con 0x7f8fc809f540 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fce660640 1 librados: init done 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8fbc004990 con 0x7f8fc809f540 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fbc004f00 con 0x7f8fc809f540 2024-08-19T09:55:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc67fc640 4 set_mon_vals no callback set 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.656+0000 7f8fc4ff9640 10 monclient: dump: 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.657+0000 7f8fce660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.657+0000 7f8fce660640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.657+0000 7f8fce660640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.657+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa8005080 con 0x7f8fc809f540 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8fbc0050a0 con 0x7f8fc809f540 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(198..198 src has 1..198) ==== 6177+0+0 (unknown 120415132 0 0) 0x7f8fbc04e540 con 0x7f8fc809f540 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc4ff9640 10 client.134733.objecter ms_dispatch 0x7f8fc8000b90 osd_map(198..198 src has 1..198) 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc4ff9640 3 client.134733.objecter handle_osd_map got epochs [198,198] > 0 2024-08-19T09:55:20.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc4ff9640 3 client.134733.objecter handle_osd_map decoding full epoch 198 2024-08-19T09:55:20.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc4ff9640 20 client.134733.objecter dump_active .. 0 homeless 2024-08-19T09:55:20.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.658+0000 7f8fc7fff640 10 monclient: get_auth_request con 0x7f8fb4043130 auth_method 0 2024-08-19T09:55:20.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.661+0000 7f8fc4ff9640 10 client.134733.objecter ms_handle_connect 0x7f8fb4043130 2024-08-19T09:55:20.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.663+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f8fbc017540 con 0x7f8fc809f540 2024-08-19T09:55:20.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.663+0000 7f8fc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:20.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.663+0000 7f8fc4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:55:20.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.923+0000 7f8fce660640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.923+0000 7f8fce660640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.923+0000 7f8fce660640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.923+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fa8005370 con 0x7f8fc809f540 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.924+0000 7f8fc4ff9640 1 -- 172.21.15.1:0/524815925 <== mon.3 v1:172.21.15.82:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2678 (unknown 254746657 0 320131321) 0x7f8fbc01b440 con 0x7f8fc809f540 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.924+0000 7f8fc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.924+0000 7f8fc4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:55:20.925 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":56,"quorum":[3,4,5,7,8],"quorum_names":["g","c","h","i","e"],"quorum_leader_name":"g","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 10 librados: watch_flush enter 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 10 librados: watch_flush exit 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 20 client.134733.objecter shutdown clearing up homeless session... 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 10 client.134733.objecter successfully canceled tick 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8fb4043130 legacy=0x7f8fb40455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 10 monclient: shutdown 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 >> v1:172.21.15.82:6790/0 conn(0x7f8fc809f540 legacy=0x7f8fc809f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 shutdown_connections 2024-08-19T09:55:20.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 >> 172.21.15.1:0/524815925 conn(0x7f8fc8068650 msgr2=0x7f8fc81bd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:20.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 shutdown_connections 2024-08-19T09:55:20.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 -- 172.21.15.1:0/524815925 wait complete. 2024-08-19T09:55:20.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:20.926+0000 7f8fce660640 1 librados: shutdown 2024-08-19T09:55:20.938 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-08-19T09:55:20.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-08-19T09:55:20.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:55:21.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.037+0000 7f4503467640 10 monclient: get_monmap_and_config 2024-08-19T09:55:21.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.037+0000 7f4503467640 10 monclient: build_initial_monmap 2024-08-19T09:55:21.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.037+0000 7f4503467640 10 monclient: monmap: 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:55:21.038456+0000 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:55:21.038456+0000 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:21.040 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:21.040 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:21.040 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 1 Processor -- start 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 1 -- start start 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient: init 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient: _add_conns ranks=[3,8,7,6,4,2,0,1,5] 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient(hunting): picked mon.noname-g con 0x7f44fc0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient(hunting): picked mon.noname-e con 0x7f44fc0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient(hunting): picked mon.noname-i con 0x7f44fc0a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:21.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc0a3c70 con 0x7f44fc0a6830 2024-08-19T09:55:21.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc0a33d0 con 0x7f44fc0a46a0 2024-08-19T09:55:21.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc0a36b0 con 0x7f44fc0694b0 2024-08-19T09:55:21.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:21.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.038+0000 7f4503467640 10 monclient(hunting): authenticate will time out at 977.114793s 2024-08-19T09:55:21.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.039+0000 7f4500a02640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f44fc0694b0 0x7f44fc0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35762/0 (socket says 172.21.15.1:35762) 2024-08-19T09:55:21.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.039+0000 7f4500a02640 1 -- 172.21.15.1:0/236110521 learned_addr learned my addr 172.21.15.1:0/236110521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f4500a02640 1 -- 172.21.15.1:0/236110521 >> v1:172.21.15.1:6793/0 conn(0x7f44fc0694b0 legacy=0x7f44fc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f4501a04640 1 -- 172.21.15.1:0/236110521 >> v1:172.21.15.82:6792/0 conn(0x7f44fc0a46a0 legacy=0x7f44fc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f4501203640 1 -- 172.21.15.1:0/236110521 >> v1:172.21.15.82:6790/0 conn(0x7f44fc0a6830 legacy=0x7f44fc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128935447 0 0) 0x7f44fc0a3c70 con 0x7f44fc0a6830 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:21.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient(hunting): my global_id is 134742 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e4003520 con 0x7f44fc0a6830 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2324286124 0 0) 0x7f44e4003520 con 0x7f44fc0a6830 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 >> v1:172.21.15.1:6793/0 conn(0x7f44fc0694b0 legacy=0x7f44fc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 >> v1:172.21.15.82:6792/0 conn(0x7f44fc0a46a0 legacy=0x7f44fc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: _finish_hunting 0 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 monclient: found mon.noname-g 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc06dbb0 con 0x7f44fc0a6830 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: _finish_auth 0 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: _check_auth_tickets 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f44f4001bf0 con 0x7f44fc0a6830 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f4503467640 5 monclient: authenticate success, global_id 134742 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f44fbfff640 10 monclient: dump: 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.048 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.041+0000 7f4503467640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44f4003610 con 0x7f44fc0a6830 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f44fbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 10 monclient: get_monmap_and_config success 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f44fbfff640 1 -- 172.21.15.1:0/236110521 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f44f4003b80 con 0x7f44fc0a6830 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f44fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 4 set_mon_vals no callback set 2024-08-19T09:55:21.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f44fbfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f44fbfff640 10 monclient: dump: 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.050 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 10 monclient: shutdown 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 1 -- 172.21.15.1:0/236110521 >> v1:172.21.15.82:6790/0 conn(0x7f44fc0a6830 legacy=0x7f44fc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 1 -- 172.21.15.1:0/236110521 shutdown_connections 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 1 -- 172.21.15.1:0/236110521 >> 172.21.15.1:0/236110521 conn(0x7f44fc0a9020 msgr2=0x7f44fc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 1 -- 172.21.15.1:0/236110521 shutdown_connections 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.042+0000 7f4503467640 1 -- 172.21.15.1:0/236110521 wait complete. 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 10 monclient: build_initial_monmap 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 10 monclient: monmap: 2024-08-19T09:55:21.051 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 1 librados: starting msgr at 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 1 librados: starting objecter 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 1 Processor -- start 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 1 -- start start 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 1 librados: setting wanted keys 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 1 librados: calling monclient init 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.043+0000 7f4503467640 10 monclient: init 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient: _add_conns ranks=[3,0,8,7,1,2,4,5,6] 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient(hunting): picked mon.noname-g con 0x7f44fc0694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient(hunting): picked mon.noname-a con 0x7f44fc0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:55:21.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient(hunting): picked mon.noname-e con 0x7f44fc0a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc142b20 con 0x7f44fc0a46a0 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc138740 con 0x7f44fc0694b0 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc138a20 con 0x7f44fc0a6830 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4503467640 10 monclient(hunting): authenticate will time out at 977.120183s 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4500a02640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f44fc0a46a0 legacy=0x7f44fc135360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4501a04640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f44fc0a6830 0x7f44fc1d6490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35770/0 (socket says 172.21.15.1:35770) 2024-08-19T09:55:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.044+0000 7f4501a04640 1 -- 172.21.15.1:0/1680688394 learned_addr learned my addr 172.21.15.1:0/1680688394 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f4501a04640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6793/0 conn(0x7f44fc0a6830 legacy=0x7f44fc1d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 client.?.objecter ms_handle_connect 0x7f44fc0a6830 2024-08-19T09:55:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 client.?.objecter ms_handle_connect 0x7f44fc0694b0 2024-08-19T09:55:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f4501203640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.82:6790/0 conn(0x7f44fc0694b0 legacy=0x7f44fc134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647307201 0 0) 0x7f44fc138740 con 0x7f44fc0694b0 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 monclient(hunting): my global_id is 134751 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e8004eb0 con 0x7f44fc0694b0 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 676816425 0 0) 0x7f44e8004eb0 con 0x7f44fc0694b0 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44fc138740 con 0x7f44fc0694b0 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f44f4001bb0 con 0x7f44fc0694b0 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.046+0000 7f44f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:55:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient(hunting): dump: 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 358559365 0 0) 0x7f44fc138740 con 0x7f44fc0694b0 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6793/0 conn(0x7f44fc0a6830 legacy=0x7f44fc1d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6789/0 conn(0x7f44fc0a46a0 legacy=0x7f44fc135360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 monclient: found mon.g 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc135a70 con 0x7f44fc0694b0 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: _finish_auth 0 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:55:21.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 5 monclient: authenticate success, global_id 134751 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 10 monclient: _renew_subs 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc137bb0 con 0x7f44fc0694b0 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44f4003be0 con 0x7f44fc0694b0 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f44f4004430 con 0x7f44fc0694b0 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44f9ffb640 10 monclient: dump: 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.061 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 10 client.134751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 10 monclient: _renew_subs 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f44fb7fe640 4 set_mon_vals no callback set 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f44fc136710 con 0x7f44fc0694b0 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.047+0000 7f4503467640 1 librados: init done 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f4503467640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f44fc051b00 con 0x7f44fc137220 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f44f4011a80 con 0x7f44fc0694b0 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(198..198 src has 1..198) ==== 6177+0+0 (unknown 120415132 0 0) 0x7f44f404e5d0 con 0x7f44fc0694b0 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f44f9ffb640 10 client.134751.objecter ms_dispatch 0x7f44fc000b90 osd_map(198..198 src has 1..198) 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f44f9ffb640 3 client.134751.objecter handle_osd_map got epochs [198,198] > 0 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f44f9ffb640 3 client.134751.objecter handle_osd_map decoding full epoch 198 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f44f9ffb640 20 client.134751.objecter dump_active .. 0 homeless 2024-08-19T09:55:21.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.048+0000 7f4501a04640 10 monclient: get_auth_request con 0x7f44e8040db0 auth_method 0 2024-08-19T09:55:21.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.050+0000 7f4500a02640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6791/0 conn(0x7f44fc137220 legacy=0x7f44fc09e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.050+0000 7f44f9ffb640 10 client.134751.objecter ms_handle_connect 0x7f44fc137220 2024-08-19T09:55:21.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.050+0000 7f44f9ffb640 10 client.134751.objecter resend_mon_ops 2024-08-19T09:55:21.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.051+0000 7f44f9ffb640 10 client.134751.objecter ms_handle_connect 0x7f44e8040db0 2024-08-19T09:55:21.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.165+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237183549 0 0) 0x7f44fc051b00 con 0x7f44fc137220 2024-08-19T09:55:21.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.165+0000 7f44f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.165+0000 7f44f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.165+0000 7f44f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.165+0000 7f44f9ffb640 10 monclient(hunting): my global_id is 27004 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.165+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44fc138740 con 0x7f44fc137220 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3524501038 0 0) 0x7f44fc138740 con 0x7f44fc137220 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44fc051b00 con 0x7f44fc137220 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f44f00019c0 con 0x7f44fc137220 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 10 monclient: dump: 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.169 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3196056505 0 0) 0x7f44fc051b00 con 0x7f44fc137220 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44fc137600 con 0x7f44fc137220 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f44f00058d0 con 0x7f44fc137220 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6791/0 conn(0x7f44fc137220 legacy=0x7f44fc09e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.166+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6791/0 conn(0x7f44fc137220 legacy=0x7f44fc09e490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.173+0000 7f4503467640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:55:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.173+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f44fc072fb0 con 0x7f44fc02e1d0 2024-08-19T09:55:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.175+0000 7f44f9ffb640 10 client.134751.objecter ms_handle_connect 0x7f44fc02e1d0 2024-08-19T09:55:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.175+0000 7f4500a02640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6791/0 conn(0x7f44fc02e1d0 legacy=0x7f44fc09eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.175+0000 7f44f9ffb640 10 client.134751.objecter resend_mon_ops 2024-08-19T09:55:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.279+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276648658 0 0) 0x7f44fc072fb0 con 0x7f44fc02e1d0 2024-08-19T09:55:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.279+0000 7f44f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:55:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.279+0000 7f44f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.279+0000 7f44f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.279+0000 7f44f9ffb640 10 monclient(hunting): my global_id is 27013 2024-08-19T09:55:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.279+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44fc051b00 con 0x7f44fc02e1d0 2024-08-19T09:55:21.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2559870201 0 0) 0x7f44fc051b00 con 0x7f44fc02e1d0 2024-08-19T09:55:21.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44fc072fb0 con 0x7f44fc02e1d0 2024-08-19T09:55:21.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f44f0004e50 con 0x7f44fc02e1d0 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 10 monclient: dump: 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.283 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 204843483 0 0) 0x7f44fc072fb0 con 0x7f44fc02e1d0 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.280+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44fc1381e0 con 0x7f44fc02e1d0 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.281+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7807 (unknown 0 0 3665996600) 0x7f44f0005a80 con 0x7f44fc02e1d0 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.281+0000 7f44f9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.281+0000 7f44f9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.281+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6791/0 conn(0x7f44fc02e1d0 legacy=0x7f44fc09eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.281+0000 7f44f9ffb640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.1:6791/0 conn(0x7f44fc02e1d0 legacy=0x7f44fc09eba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 327551, 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:55:21.285 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:21.286 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:21.287 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:55:21.288 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:21.289 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.290 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:55:21.292 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:55:21.295 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.296 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.298 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:55:21.299 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:55:21.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:55:21.302 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:21.303 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:21.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:21.305 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:55:21.305 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:55:21.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 10 librados: watch_flush enter 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 10 librados: watch_flush exit 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 20 client.134751.objecter shutdown clearing up homeless session... 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 10 client.134751.objecter successfully canceled tick 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.82:6812/2549104415 conn(0x7f44e8040db0 legacy=0x7f44e8043270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 10 monclient: shutdown 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 >> v1:172.21.15.82:6790/0 conn(0x7f44fc0694b0 legacy=0x7f44fc134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4501a04640 1 -- 172.21.15.1:0/1680688394 reap_dead start 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4501a04640 1 -- 172.21.15.1:0/1680688394 reap_dead start 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 shutdown_connections 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 >> 172.21.15.1:0/1680688394 conn(0x7f44fc0a9020 msgr2=0x7f44fc0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 shutdown_connections 2024-08-19T09:55:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 -- 172.21.15.1:0/1680688394 wait complete. 2024-08-19T09:55:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.282+0000 7f4503467640 1 librados: shutdown 2024-08-19T09:55:21.308 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:55:21.308 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon e 2024-08-19T09:55:21.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:55:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.407+0000 7f6084084640 10 monclient: get_monmap_and_config 2024-08-19T09:55:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.407+0000 7f6084084640 10 monclient: build_initial_monmap 2024-08-19T09:55:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.407+0000 7f6084084640 10 monclient: monmap: 2024-08-19T09:55:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:55:21.408413+0000 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:55:21.408413+0000 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:21.410 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 1 Processor -- start 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 1 -- start start 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient: init 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient: _add_conns ranks=[5,1,8,6,0,2,4,3,7] 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient(hunting): picked mon.noname-h con 0x7f607c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient(hunting): picked mon.noname-f con 0x7f607c0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:55:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient(hunting): picked mon.noname-e con 0x7f607c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c0a3b40 con 0x7f607c0a6ba0 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c0a8130 con 0x7f607c0a67c0 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c06ede0 con 0x7f607c13f5c0 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6084084640 10 monclient(hunting): authenticate will time out at 977.484854s 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f608161f640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f607c0a6ba0 legacy=0x7f607c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6082621640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f607c13f5c0 0x7f607c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35786/0 (socket says 172.21.15.1:35786) 2024-08-19T09:55:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.408+0000 7f6082621640 1 -- 172.21.15.1:0/35318520 learned_addr learned my addr 172.21.15.1:0/35318520 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6082621640 1 -- 172.21.15.1:0/35318520 >> v1:172.21.15.1:6793/0 conn(0x7f607c13f5c0 legacy=0x7f607c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824481439 0 0) 0x7f607c06ede0 con 0x7f607c13f5c0 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient(hunting): _negotiate 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient(hunting): my global_id is 27548 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6064003520 con 0x7f607c13f5c0 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6081e20640 1 -- 172.21.15.1:0/35318520 >> v1:172.21.15.82:6791/0 conn(0x7f607c0a67c0 legacy=0x7f607c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 332572626 0 0) 0x7f6064003520 con 0x7f607c13f5c0 2024-08-19T09:55:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 >> v1:172.21.15.82:6791/0 conn(0x7f607c0a67c0 legacy=0x7f607c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 >> v1:172.21.15.82:6789/0 conn(0x7f607c0a6ba0 legacy=0x7f607c0a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: _finish_hunting 0 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 monclient: found mon.noname-e 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c0a92a0 con 0x7f607c13f5c0 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: _finish_auth 0 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: _check_auth_tickets 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6074001e50 con 0x7f607c13f5c0 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:55:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: dump: 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6084084640 5 monclient: authenticate success, global_id 27548 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6084084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6074004c20 con 0x7f607c13f5c0 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 1 -- 172.21.15.1:0/35318520 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6074005190 con 0x7f607c13f5c0 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6084084640 10 monclient: get_monmap_and_config success 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6084084640 4 set_mon_vals no callback set 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:55:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.411+0000 7f6080e1e640 10 monclient: dump: 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 10 monclient: shutdown 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 1 -- 172.21.15.1:0/35318520 >> v1:172.21.15.1:6793/0 conn(0x7f607c13f5c0 legacy=0x7f607c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 1 -- 172.21.15.1:0/35318520 shutdown_connections 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 1 -- 172.21.15.1:0/35318520 >> 172.21.15.1:0/35318520 conn(0x7f607c0ac4b0 msgr2=0x7f607c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 1 -- 172.21.15.1:0/35318520 shutdown_connections 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 1 -- 172.21.15.1:0/35318520 wait complete. 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 10 monclient: build_initial_monmap 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.412+0000 7f6084084640 10 monclient: monmap: 2024-08-19T09:55:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 librados: starting msgr at 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 librados: starting objecter 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 Processor -- start 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 -- start start 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 librados: setting wanted keys 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 librados: calling monclient init 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient: init 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient: _add_conns ranks=[2,7,1,6,3,0,4,5,8] 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient(hunting): picked mon.noname-b con 0x7f607c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient(hunting): picked mon.noname-i con 0x7f607c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:21.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient(hunting): picked mon.noname-f con 0x7f607c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c13a6f0 con 0x7f607c13f5c0 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6081e20640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f607c0a67c0 legacy=0x7f607c09ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c13a9d0 con 0x7f607c0a67c0 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c139ce0 con 0x7f607c0a6ba0 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6084084640 10 monclient(hunting): authenticate will time out at 977.489712s 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f6082621640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f607c13f5c0 legacy=0x7f607c1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f608161f640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f607c0a6ba0 0x7f607c0a0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:32976/0 (socket says 172.21.15.1:32976) 2024-08-19T09:55:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.413+0000 7f608161f640 1 -- 172.21.15.1:0/2598851283 learned_addr learned my addr 172.21.15.1:0/2598851283 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:21.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f608161f640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6792/0 conn(0x7f607c0a6ba0 legacy=0x7f607c0a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:21.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 client.?.objecter ms_handle_connect 0x7f607c0a6ba0 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346140003 0 0) 0x7f607c139ce0 con 0x7f607c0a6ba0 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): my global_id is 144232 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6068004eb0 con 0x7f607c0a6ba0 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f608161f640 1 --1- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6792/0 conn(0x7f607c0a6ba0 0x7f607c0a0620 :-1 s=OPENED pgs=156 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f6068004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3307692359 0 0) 0x7f6068004eb0 con 0x7f607c0a6ba0 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f607c139ce0 con 0x7f607c0a6ba0 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6070004670 con 0x7f607c0a6ba0 2024-08-19T09:55:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): dump: 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:21.428 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:21.428 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:21.428 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:21.428 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:21.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.416+0000 7f607a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:55:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.614+0000 7f6081e20640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6790/0 conn(0x7f607c0a67c0 legacy=0x7f607c09ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:21.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:21.614+0000 7f6082621640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6789/0 conn(0x7f607c13f5c0 legacy=0x7f607c1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.015+0000 7f6081e20640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6790/0 conn(0x7f607c0a67c0 legacy=0x7f607c09ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:22.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.015+0000 7f6082621640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6789/0 conn(0x7f607c13f5c0 legacy=0x7f607c1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3593423230 0 0) 0x7f607c139ce0 con 0x7f607c0a6ba0 2024-08-19T09:55:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6790/0 conn(0x7f607c0a67c0 legacy=0x7f607c09ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6789/0 conn(0x7f607c13f5c0 legacy=0x7f607c1d6250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 monclient: found mon.i 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c0a28a0 con 0x7f607c0a6ba0 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 5 monclient: authenticate success, global_id 144232 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 10 monclient: _renew_subs 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f607c0a1640 con 0x7f607c0a6ba0 2024-08-19T09:55:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 10 client.144232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 10 monclient: _renew_subs 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f607c13f320 con 0x7f607c0a6ba0 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 1 librados: init done 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6070004980 con 0x7f607c0a6ba0 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6070004e70 con 0x7f607c0a6ba0 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 10 monclient: dump: 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.043 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607bfff640 4 set_mon_vals no callback set 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f6084084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.040+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6070005010 con 0x7f607c0a6ba0 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f605c0065c0 con 0x7f605c002860 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(199..199 src has 1..199) ==== 6174+0+0 (unknown 3706399979 0 0) 0x7f607004e620 con 0x7f607c0a6ba0 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f607a7fc640 10 client.144232.objecter ms_dispatch 0x7f607c000b90 osd_map(199..199 src has 1..199) 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f607a7fc640 3 client.144232.objecter handle_osd_map got epochs [199,199] > 0 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f607a7fc640 3 client.144232.objecter handle_osd_map decoding full epoch 199 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f607a7fc640 20 client.144232.objecter dump_active .. 0 homeless 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.041+0000 7f6082621640 10 monclient: get_auth_request con 0x7f6068040d50 auth_method 0 2024-08-19T09:55:22.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f6081e20640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6793/0 conn(0x7f605c002860 legacy=0x7f605c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 client.144232.objecter ms_handle_connect 0x7f605c002860 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 client.144232.objecter resend_mon_ops 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805874122 0 0) 0x7f605c0065c0 con 0x7f605c002860 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient(hunting): my global_id is 27557 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f607c139ce0 con 0x7f605c002860 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2978645973 0 0) 0x7f607c139ce0 con 0x7f605c002860 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f605c0065c0 con 0x7f605c002860 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f606c001cc0 con 0x7f605c002860 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:55:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.043+0000 7f607a7fc640 10 monclient: dump: 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 10 client.144232.objecter ms_handle_connect 0x7f6068040d50 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1266204735 0 0) 0x7f605c0065c0 con 0x7f605c002860 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f605c00a160 con 0x7f605c002860 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f606c0057c0 con 0x7f605c002860 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6793/0 conn(0x7f605c002860 legacy=0x7f605c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.044+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6793/0 conn(0x7f605c002860 legacy=0x7f605c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:22.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.051+0000 7f6084084640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:55:22.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.051+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f605c009e50 con 0x7f605c009110 2024-08-19T09:55:22.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 10 client.144232.objecter ms_handle_connect 0x7f605c009110 2024-08-19T09:55:22.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 10 client.144232.objecter resend_mon_ops 2024-08-19T09:55:22.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f6081e20640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6793/0 conn(0x7f605c009110 legacy=0x7f605c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754290049 0 0) 0x7f605c009e50 con 0x7f605c009110 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 10 monclient(hunting): my global_id is 27566 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.053+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f605c0065c0 con 0x7f605c009110 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3523924331 0 0) 0x7f605c0065c0 con 0x7f605c009110 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f605c009e50 con 0x7f605c009110 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f606c003d80 con 0x7f605c009110 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 10 monclient: dump: 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.057 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3942643660 0 0) 0x7f605c009e50 con 0x7f605c009110 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f605c00c3f0 con 0x7f605c009110 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7640 (unknown 0 0 326475578) 0x7f606c0040c0 con 0x7f605c009110 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6793/0 conn(0x7f605c009110 legacy=0x7f605c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f607a7fc640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.1:6793/0 conn(0x7f605c009110 legacy=0x7f605c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:22.058 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 328326, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.059 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.060 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.061 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:55:22.062 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:55:22.063 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.066 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.068 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:55:22.070 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.071 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.073 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:55:22.075 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:22.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.078 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:55:22.079 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.079 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.079 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:55:22.079 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.054+0000 7f6084084640 10 librados: watch_flush enter 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 10 librados: watch_flush exit 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 20 client.144232.objecter shutdown clearing up homeless session... 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 10 client.144232.objecter successfully canceled tick 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6068040d50 legacy=0x7f6068043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 10 monclient: shutdown 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 >> v1:172.21.15.82:6792/0 conn(0x7f607c0a6ba0 legacy=0x7f607c0a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6082621640 1 -- 172.21.15.1:0/2598851283 reap_dead start 2024-08-19T09:55:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6082621640 1 -- 172.21.15.1:0/2598851283 reap_dead start 2024-08-19T09:55:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 shutdown_connections 2024-08-19T09:55:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 >> 172.21.15.1:0/2598851283 conn(0x7f607c0ac4b0 msgr2=0x7f607c1389a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 shutdown_connections 2024-08-19T09:55:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 -- 172.21.15.1:0/2598851283 wait complete. 2024-08-19T09:55:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.055+0000 7f6084084640 1 librados: shutdown 2024-08-19T09:55:22.082 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:55:22.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-08-19T09:55:22.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.192+0000 7feb7253a640 10 monclient: get_monmap_and_config 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.192+0000 7feb7253a640 10 monclient: build_initial_monmap 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.192+0000 7feb7253a640 10 monclient: monmap: 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:55:22.193035+0000 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:55:22.193035+0000 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:22.194 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.192+0000 7feb7253a640 1 Processor -- start 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.192+0000 7feb7253a640 1 -- start start 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.192+0000 7feb7253a640 10 monclient: init 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient: _add_conns ranks=[2,7,0,3,8,4,6,1,5] 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient(hunting): picked mon.noname-b con 0x7feb6c0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient(hunting): picked mon.noname-i con 0x7feb6c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient(hunting): picked mon.noname-a con 0x7feb6c0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c0a3c70 con 0x7feb6c0a46a0 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb6bfff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7feb6c0a6830 legacy=0x7feb6c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c0a33d0 con 0x7feb6c0a6830 2024-08-19T09:55:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c0a36b0 con 0x7feb6c0694b0 2024-08-19T09:55:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb7253a640 10 monclient(hunting): authenticate will time out at 978.269556s 2024-08-19T09:55:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb70ad7640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7feb6c0a46a0 legacy=0x7feb6c141850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb6b7fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7feb6c0694b0 0x7feb6c0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:32984/0 (socket says 172.21.15.1:32984) 2024-08-19T09:55:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.193+0000 7feb6b7fe640 1 -- 172.21.15.1:0/3303089128 learned_addr learned my addr 172.21.15.1:0/3303089128 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.195+0000 7feb6b7fe640 1 -- 172.21.15.1:0/3303089128 >> v1:172.21.15.82:6792/0 conn(0x7feb6c0694b0 legacy=0x7feb6c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.394+0000 7feb6bfff640 1 -- 172.21.15.1:0/3303089128 >> v1:172.21.15.1:6790/0 conn(0x7feb6c0a6830 legacy=0x7feb6c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:22.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.394+0000 7feb70ad7640 1 -- 172.21.15.1:0/3303089128 >> v1:172.21.15.1:6789/0 conn(0x7feb6c0a46a0 legacy=0x7feb6c141850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:22.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.688+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249980781 0 0) 0x7feb6c0a36b0 con 0x7feb6c0694b0 2024-08-19T09:55:22.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.689+0000 7feb6affd640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.689+0000 7feb6affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.689+0000 7feb6affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.689+0000 7feb6affd640 10 monclient(hunting): my global_id is 144241 2024-08-19T09:55:22.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.689+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb54003520 con 0x7feb6c0694b0 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.697+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 263760397 0 0) 0x7feb54003520 con 0x7feb6c0694b0 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.697+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 >> v1:172.21.15.1:6790/0 conn(0x7feb6c0a6830 legacy=0x7feb6c0a6c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 >> v1:172.21.15.1:6789/0 conn(0x7feb6c0a46a0 legacy=0x7feb6c141850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: _finish_hunting 0 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 1 monclient: found mon.noname-i 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:55:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c06dbb0 con 0x7feb6c0694b0 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: _finish_auth 0 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: _check_auth_tickets 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feb60001ac0 con 0x7feb6c0694b0 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: dump: 2024-08-19T09:55:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb7253a640 5 monclient: authenticate success, global_id 144241 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb7253a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb600048a0 con 0x7feb6c0694b0 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb7253a640 10 monclient: get_monmap_and_config success 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb7253a640 4 set_mon_vals no callback set 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 1 -- 172.21.15.1:0/3303089128 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feb60004d90 con 0x7feb6c0694b0 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:55:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.698+0000 7feb6affd640 10 monclient: dump: 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.705 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.699+0000 7feb7253a640 10 monclient: shutdown 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.699+0000 7feb7253a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.699+0000 7feb7253a640 1 -- 172.21.15.1:0/3303089128 >> v1:172.21.15.82:6792/0 conn(0x7feb6c0694b0 legacy=0x7feb6c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.700+0000 7feb7253a640 1 -- 172.21.15.1:0/3303089128 shutdown_connections 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.700+0000 7feb7253a640 1 -- 172.21.15.1:0/3303089128 >> 172.21.15.1:0/3303089128 conn(0x7feb6c0a9020 msgr2=0x7feb6c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.700+0000 7feb7253a640 1 -- 172.21.15.1:0/3303089128 shutdown_connections 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.700+0000 7feb7253a640 1 -- 172.21.15.1:0/3303089128 wait complete. 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.701+0000 7feb7253a640 10 monclient: build_initial_monmap 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.701+0000 7feb7253a640 10 monclient: monmap: 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:55:22.711 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.701+0000 7feb7253a640 1 librados: starting msgr at 2024-08-19T09:55:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.701+0000 7feb7253a640 1 librados: starting objecter 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.702+0000 7feb7253a640 1 Processor -- start 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.702+0000 7feb7253a640 1 -- start start 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.702+0000 7feb7253a640 1 librados: setting wanted keys 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.702+0000 7feb7253a640 1 librados: calling monclient init 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.702+0000 7feb7253a640 10 monclient: init 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient: _add_conns ranks=[7,0,8,2,5,3,6,1,4] 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient(hunting): picked mon.noname-i con 0x7feb6c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient(hunting): picked mon.noname-a con 0x7feb6c0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient(hunting): picked mon.noname-e con 0x7feb6c0a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c12bab0 con 0x7feb6c0a46a0 2024-08-19T09:55:22.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c12da40 con 0x7feb6c0694b0 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb6b7fe640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7feb6c0a46a0 legacy=0x7feb6c12b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c0ab9a0 con 0x7feb6c0a6830 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb7253a640 10 monclient(hunting): authenticate will time out at 978.779595s 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb70ad7640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7feb6c0a6830 0x7feb6c1da490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35812/0 (socket says 172.21.15.1:35812) 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.703+0000 7feb70ad7640 1 -- 172.21.15.1:0/3150941592 learned_addr learned my addr 172.21.15.1:0/3150941592 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.705+0000 7feb70ad7640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.1:6793/0 conn(0x7feb6c0a6830 legacy=0x7feb6c1da490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.705+0000 7feb68ff9640 10 client.?.objecter ms_handle_connect 0x7feb6c0a6830 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.705+0000 7feb68ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.705+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550067522 0 0) 0x7feb6c0ab9a0 con 0x7feb6c0a6830 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): my global_id is 27575 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb58004eb0 con 0x7feb6c0a6830 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 client.?.objecter ms_handle_connect 0x7feb6c0694b0 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb6bfff640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6792/0 conn(0x7feb6c0694b0 legacy=0x7feb6c12ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1527300203 0 0) 0x7feb58004eb0 con 0x7feb6c0a6830 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb6c0ab9a0 con 0x7feb6c0a6830 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feb640049e0 con 0x7feb6c0a6830 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): dump: 2024-08-19T09:55:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.716 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195622180 0 0) 0x7feb6c12da40 con 0x7feb6c0694b0 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient(hunting): my global_id is 144250 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb58004eb0 con 0x7feb6c0694b0 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2580060822 0 0) 0x7feb6c0ab9a0 con 0x7feb6c0a6830 2024-08-19T09:55:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6792/0 conn(0x7feb6c0694b0 legacy=0x7feb6c12ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.1:6789/0 conn(0x7feb6c0a46a0 legacy=0x7feb6c12b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 monclient: found mon.e 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c142890 con 0x7feb6c0a6830 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient: _finish_auth 0 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 5 monclient: authenticate success, global_id 27575 2024-08-19T09:55:22.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 10 monclient: _renew_subs 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb6c12d480 con 0x7feb6c0a6830 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 10 client.27575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 10 monclient: _renew_subs 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7feb6c0a1e60 con 0x7feb6c0a6830 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb7253a640 1 librados: init done 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.706+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb64004c00 con 0x7feb6c0a6830 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.707+0000 7feb68ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.707+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feb64005170 con 0x7feb6c0a6830 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.707+0000 7feb68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.707+0000 7feb69ffb640 4 set_mon_vals no callback set 2024-08-19T09:55:22.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.707+0000 7feb68ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.707+0000 7feb68ff9640 10 monclient: dump: 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb7253a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feb4c0065c0 con 0x7feb4c002860 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7feb64005480 con 0x7feb6c0a6830 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(200..200 src has 1..200) ==== 5951+0+0 (unknown 2734063010 0 0) 0x7feb6404e7c0 con 0x7feb6c0a6830 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb68ff9640 10 client.27575.objecter ms_dispatch 0x7feb6c000b90 osd_map(200..200 src has 1..200) 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb68ff9640 3 client.27575.objecter handle_osd_map got epochs [200,200] > 0 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb68ff9640 3 client.27575.objecter handle_osd_map decoding full epoch 200 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb68ff9640 20 client.27575.objecter dump_active .. 0 homeless 2024-08-19T09:55:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.708+0000 7feb6b7fe640 10 monclient: get_auth_request con 0x7feb58040ed0 auth_method 0 2024-08-19T09:55:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb6bfff640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6790/0 conn(0x7feb4c002860 legacy=0x7feb4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 client.27575.objecter ms_handle_connect 0x7feb4c002860 2024-08-19T09:55:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 client.27575.objecter resend_mon_ops 2024-08-19T09:55:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2245058496 0 0) 0x7feb4c0065c0 con 0x7feb4c002860 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient(hunting): my global_id is 134760 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb6c0ab9a0 con 0x7feb4c002860 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3436831149 0 0) 0x7feb6c0ab9a0 con 0x7feb4c002860 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb4c0065c0 con 0x7feb4c002860 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 client.27575.objecter ms_handle_connect 0x7feb58040ed0 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feb5c0043f0 con 0x7feb4c002860 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.710+0000 7feb68ff9640 10 monclient: dump: 2024-08-19T09:55:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2478866984 0 0) 0x7feb4c0065c0 con 0x7feb4c002860 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb4c00a160 con 0x7feb4c002860 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7feb5c005950 con 0x7feb4c002860 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6790/0 conn(0x7feb4c002860 legacy=0x7feb4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.711+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6790/0 conn(0x7feb4c002860 legacy=0x7feb4c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.718+0000 7feb7253a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.718+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feb4c009e50 con 0x7feb4c009110 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb6bfff640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6790/0 conn(0x7feb4c009110 legacy=0x7feb4c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 client.27575.objecter ms_handle_connect 0x7feb4c009110 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 client.27575.objecter resend_mon_ops 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369068141 0 0) 0x7feb4c009e50 con 0x7feb4c009110 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient(hunting): my global_id is 134769 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb4c0065c0 con 0x7feb4c009110 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2122948013 0 0) 0x7feb4c0065c0 con 0x7feb4c009110 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb4c009e50 con 0x7feb4c009110 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feb5c004d30 con 0x7feb4c009110 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 10 monclient: dump: 2024-08-19T09:55:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.729 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3961735279 0 0) 0x7feb4c009e50 con 0x7feb4c009110 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.721+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb4c00c3f0 con 0x7feb4c009110 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7642 (unknown 0 0 3658712198) 0x7feb5c005be0 con 0x7feb4c009110 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb68ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb68ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6790/0 conn(0x7feb4c009110 legacy=0x7feb4c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb68ff9640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6790/0 conn(0x7feb4c009110 legacy=0x7feb4c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 179782, 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:55:22.731 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.732 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.733 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:55:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:55:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:55:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:55:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.743 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:55:22.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:55:22.748 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:55:22.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.750 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:55:22.751 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 10 librados: watch_flush enter 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 10 librados: watch_flush exit 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 20 client.27575.objecter shutdown clearing up homeless session... 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 10 client.27575.objecter successfully canceled tick 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.82:6812/2549104415 conn(0x7feb58040ed0 legacy=0x7feb58043390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 10 monclient: shutdown 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 >> v1:172.21.15.1:6793/0 conn(0x7feb6c0a6830 legacy=0x7feb6c1da490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb70ad7640 1 -- 172.21.15.1:0/3150941592 reap_dead start 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb70ad7640 1 -- 172.21.15.1:0/3150941592 reap_dead start 2024-08-19T09:55:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 shutdown_connections 2024-08-19T09:55:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 >> 172.21.15.1:0/3150941592 conn(0x7feb6c0a9020 msgr2=0x7feb6c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 shutdown_connections 2024-08-19T09:55:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 -- 172.21.15.1:0/3150941592 wait complete. 2024-08-19T09:55:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.722+0000 7feb7253a640 1 librados: shutdown 2024-08-19T09:55:22.754 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:55:22.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-08-19T09:55:22.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:55:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.858+0000 7f5235b9f640 10 monclient: get_monmap_and_config 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.858+0000 7f5235b9f640 10 monclient: build_initial_monmap 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.858+0000 7f5235b9f640 10 monclient: monmap: 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:55:22.859808+0000 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:55:22.859808+0000 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:22.861 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.859+0000 7f5235b9f640 1 Processor -- start 2024-08-19T09:55:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.859+0000 7f5235b9f640 1 -- start start 2024-08-19T09:55:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.859+0000 7f5235b9f640 10 monclient: init 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient: _add_conns ranks=[2,6,7,5,0,8,4,1,3] 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient(hunting): picked mon.noname-b con 0x7f52300a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient(hunting): picked mon.noname-d con 0x7f52300a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient(hunting): picked mon.noname-i con 0x7f523013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52300a3b40 con 0x7f52300a67c0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52300a8130 con 0x7f52300a6ba0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f523006ede0 con 0x7f523013f5c0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f5235b9f640 10 monclient(hunting): authenticate will time out at 978.936161s 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f522f7fe640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f52300a67c0 legacy=0x7f52300a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f522effd640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f52300a6ba0 legacy=0x7f52300a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f522ffff640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f523013f5c0 0x7f52301419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:33012/0 (socket says 172.21.15.1:33012) 2024-08-19T09:55:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.860+0000 7f522ffff640 1 -- 172.21.15.1:0/378848800 learned_addr learned my addr 172.21.15.1:0/378848800 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522ffff640 1 -- 172.21.15.1:0/378848800 >> v1:172.21.15.82:6792/0 conn(0x7f523013f5c0 legacy=0x7f52301419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82377048 0 0) 0x7f523006ede0 con 0x7f523013f5c0 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522e7fc640 10 monclient(hunting): my global_id is 144259 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.862+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5218003520 con 0x7f523013f5c0 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2906510601 0 0) 0x7f5218003520 con 0x7f523013f5c0 2024-08-19T09:55:22.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 >> v1:172.21.15.1:6792/0 conn(0x7f52300a6ba0 legacy=0x7f52300a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 >> v1:172.21.15.1:6790/0 conn(0x7f52300a67c0 legacy=0x7f52300a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 monclient: found mon.noname-i 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52300a92a0 con 0x7f523013f5c0 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5228001ad0 con 0x7f523013f5c0 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 5 monclient: authenticate success, global_id 144259 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: dump: 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52280048a0 con 0x7f523013f5c0 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 1 -- 172.21.15.1:0/378848800 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5228004e10 con 0x7f523013f5c0 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 10 monclient: get_monmap_and_config success 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 4 set_mon_vals no callback set 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f522e7fc640 10 monclient: dump: 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 10 monclient: shutdown 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 1 -- 172.21.15.1:0/378848800 >> v1:172.21.15.82:6792/0 conn(0x7f523013f5c0 legacy=0x7f52301419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 1 -- 172.21.15.1:0/378848800 shutdown_connections 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 1 -- 172.21.15.1:0/378848800 >> 172.21.15.1:0/378848800 conn(0x7f52300ac4b0 msgr2=0x7f52300a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 1 -- 172.21.15.1:0/378848800 shutdown_connections 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.863+0000 7f5235b9f640 1 -- 172.21.15.1:0/378848800 wait complete. 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 10 monclient: build_initial_monmap 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 10 monclient: monmap: 2024-08-19T09:55:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 1 librados: starting msgr at 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 1 librados: starting objecter 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 1 Processor -- start 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 1 -- start start 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 1 librados: setting wanted keys 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 1 librados: calling monclient init 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.864+0000 7f5235b9f640 10 monclient: init 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient: _add_conns ranks=[1,8,4,3,5,0,7,6,2] 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient(hunting): picked mon.noname-f con 0x7f52300a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient(hunting): picked mon.noname-e con 0x7f52300a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f523013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:55:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52300a1430 con 0x7f52300a67c0 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52301429d0 con 0x7f523013f5c0 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5230139d00 con 0x7f52300a6ba0 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f522f7fe640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f52300a67c0 legacy=0x7f523013ea00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f5235b9f640 10 monclient(hunting): authenticate will time out at 978.941314s 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f522effd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f52300a6ba0 0x7f52301d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35828/0 (socket says 172.21.15.1:35828) 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f522effd640 1 -- 172.21.15.1:0/115040209 learned_addr learned my addr 172.21.15.1:0/115040209 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.865+0000 7f522ffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f523013f5c0 0x7f52301d62b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48910/0 (socket says 172.21.15.1:48910) 2024-08-19T09:55:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f522effd640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.1:6793/0 conn(0x7f52300a6ba0 legacy=0x7f52301d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f522ffff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.1:6791/0 conn(0x7f523013f5c0 legacy=0x7f52301d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 client.?.objecter ms_handle_connect 0x7f52300a6ba0 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 client.?.objecter ms_handle_connect 0x7f523013f5c0 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031821597 0 0) 0x7f5230139d00 con 0x7f52300a6ba0 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): my global_id is 27584 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f521c004eb0 con 0x7f52300a6ba0 2024-08-19T09:55:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 853078991 0 0) 0x7f521c004eb0 con 0x7f52300a6ba0 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5230139d00 con 0x7f52300a6ba0 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f52240046a0 con 0x7f52300a6ba0 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.867+0000 7f5217fff640 10 monclient(hunting): dump: 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.881 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2594077397 0 0) 0x7f5230139d00 con 0x7f52300a6ba0 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.1:6791/0 conn(0x7f523013f5c0 legacy=0x7f52301d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6789/0 conn(0x7f52300a67c0 legacy=0x7f523013ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:55:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: _finish_hunting 0 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 monclient: found mon.e 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5230139fe0 con 0x7f52300a6ba0 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: _finish_auth 0 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: _check_auth_tickets 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 5 monclient: authenticate success, global_id 27584 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 10 monclient: _renew_subs 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52301cdaa0 con 0x7f52300a6ba0 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 10 client.27584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 10 monclient: _renew_subs 2024-08-19T09:55:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f52301ce260 con 0x7f52300a6ba0 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 1 librados: init done 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52240049b0 con 0x7f52300a6ba0 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5224004f20 con 0x7f52300a6ba0 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f522dffb640 4 set_mon_vals no callback set 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5217fff640 10 monclient: dump: 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.868+0000 7f5235b9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f52100065c0 con 0x7f5210002860 2024-08-19T09:55:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f52240050c0 con 0x7f52300a6ba0 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(200..200 src has 1..200) ==== 5951+0+0 (unknown 2734063010 0 0) 0x7f522404e500 con 0x7f52300a6ba0 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5217fff640 10 client.27584.objecter ms_dispatch 0x7f5230000b90 osd_map(200..200 src has 1..200) 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5217fff640 3 client.27584.objecter handle_osd_map got epochs [200,200] > 0 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5217fff640 3 client.27584.objecter handle_osd_map decoding full epoch 200 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f5217fff640 20 client.27584.objecter dump_active .. 0 homeless 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.869+0000 7f522ffff640 10 monclient: get_auth_request con 0x7f521c040e60 auth_method 0 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f522f7fe640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6791/0 conn(0x7f5210002860 legacy=0x7f5210004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 client.27584.objecter ms_handle_connect 0x7f5210002860 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 client.27584.objecter resend_mon_ops 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432211531 0 0) 0x7f52100065c0 con 0x7f5210002860 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient(hunting): my global_id is 26753 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5230139d00 con 0x7f5210002860 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2062273157 0 0) 0x7f5230139d00 con 0x7f5210002860 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52100065c0 con 0x7f5210002860 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f52200047f0 con 0x7f5210002860 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 10 monclient: dump: 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.871+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2938714412 0 0) 0x7f52100065c0 con 0x7f5210002860 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f521000a160 con 0x7f5210002860 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 10 client.27584.objecter ms_handle_connect 0x7f521c040e60 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5220006b30 con 0x7f5210002860 2024-08-19T09:55:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:55:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6791/0 conn(0x7f5210002860 legacy=0x7f5210004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.872+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6791/0 conn(0x7f5210002860 legacy=0x7f5210004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.878+0000 7f5235b9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:55:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.878+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5210009e70 con 0x7f5210009130 2024-08-19T09:55:22.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f522f7fe640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6791/0 conn(0x7f5210009130 legacy=0x7f52100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:22.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 client.27584.objecter ms_handle_connect 0x7f5210009130 2024-08-19T09:55:22.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 client.27584.objecter resend_mon_ops 2024-08-19T09:55:22.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594395690 0 0) 0x7f5210009e70 con 0x7f5210009130 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient(hunting): _negotiate 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient(hunting): my global_id is 26762 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52100065c0 con 0x7f5210009130 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2303035895 0 0) 0x7f52100065c0 con 0x7f5210009130 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5210009e70 con 0x7f5210009130 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5220005150 con 0x7f5210009130 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 10 monclient: dump: 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:22.892 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 501654908 0 0) 0x7f5210009e70 con 0x7f5210009130 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.881+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f521000c410 con 0x7f5210009130 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7640 (unknown 0 0 4148773837) 0x7f5220006640 con 0x7f5210009130 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5217fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5217fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6791/0 conn(0x7f5210009130 legacy=0x7f52100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5217fff640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6791/0 conn(0x7f5210009130 legacy=0x7f52100095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:55:22.893 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 329147, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:22.894 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.895 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:55:22.897 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:22.898 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:55:22.901 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:55:22.902 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.903 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.904 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.905 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.906 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:55:22.907 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:55:22.908 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:55:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:22.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:22.914 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:55:22.914 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:55:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 10 librados: watch_flush enter 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 10 librados: watch_flush exit 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 20 client.27584.objecter shutdown clearing up homeless session... 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 10 client.27584.objecter successfully canceled tick 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.82:6812/2549104415 conn(0x7f521c040e60 legacy=0x7f521c043320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 10 monclient: shutdown 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 >> v1:172.21.15.1:6793/0 conn(0x7f52300a6ba0 legacy=0x7f52301d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f522ffff640 1 -- 172.21.15.1:0/115040209 reap_dead start 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f522ffff640 1 -- 172.21.15.1:0/115040209 reap_dead start 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 shutdown_connections 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.882+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 >> 172.21.15.1:0/115040209 conn(0x7f52300ac4b0 msgr2=0x7f52300aa160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.883+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 shutdown_connections 2024-08-19T09:55:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.883+0000 7f5235b9f640 1 -- 172.21.15.1:0/115040209 wait complete. 2024-08-19T09:55:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:22.883+0000 7f5235b9f640 1 librados: shutdown 2024-08-19T09:55:22.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:55:22.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-08-19T09:55:22.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:55:23.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.015+0000 7fcf1a2fd640 10 monclient: get_monmap_and_config 2024-08-19T09:55:23.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.015+0000 7fcf1a2fd640 10 monclient: build_initial_monmap 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.015+0000 7fcf1a2fd640 10 monclient: monmap: 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:55:23.016519+0000 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:55:23.016519+0000 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:23.018 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 1 Processor -- start 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 1 -- start start 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient: init 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient: _add_conns ranks=[8,4,7,3,5,2,6,1,0] 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient(hunting): picked mon.noname-e con 0x7fcf140a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient(hunting): picked mon.noname-c con 0x7fcf140a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient(hunting): picked mon.noname-i con 0x7fcf1413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf140a3b40 con 0x7fcf140a6ba0 2024-08-19T09:55:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf140a8130 con 0x7fcf1413f5c0 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1406ede0 con 0x7fcf140a67c0 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.016+0000 7fcf1a2fd640 10 monclient(hunting): authenticate will time out at 979.092994s 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.017+0000 7fcf13fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcf140a67c0 0x7fcf140a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35836/0 (socket says 172.21.15.1:35836) 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.017+0000 7fcf137fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fcf140a6ba0 0x7fcf140a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48918/0 (socket says 172.21.15.1:48918) 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.017+0000 7fcf13fff640 1 -- 172.21.15.1:0/1476017669 learned_addr learned my addr 172.21.15.1:0/1476017669 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf137fe640 1 -- 172.21.15.1:0/1476017669 >> v1:172.21.15.1:6791/0 conn(0x7fcf140a6ba0 legacy=0x7fcf140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf13fff640 1 -- 172.21.15.1:0/1476017669 >> v1:172.21.15.1:6793/0 conn(0x7fcf140a67c0 legacy=0x7fcf140a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf1889a640 1 -- 172.21.15.1:0/1476017669 >> v1:172.21.15.82:6792/0 conn(0x7fcf1413f5c0 legacy=0x7fcf141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3540951433 0 0) 0x7fcf140a3b40 con 0x7fcf140a6ba0 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): my global_id is 27022 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcefc003520 con 0x7fcf140a6ba0 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3173691860 0 0) 0x7fcf1406ede0 con 0x7fcf140a67c0 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): my global_id is 27593 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf140a3b40 con 0x7fcf140a67c0 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887769063 0 0) 0x7fcf140a8130 con 0x7fcf1413f5c0 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient(hunting): my global_id is 144268 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1406ede0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4204285043 0 0) 0x7fcefc003520 con 0x7fcf140a6ba0 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 >> v1:172.21.15.1:6793/0 conn(0x7fcf140a67c0 legacy=0x7fcf140a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 >> v1:172.21.15.82:6792/0 conn(0x7fcf1413f5c0 legacy=0x7fcf141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 monclient: found mon.noname-c 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf140a92a0 con 0x7fcf140a6ba0 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: _finish_auth 0 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:55:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf08001ae0 con 0x7fcf140a6ba0 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf1a2fd640 5 monclient: authenticate success, global_id 27022 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf1a2fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: dump: 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf1a2fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf080047e0 con 0x7fcf140a6ba0 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 1 -- 172.21.15.1:0/1476017669 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf08004d50 con 0x7fcf140a6ba0 2024-08-19T09:55:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf1a2fd640 10 monclient: get_monmap_and_config success 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf1a2fd640 4 set_mon_vals no callback set 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.019+0000 7fcf12ffd640 10 monclient: dump: 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.030 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 10 monclient: shutdown 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1476017669 >> v1:172.21.15.1:6791/0 conn(0x7fcf140a6ba0 legacy=0x7fcf140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1476017669 shutdown_connections 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1476017669 >> 172.21.15.1:0/1476017669 conn(0x7fcf140ac4b0 msgr2=0x7fcf140a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1476017669 shutdown_connections 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1476017669 wait complete. 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 10 monclient: build_initial_monmap 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.020+0000 7fcf1a2fd640 10 monclient: monmap: 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:55:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 librados: starting msgr at 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 librados: starting objecter 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 Processor -- start 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 -- start start 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 librados: setting wanted keys 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 librados: calling monclient init 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient: init 2024-08-19T09:55:23.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient: _reopen_session rank -1 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient: _add_conns ranks=[8,3,4,6,5,2,1,7,0] 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient(hunting): picked mon.noname-e con 0x7fcf140a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient(hunting): picked mon.noname-g con 0x7fcf140a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1413a3a0 con 0x7fcf140a6ba0 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1413a680 con 0x7fcf1413f5c0 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf140a1e70 con 0x7fcf140a67c0 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient(hunting): _renew_subs 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf1a2fd640 10 monclient(hunting): authenticate will time out at 979.097901s 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf13fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcf140a67c0 0x7fcf1413ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35846/0 (socket says 172.21.15.1:35846) 2024-08-19T09:55:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.021+0000 7fcf13fff640 1 -- 172.21.15.1:0/1753796162 learned_addr learned my addr 172.21.15.1:0/1753796162 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:55:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.024+0000 7fcf1889a640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.1:6791/0 conn(0x7fcf1413f5c0 legacy=0x7fcf141d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.024+0000 7fcf13fff640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.1:6793/0 conn(0x7fcf140a67c0 legacy=0x7fcf1413ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.024+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.024+0000 7fcf137fe640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6790/0 conn(0x7fcf140a6ba0 legacy=0x7fcf141d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 client.?.objecter ms_handle_connect 0x7fcf1413f5c0 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 client.?.objecter ms_handle_connect 0x7fcf140a67c0 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 client.?.objecter ms_handle_connect 0x7fcf140a6ba0 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889838348 0 0) 0x7fcf1413a680 con 0x7fcf1413f5c0 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): my global_id is 27031 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf00004eb0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4271285095 0 0) 0x7fcf140a1e70 con 0x7fcf140a67c0 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): my global_id is 27602 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1413a680 con 0x7fcf140a67c0 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943056806 0 0) 0x7fcf1413a3a0 con 0x7fcf140a6ba0 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): my global_id is 134778 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf140a1e70 con 0x7fcf140a6ba0 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2490019211 0 0) 0x7fcf00004eb0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf1413a3a0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf0c002020 con 0x7fcf1413f5c0 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): dump: 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:55:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1089380040 0 0) 0x7fcf1413a680 con 0x7fcf140a67c0 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf00004eb0 con 0x7fcf140a67c0 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf04001960 con 0x7fcf140a67c0 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): dump: 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1510907178 0 0) 0x7fcf140a1e70 con 0x7fcf140a6ba0 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf1413a680 con 0x7fcf140a6ba0 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf08004cd0 con 0x7fcf140a6ba0 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.026+0000 7fcf10ff9640 10 monclient(hunting): dump: 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.045 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.046 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1969294652 0 0) 0x7fcf1413a3a0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.1:6793/0 conn(0x7fcf140a67c0 legacy=0x7fcf1413ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6790/0 conn(0x7fcf140a6ba0 legacy=0x7fcf141d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 monclient: found mon.c 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf14142ca0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: _finish_auth 0 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 5 monclient: authenticate success, global_id 27031 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 10 monclient: _renew_subs 2024-08-19T09:55:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1413a0c0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 10 client.27031.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 10 monclient: _renew_subs 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf141cda50 con 0x7fcf1413f5c0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 1 librados: init done 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf0c003c40 con 0x7fcf1413f5c0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf0c0041b0 con 0x7fcf1413f5c0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf127fc640 4 set_mon_vals no callback set 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:55:23.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 10 monclient: dump: 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcef40065c0 con 0x7fcef4002860 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.027+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcf0c004350 con 0x7fcf1413f5c0 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.028+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(200..200 src has 1..200) ==== 5951+0+0 (unknown 2734063010 0 0) 0x7fcf0c04e670 con 0x7fcf1413f5c0 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.028+0000 7fcf10ff9640 10 client.27031.objecter ms_dispatch 0x7fcf14000b90 osd_map(200..200 src has 1..200) 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.028+0000 7fcf10ff9640 3 client.27031.objecter handle_osd_map got epochs [200,200] > 0 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.028+0000 7fcf10ff9640 3 client.27031.objecter handle_osd_map decoding full epoch 200 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.028+0000 7fcf10ff9640 20 client.27031.objecter dump_active .. 0 homeless 2024-08-19T09:55:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.028+0000 7fcf137fe640 10 monclient: get_auth_request con 0x7fcf000419a0 auth_method 0 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf13fff640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6792/0 conn(0x7fcef4002860 legacy=0x7fcef4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 client.27031.objecter ms_handle_connect 0x7fcef4002860 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 client.27031.objecter resend_mon_ops 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1256619470 0 0) 0x7fcef40065c0 con 0x7fcef4002860 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient(hunting): my global_id is 144277 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1413a3a0 con 0x7fcef4002860 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 client.27031.objecter ms_handle_connect 0x7fcf000419a0 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3485035342 0 0) 0x7fcf1413a3a0 con 0x7fcef4002860 2024-08-19T09:55:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcef40065c0 con 0x7fcef4002860 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf04005050 con 0x7fcef4002860 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 10 monclient: dump: 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3040772017 0 0) 0x7fcef40065c0 con 0x7fcef4002860 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.030+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcef400a160 con 0x7fcef4002860 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.031+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcf04005210 con 0x7fcef4002860 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.031+0000 7fcf10ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.031+0000 7fcf10ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:55:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.031+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6792/0 conn(0x7fcef4002860 legacy=0x7fcef4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.031+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6792/0 conn(0x7fcef4002860 legacy=0x7fcef4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.037+0000 7fcf1a2fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:55:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.037+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcef4009e50 con 0x7fcef4009110 2024-08-19T09:55:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.039+0000 7fcf13fff640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6792/0 conn(0x7fcef4009110 legacy=0x7fcef40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.039+0000 7fcf10ff9640 10 client.27031.objecter ms_handle_connect 0x7fcef4009110 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.039+0000 7fcf10ff9640 10 client.27031.objecter resend_mon_ops 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267004474 0 0) 0x7fcef4009e50 con 0x7fcef4009110 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient(hunting): my global_id is 144286 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcef40065c0 con 0x7fcef4009110 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2744602574 0 0) 0x7fcef40065c0 con 0x7fcef4009110 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcef4009e50 con 0x7fcef4009110 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcf04005ac0 con 0x7fcef4009110 2024-08-19T09:55:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient: dump: 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:55:23.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 519335752 0 0) 0x7fcef4009e50 con 0x7fcef4009110 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcef400c3a0 con 0x7fcef4009110 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7472 (unknown 0 0 125794156) 0x7fcf04007f50 con 0x7fcef4009110 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.040+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6792/0 conn(0x7fcef4009110 legacy=0x7fcef40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf10ff9640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6792/0 conn(0x7fcef4009110 legacy=0x7fcef40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 47420, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:55:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:23.060 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:23.061 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:55:23.062 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:55:23.063 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.064 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.068 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.069 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.072 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.073 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:55:23.074 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:55:23.076 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:55:23.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:55:23.078 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:55:23.079 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 10 librados: watch_flush enter 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 10 librados: watch_flush exit 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 20 client.27031.objecter shutdown clearing up homeless session... 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 10 client.27031.objecter successfully canceled tick 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcf000419a0 legacy=0x7fcf00043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 10 monclient: shutdown 2024-08-19T09:55:23.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 >> v1:172.21.15.1:6791/0 conn(0x7fcf1413f5c0 legacy=0x7fcf141d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1889a640 1 -- 172.21.15.1:0/1753796162 reap_dead start 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1889a640 1 -- 172.21.15.1:0/1753796162 reap_dead start 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 shutdown_connections 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 >> 172.21.15.1:0/1753796162 conn(0x7fcf140ac4b0 msgr2=0x7fcf140a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 shutdown_connections 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 -- 172.21.15.1:0/1753796162 wait complete. 2024-08-19T09:55:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:55:23.041+0000 7fcf1a2fd640 1 librados: shutdown 2024-08-19T09:55:23.082 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:55:23.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T09:55:41.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAe 2024-08-19T09:55:41.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7213 ms) 2024-08-19T09:55:41.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-08-19T09:55:41.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6815 ms) 2024-08-19T09:55:41.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6970 ms) 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6999 ms) 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7021 ms) 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7032 ms) 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6999 ms) 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (8585 ms) 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-08-19T09:55:41.184 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6405 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6964 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7357 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (17356 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7918 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (32352 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6228 ms) 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-08-19T09:55:41.185 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6990 ms) 2024-08-19T09:55:41.186 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (149206 ms total) 2024-08-19T09:55:41.186 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: 2024-08-19T09:55:41.186 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [----------] Global test environment tear-down 2024-08-19T09:55:41.186 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (261879 ms total) 2024-08-19T09:55:41.186 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio: [ PASSED ] 42 tests. 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: seed 25997 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (232 ms) 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (11728 ms) 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-08-19T09:55:45.179 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (52 ms) 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (12012 ms total) 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7710 ms) 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7320 ms) 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (11359 ms) 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: my_snaps [3] 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: my_snaps [4,3] 2024-08-19T09:55:45.180 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: my_snaps [5,4,3] 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: promoting some heads 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: promoting from clones for snap 6 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: promoting from clones for snap 5 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: promoting from clones for snap 4 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: promoting from clones for snap 3 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: waiting for scrubs... 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: done waiting 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (64864 ms) 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10311 ms) 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7636 ms) 2024-08-19T09:55:45.181 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7989 ms) 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8004 ms) 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10022 ms) 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9010 ms) 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11618 ms) 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12416 ms) 2024-08-19T09:55:45.182 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8518 ms) 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (18804 ms) 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (38602 ms) 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7206 ms) 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-08-19T09:55:45.183 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8006 ms) 2024-08-19T09:56:00.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-08-19T09:56:00.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: hmm, no HitSet yet 2024-08-19T09:56:00.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-08-19T09:56:00.491 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9008 ms) 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg_num = 32 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 0 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 1 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 2 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 3 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 4 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 5 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 6 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 7 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 8 ls 1724061361,0 2024-08-19T09:56:00.492 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 9 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 10 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 11 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 12 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 13 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 14 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 15 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 16 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 17 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 18 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 19 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 20 ls 1724061361,0 2024-08-19T09:56:00.493 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 21 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 22 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 23 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 24 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 25 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 26 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 27 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 28 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 29 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 30 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg 31 ls 1724061361,0 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: pg_num = 32 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6cac518f:::0:head 2024-08-19T09:56:00.494 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:02547ec2:::1:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f905c69b:::2:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:cfc208b3:::3:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d83876eb:::4:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b29083e3:::5:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c4fdafeb:::6:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:5c6b0b28:::7:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:bd63b0f1:::8:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e960b815:::9:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:52ea6a34:::10:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:89d3ae78:::11:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:de5d7c5f:::12:head 2024-08-19T09:56:00.495 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:566253c9:::13:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:62a1935d:::14:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:863748b0:::15:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3958e169:::16:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4d4dabf9:::17:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:8391935d:::18:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:28883081:::19:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:69259c59:::20:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4bdb80b7:::21:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a11c5d71:::22:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:271af37b:::23:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:95b121be:::24:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:58d1031b:::25:head 2024-08-19T09:56:00.496 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:0a050783:::26:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c709704c:::27:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:cbe56eaf:::28:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:86b4b162:::29:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:70d89383:::30:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:dd450c7c:::31:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6d5729b1:::32:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c388f3fb:::33:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:56cfea31:::34:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9dbc1bf7:::35:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:40b74ccd:::36:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4d5aaf42:::37:head 2024-08-19T09:56:00.497 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:920f362c:::38:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6cc53222:::39:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9cad833f:::40:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:1ea84d41:::41:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c4480ef6:::42:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a694361e:::43:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d1bd33e9:::44:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ddc2cd5d:::45:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:2b782207:::46:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7b187fca:::47:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:90ecdf6f:::48:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a5ed95fe:::49:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ea0eaa55:::50:head 2024-08-19T09:56:00.498 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f33ef17b:::51:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a0d1b2f6:::52:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:60c5229e:::53:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:edcbc575:::54:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:102cf253:::55:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:efb7fb0b:::56:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:50d0a326:::57:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d4dc5daf:::58:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3a130462:::59:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ec87ed71:::60:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d5bc9454:::61:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3ddfe313:::62:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7c2816b9:::63:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:47e00e4d:::64:head 2024-08-19T09:56:00.499 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c6410c18:::65:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b48ed237:::66:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:cd63ad31:::67:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b179e92b:::68:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:0d9f741a:::69:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6d3352ae:::70:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c6d5c19e:::71:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:bc4729c3:::72:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:77e930b9:::73:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:0abeecfd:::74:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b7c37e15:::75:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b6378398:::76:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:02bd68de:::77:head 2024-08-19T09:56:00.500 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:cc795d2d:::78:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:630d4fea:::79:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e0d29ef5:::80:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:fd6f13d2:::81:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:606461d5:::82:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:eadbdc43:::83:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:8761d0bb:::84:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9ef0186f:::85:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e0d41294:::86:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:961de695:::87:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:1423148f:::88:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:633a8fa2:::89:head 2024-08-19T09:56:00.501 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a8653809:::90:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3dac8b33:::91:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:35aad435:::92:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f6dcc343:::93:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:dbbdad87:::94:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:1cb48ce0:::95:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:03cd461c:::96:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:17a4ea99:::97:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9993c9a7:::98:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6394211c:::99:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:94c7ae57:::100:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6fdee5bb:::101:head 2024-08-19T09:56:00.502 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9a477fd1:::102:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:eb850916:::103:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:affc56b9:::104:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b42dc814:::105:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f319f8f0:::106:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9a40b9de:::107:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:8b524f28:::108:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e3de589f:::109:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:90f90a5b:::110:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a7b4f1d7:::111:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:af51766e:::112:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b6f90bd1:::113:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e0261208:::114:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c9569ef7:::115:head 2024-08-19T09:56:00.503 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:61bebe50:::116:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:fe93412b:::117:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d3d38bee:::118:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3100ba0c:::119:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d0560ada:::120:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f0ea8b35:::121:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:766f231a:::122:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a07a2582:::123:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:bd7c6b3a:::124:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:fb2ddaff:::125:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4408e1fe:::126:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ee1df7a7:::127:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c3002909:::128:head 2024-08-19T09:56:00.504 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4f48ffa9:::129:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:edf38733:::130:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c08425c0:::131:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:5f902d98:::132:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:41ea2c93:::133:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:813cee13:::134:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:0131818d:::135:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:26ba5a85:::136:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:381b8a5a:::137:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:28797e47:::138:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:bfca7f22:::139:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:36807075:::140:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:80b03975:::141:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:5c15709b:::142:head 2024-08-19T09:56:00.505 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f39ea15e:::143:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ea992956:::144:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:48887b1c:::145:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:9f24a9dd:::146:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:987f100b:::147:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d2dd3581:::148:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7fed1808:::149:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c80b70e9:::150:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:85ed90f9:::151:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:36428b24:::152:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d044c34a:::153:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7c18bf58:::154:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d1c21232:::155:head 2024-08-19T09:56:00.506 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a7a3c575:::156:head 2024-08-19T09:56:00.507 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:87da0633:::157:head 2024-08-19T09:56:00.507 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d5ac3822:::158:head 2024-08-19T09:56:00.507 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3f20522d:::159:head 2024-08-19T09:56:00.507 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6ca26563:::160:head 2024-08-19T09:56:00.507 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:532ce135:::161:head 2024-08-19T09:56:00.507 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:c78863e6:::162:head 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7462ddf6:io.RoundTripAppendPP (6035 ms) 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2966 ms) 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3012 ms) 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3009 ms) 2024-08-19T09:56:37.232 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: setting pool EIO 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: max_success 96, min_failed 97 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4001 ms) 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3015 ms) 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (152705 ms total) 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3022 ms) 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9963 ms) 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-08-19T09:56:37.233 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14995 ms) 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2991 ms) 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30971 ms total) 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3078 ms) 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3078 ms total) 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-08-19T09:56:37.234 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (25869 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (31788 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7256 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6207 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3020 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7018 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6971 ms) 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-08-19T09:56:37.235 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6999 ms) 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7891 ms) 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6127 ms) 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6983 ms) 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (8022 ms) 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-19T09:56:37.236 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7021 ms) 2024-08-19T09:56:53.083 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T09:56:53.084 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T09:56:53.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T09:56:53.089 INFO:tasks.ceph.mon.a:Started 2024-08-19T09:56:53.089 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T09:56:53.089 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T09:56:53.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T09:56:53.092 INFO:tasks.ceph.mon.d:Started 2024-08-19T09:56:53.092 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-19T09:56:53.092 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-19T09:56:53.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-19T09:56:53.095 INFO:tasks.ceph.mon.b:Started 2024-08-19T09:56:53.095 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-08-19T09:56:53.095 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T09:56:53.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T09:56:53.098 INFO:tasks.ceph.mon.f:Started 2024-08-19T09:56:53.098 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-08-19T09:56:53.098 INFO:tasks.ceph.mon.g:Sent signal 19 2024-08-19T09:56:53.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:57:08.099 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-08-19T09:57:08.100 INFO:tasks.ceph.mon.g:Sent signal 18 2024-08-19T09:57:08.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:57:08.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:57:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.214+0000 7f8bb4114640 10 monclient: get_monmap_and_config 2024-08-19T09:57:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.214+0000 7f8bb4114640 10 monclient: build_initial_monmap 2024-08-19T09:57:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.214+0000 7f8bb4114640 10 monclient: monmap: 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:08.216409+0000 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:08.216409+0000 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:08.218 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 1 Processor -- start 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 1 -- start start 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient: init 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient: _add_conns ranks=[1,7,4,3,0,2,8,6,5] 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient(hunting): picked mon.noname-f con 0x7f8bac0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:08.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient(hunting): picked mon.noname-i con 0x7f8bac0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient(hunting): picked mon.noname-c con 0x7f8bac0a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac0a3c70 con 0x7f8bac0a6830 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac0a33d0 con 0x7f8bac0a46a0 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac0a36b0 con 0x7f8bac0694b0 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb4114640 10 monclient(hunting): authenticate will time out at 1084.292821s 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb26b1640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f8bac0a46a0 0x7f8bac141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37358/0 (socket says 172.21.15.1:37358) 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.215+0000 7f8bb26b1640 1 -- 172.21.15.1:0/4198671409 learned_addr learned my addr 172.21.15.1:0/4198671409 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.218+0000 7f8bb26b1640 1 -- 172.21.15.1:0/4198671409 >> v1:172.21.15.1:6791/0 conn(0x7f8bac0a46a0 legacy=0x7f8bac141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.218+0000 7f8bb16af640 1 -- 172.21.15.1:0/4198671409 >> v1:172.21.15.82:6792/0 conn(0x7f8bac0694b0 legacy=0x7f8bac0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:08.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.218+0000 7f8bb1eb0640 1 -- 172.21.15.1:0/4198671409 >> v1:172.21.15.82:6789/0 conn(0x7f8bac0a6830 legacy=0x7f8bac0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:08.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519370019 0 0) 0x7f8bac0a3c70 con 0x7f8bac0a6830 2024-08-19T09:57:08.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): _negotiate 2024-08-19T09:57:08.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:08.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): my global_id is 164098 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b98003520 con 0x7f8bac0a6830 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3828297874 0 0) 0x7f8bac0a36b0 con 0x7f8bac0694b0 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): _negotiate 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 10 monclient(hunting): my global_id is 144403 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bac0a3c70 con 0x7f8bac0694b0 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 856619426 0 0) 0x7f8b98003520 con 0x7f8bac0a6830 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.661+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 >> v1:172.21.15.82:6792/0 conn(0x7f8bac0694b0 legacy=0x7f8bac0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 >> v1:172.21.15.1:6791/0 conn(0x7f8bac0a46a0 legacy=0x7f8bac141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: _finish_hunting 0 2024-08-19T09:57:08.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 1 monclient: found mon.noname-f 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac06dbb0 con 0x7f8bac0a6830 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: _finish_auth 0 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: _check_auth_tickets 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8ba8001b30 con 0x7f8bac0a6830 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: dump: 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:08.666 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:08.667 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 5 monclient: authenticate success, global_id 164098 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ba8003610 con 0x7f8bac0a6830 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 1 -- 172.21.15.1:0/4198671409 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8ba8003b80 con 0x7f8bac0a6830 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 4 set_mon_vals no callback set 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb0eae640 10 monclient: dump: 2024-08-19T09:57:08.668 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:08.669 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 10 monclient: shutdown 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 1 -- 172.21.15.1:0/4198671409 >> v1:172.21.15.82:6789/0 conn(0x7f8bac0a6830 legacy=0x7f8bac0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 1 -- 172.21.15.1:0/4198671409 shutdown_connections 2024-08-19T09:57:08.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 1 -- 172.21.15.1:0/4198671409 >> 172.21.15.1:0/4198671409 conn(0x7f8bac0a9020 msgr2=0x7f8bac068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.662+0000 7f8bb4114640 1 -- 172.21.15.1:0/4198671409 shutdown_connections 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.663+0000 7f8bb4114640 1 -- 172.21.15.1:0/4198671409 wait complete. 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.665+0000 7f8bb4114640 10 monclient: build_initial_monmap 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.665+0000 7f8bb4114640 10 monclient: monmap: 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:08.672 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.665+0000 7f8bb4114640 1 librados: starting msgr at 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.665+0000 7f8bb4114640 1 librados: starting objecter 2024-08-19T09:57:08.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.665+0000 7f8bb4114640 1 Processor -- start 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.667+0000 7f8bb4114640 1 -- start start 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.667+0000 7f8bb4114640 1 librados: setting wanted keys 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.667+0000 7f8bb4114640 1 librados: calling monclient init 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.667+0000 7f8bb4114640 10 monclient: init 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient: _add_conns ranks=[8,6,5,2,3,4,7,0,1] 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient(hunting): picked mon.noname-e con 0x7f8bac0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient(hunting): picked mon.noname-d con 0x7f8bac0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient(hunting): picked mon.noname-h con 0x7f8bac0a6830 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac142d30 con 0x7f8bac0a6830 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac137d60 con 0x7f8bac0a46a0 2024-08-19T09:57:08.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac136870 con 0x7f8bac0694b0 2024-08-19T09:57:08.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:08.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb4114640 10 monclient(hunting): authenticate will time out at 1084.744594s 2024-08-19T09:57:08.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb16af640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f8bac0a46a0 0x7f8bac135690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36794/0 (socket says 172.21.15.1:36794) 2024-08-19T09:57:08.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.668+0000 7f8bb16af640 1 -- 172.21.15.1:0/1072199681 learned_addr learned my addr 172.21.15.1:0/1072199681 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:08.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.673+0000 7f8bb26b1640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.82:6791/0 conn(0x7f8bac0a6830 legacy=0x7f8bac135da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.673+0000 7f8bb16af640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.1:6792/0 conn(0x7f8bac0a46a0 legacy=0x7f8bac135690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.673+0000 7f8bb1eb0640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.1:6793/0 conn(0x7f8bac0694b0 legacy=0x7f8bac134f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 client.?.objecter ms_handle_connect 0x7f8bac0a6830 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 client.?.objecter ms_handle_connect 0x7f8bac0a46a0 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 client.?.objecter ms_handle_connect 0x7f8bac0694b0 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444307651 0 0) 0x7f8bac137d60 con 0x7f8bac0a46a0 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:08.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): my global_id is 164103 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b90004eb0 con 0x7f8bac0a46a0 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021123528 0 0) 0x7f8bac142d30 con 0x7f8bac0a6830 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 10 monclient(hunting): my global_id is 26879 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.674+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bac137d60 con 0x7f8bac0a6830 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.675+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2248699748 0 0) 0x7f8bac136870 con 0x7f8bac0694b0 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.675+0000 7f8ba27fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.675+0000 7f8ba27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:08.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.675+0000 7f8ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.675+0000 7f8ba27fc640 10 monclient(hunting): my global_id is 27701 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.675+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bac142d30 con 0x7f8bac0694b0 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3437211589 0 0) 0x7f8b90004eb0 con 0x7f8bac0a46a0 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bac136870 con 0x7f8bac0a46a0 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8b9c003610 con 0x7f8bac0a46a0 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient(hunting): dump: 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:08.687 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3283230333 0 0) 0x7f8bac136870 con 0x7f8bac0a46a0 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.1:6793/0 conn(0x7f8bac0694b0 legacy=0x7f8bac134f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.82:6791/0 conn(0x7f8bac0a6830 legacy=0x7f8bac135da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: _finish_hunting 0 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 monclient: found mon.d 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac138bd0 con 0x7f8bac0a46a0 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: _finish_auth 0 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: _check_auth_tickets 2024-08-19T09:57:08.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b9c003b50 con 0x7f8bac0a46a0 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8b9c004080 con 0x7f8bac0a46a0 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.676+0000 7f8ba27fc640 10 monclient: dump: 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:08.690 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:08.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 5 monclient: authenticate success, global_id 164103 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 10 monclient: _renew_subs 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac1424b0 con 0x7f8bac0a46a0 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 10 client.164103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 10 monclient: _renew_subs 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bac1d7410 con 0x7f8bac0a46a0 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.680+0000 7f8bb4114640 1 librados: init done 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.681+0000 7f8ba37fe640 4 set_mon_vals no callback set 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.682+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8b9c004220 con 0x7f8bac0a46a0 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.682+0000 7f8bb4114640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.682+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(294..294 src has 1..294) ==== 5875+0+0 (unknown 2685745609 0 0) 0x7f8b9c04e470 con 0x7f8bac0a46a0 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.682+0000 7f8ba27fc640 10 client.164103.objecter ms_dispatch 0x7f8bac000b90 osd_map(294..294 src has 1..294) 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.682+0000 7f8ba27fc640 3 client.164103.objecter handle_osd_map got epochs [294,294] > 0 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.682+0000 7f8ba27fc640 3 client.164103.objecter handle_osd_map decoding full epoch 294 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.683+0000 7f8bb4114640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.683+0000 7f8bb4114640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.683+0000 7f8ba27fc640 20 client.164103.objecter dump_active .. 0 homeless 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.683+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b94005080 con 0x7f8bac0a46a0 2024-08-19T09:57:08.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.683+0000 7f8bb1eb0640 10 monclient: get_auth_request con 0x7f8b90041210 auth_method 0 2024-08-19T09:57:08.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.685+0000 7f8ba27fc640 10 client.164103.objecter ms_handle_connect 0x7f8b90041210 2024-08-19T09:57:08.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.688+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f8b9c017680 con 0x7f8bac0a46a0 2024-08-19T09:57:08.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.688+0000 7f8ba27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:08.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.688+0000 7f8ba27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8bb4114640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:57:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8bb4114640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:57:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8bb4114640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8b94005370 con 0x7f8bac0a46a0 2024-08-19T09:57:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8ba27fc640 1 -- 172.21.15.1:0/1072199681 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 3080437849) 0x7f8b9c016f80 con 0x7f8bac0a46a0 2024-08-19T09:57:08.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8ba27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:57:08.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.945+0000 7f8ba27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:08.947 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":66,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:57:08.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 10 librados: watch_flush enter 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 10 librados: watch_flush exit 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 20 client.164103.objecter shutdown clearing up homeless session... 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 10 client.164103.objecter successfully canceled tick 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8b90041210 legacy=0x7f8b900436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 10 monclient: shutdown 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 >> v1:172.21.15.1:6792/0 conn(0x7f8bac0a46a0 legacy=0x7f8bac135690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 shutdown_connections 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 >> 172.21.15.1:0/1072199681 conn(0x7f8bac0a9020 msgr2=0x7f8bac0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 shutdown_connections 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 -- 172.21.15.1:0/1072199681 wait complete. 2024-08-19T09:57:08.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:08.947+0000 7f8bb4114640 1 librados: shutdown 2024-08-19T09:57:08.959 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:57:08.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:57:08.959 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:57:09.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.058+0000 7ff4ca3f7640 10 monclient: get_monmap_and_config 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.058+0000 7ff4ca3f7640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.058+0000 7ff4ca3f7640 10 monclient: monmap: 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:09.058956+0000 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:09.058956+0000 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.059 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.058+0000 7ff4ca3f7640 1 Processor -- start 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.058+0000 7ff4ca3f7640 1 -- start start 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.058+0000 7ff4ca3f7640 10 monclient: init 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient: _add_conns ranks=[4,6,1,7,2,8,3,5,0] 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient(hunting): picked mon.noname-d con 0x7ff4c40a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient(hunting): picked mon.noname-f con 0x7ff4c413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c40a3b40 con 0x7ff4c413f5c0 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c40a8130 con 0x7ff4c40a67c0 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c406ede0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4ca3f7640 10 monclient(hunting): authenticate will time out at 1085.135346s 2024-08-19T09:57:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4c3fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff4c40a67c0 0x7ff4c40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37366/0 (socket says 172.21.15.1:37366) 2024-08-19T09:57:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4c37fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ff4c40a6ba0 0x7ff4c40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36804/0 (socket says 172.21.15.1:36804) 2024-08-19T09:57:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.059+0000 7ff4c3fff640 1 -- 172.21.15.1:0/1824437143 learned_addr learned my addr 172.21.15.1:0/1824437143 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c37fe640 1 -- 172.21.15.1:0/1824437143 >> v1:172.21.15.1:6792/0 conn(0x7ff4c40a6ba0 legacy=0x7ff4c40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c3fff640 1 -- 172.21.15.1:0/1824437143 >> v1:172.21.15.1:6791/0 conn(0x7ff4c40a67c0 legacy=0x7ff4c40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c8994640 1 -- 172.21.15.1:0/1824437143 >> v1:172.21.15.82:6789/0 conn(0x7ff4c413f5c0 legacy=0x7ff4c41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828334078 0 0) 0x7ff4c406ede0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): my global_id is 164112 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4ac003520 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519118406 0 0) 0x7ff4c40a8130 con 0x7ff4c40a67c0 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 10 monclient(hunting): my global_id is 27157 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.061+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4c406ede0 con 0x7ff4c40a67c0 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4269712888 0 0) 0x7ff4ac003520 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 >> v1:172.21.15.1:6791/0 conn(0x7ff4c40a67c0 legacy=0x7ff4c40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 >> v1:172.21.15.82:6789/0 conn(0x7ff4c413f5c0 legacy=0x7ff4c41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 monclient: found mon.noname-d 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c40a92a0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 5 monclient: authenticate success, global_id 164112 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4b8001ad0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: dump: 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4b80047e0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 1 -- 172.21.15.1:0/1824437143 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4b8004d10 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 4 set_mon_vals no callback set 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4c2ffd640 10 monclient: dump: 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 10 monclient: shutdown 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1824437143 >> v1:172.21.15.1:6792/0 conn(0x7ff4c40a6ba0 legacy=0x7ff4c40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1824437143 shutdown_connections 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1824437143 >> 172.21.15.1:0/1824437143 conn(0x7ff4c40ac4b0 msgr2=0x7ff4c40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1824437143 shutdown_connections 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.062+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1824437143 wait complete. 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 10 monclient: monmap: 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 1 librados: starting msgr at 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 1 librados: starting objecter 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 1 Processor -- start 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 1 -- start start 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 1 librados: setting wanted keys 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 1 librados: calling monclient init 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.063+0000 7ff4ca3f7640 10 monclient: init 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient: _add_conns ranks=[5,3,0,6,8,2,1,4,7] 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient(hunting): picked mon.noname-h con 0x7ff4c40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient(hunting): picked mon.noname-g con 0x7ff4c40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c413f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c40a1680 con 0x7ff4c413f5c0 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c413a6d0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c413a9b0 con 0x7ff4c40a67c0 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4ca3f7640 10 monclient(hunting): authenticate will time out at 1085.140358s 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4c8994640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4c413f5c0 0x7ff4c41d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38150/0 (socket says 172.21.15.1:38150) 2024-08-19T09:57:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.064+0000 7ff4c8994640 1 -- 172.21.15.1:0/1954122116 learned_addr learned my addr 172.21.15.1:0/1954122116 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c8994640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4c413f5c0 legacy=0x7ff4c41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff4c413f5c0 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff4c40a67c0 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c3fff640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.82:6791/0 conn(0x7ff4c40a67c0 legacy=0x7ff4c409fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff4c40a6ba0 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c37fe640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.82:6790/0 conn(0x7ff4c40a6ba0 legacy=0x7ff4c40a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135744935 0 0) 0x7ff4c40a1680 con 0x7ff4c413f5c0 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 164115 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4b0004eb0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3548665933 0 0) 0x7ff4c413a9b0 con 0x7ff4c40a67c0 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 26888 2024-08-19T09:57:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4c40a1680 con 0x7ff4c40a67c0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855381424 0 0) 0x7ff4c413a6d0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 134877 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.066+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4c413a9b0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1768382651 0 0) 0x7ff4b0004eb0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4c413a6d0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4bc0020a0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.085 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2690079993 0 0) 0x7ff4c40a1680 con 0x7ff4c40a67c0 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4b0004eb0 con 0x7ff4c40a67c0 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4b40047e0 con 0x7ff4c40a67c0 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:09.086 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3689849353 0 0) 0x7ff4c413a9b0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4c40a1680 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4b8004cd0 con 0x7ff4c40a6ba0 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2054623463 0 0) 0x7ff4c413a6d0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.82:6791/0 conn(0x7ff4c40a67c0 legacy=0x7ff4c409fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.82:6790/0 conn(0x7ff4c40a6ba0 legacy=0x7ff4c40a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 monclient: found mon.a 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c40aa000 con 0x7ff4c413f5c0 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 5 monclient: authenticate success, global_id 164115 2024-08-19T09:57:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 10 monclient: _renew_subs 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c40a21e0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 10 client.164115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 10 monclient: _renew_subs 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4c40a0e80 con 0x7ff4c413f5c0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4ca3f7640 1 librados: init done 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4bc003d90 con 0x7ff4c413f5c0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4bc0042c0 con 0x7ff4c413f5c0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c27fc640 4 set_mon_vals no callback set 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.067+0000 7ff4c0ff9640 10 monclient: dump: 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4ca3f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff4a40065c0 con 0x7ff4a4002860 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff4bc004460 con 0x7ff4c413f5c0 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(294..294 src has 1..294) ==== 5875+0+0 (unknown 2685745609 0 0) 0x7ff4bc04e690 con 0x7ff4c413f5c0 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c0ff9640 10 client.164115.objecter ms_dispatch 0x7ff4c4000b90 osd_map(294..294 src has 1..294) 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c0ff9640 3 client.164115.objecter handle_osd_map got epochs [294,294] > 0 2024-08-19T09:57:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c0ff9640 3 client.164115.objecter handle_osd_map decoding full epoch 294 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c0ff9640 20 client.164115.objecter dump_active .. 0 homeless 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.068+0000 7ff4c37fe640 10 monclient: get_auth_request con 0x7ff4b0041910 auth_method 0 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 client.164115.objecter ms_handle_connect 0x7ff4a4002860 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 client.164115.objecter resend_mon_ops 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c3fff640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4a4002860 legacy=0x7ff4a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726521193 0 0) 0x7ff4a40065c0 con 0x7ff4a4002860 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 164124 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4c413a6d0 con 0x7ff4a4002860 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1349694949 0 0) 0x7ff4c413a6d0 con 0x7ff4a4002860 2024-08-19T09:57:09.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4a40065c0 con 0x7ff4a4002860 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4b4005160 con 0x7ff4a4002860 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.070+0000 7ff4c0ff9640 10 monclient: dump: 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2573379624 0 0) 0x7ff4a40065c0 con 0x7ff4a4002860 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4a400a160 con 0x7ff4a4002860 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 10 client.164115.objecter ms_handle_connect 0x7ff4b0041910 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff4b40073a0 con 0x7ff4a4002860 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4a4002860 legacy=0x7ff4a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.071+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4a4002860 legacy=0x7ff4a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.077+0000 7ff4ca3f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.077+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff4a4009e50 con 0x7ff4a4009110 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c3fff640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4a4009110 legacy=0x7ff4a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 10 client.164115.objecter ms_handle_connect 0x7ff4a4009110 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 10 client.164115.objecter resend_mon_ops 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1020690471 0 0) 0x7ff4a4009e50 con 0x7ff4a4009110 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 164133 2024-08-19T09:57:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.079+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4a40065c0 con 0x7ff4a4009110 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 127855408 0 0) 0x7ff4a40065c0 con 0x7ff4a4009110 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4a4009e50 con 0x7ff4a4009110 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff4b4006c10 con 0x7ff4a4009110 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 10 monclient: dump: 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 581684916 0 0) 0x7ff4a4009e50 con 0x7ff4a4009110 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4a400c3a0 con 0x7ff4a4009110 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3039990420) 0x7ff4b4006ea0 con 0x7ff4a4009110 2024-08-19T09:57:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4a4009110 legacy=0x7ff4a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c0ff9640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4a4009110 legacy=0x7ff4a40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:09.100 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15898, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.101 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.102 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.103 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:09.104 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:09.105 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:09.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.108 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.112 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.113 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.114 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.117 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:09.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.121 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:09.121 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 10 librados: watch_flush enter 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 10 librados: watch_flush exit 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 20 client.164115.objecter shutdown clearing up homeless session... 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 10 client.164115.objecter successfully canceled tick 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff4b0041910 legacy=0x7ff4b0043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 10 monclient: shutdown 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 >> v1:172.21.15.1:6789/0 conn(0x7ff4c413f5c0 legacy=0x7ff4c41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c8994640 1 -- 172.21.15.1:0/1954122116 reap_dead start 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4c8994640 1 -- 172.21.15.1:0/1954122116 reap_dead start 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 shutdown_connections 2024-08-19T09:57:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.080+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 >> 172.21.15.1:0/1954122116 conn(0x7ff4c40ac4b0 msgr2=0x7ff4c40ab5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.081+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 shutdown_connections 2024-08-19T09:57:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.081+0000 7ff4ca3f7640 1 -- 172.21.15.1:0/1954122116 wait complete. 2024-08-19T09:57:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.081+0000 7ff4ca3f7640 1 librados: shutdown 2024-08-19T09:57:09.124 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:09.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:57:09.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:57:09.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.223+0000 7f02b81ba640 10 monclient: get_monmap_and_config 2024-08-19T09:57:09.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.223+0000 7f02b81ba640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.223+0000 7f02b81ba640 10 monclient: monmap: 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:09.224739+0000 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:09.224739+0000 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.226 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.226 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.226 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.226 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.224+0000 7f02b81ba640 1 Processor -- start 2024-08-19T09:57:09.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.224+0000 7f02b81ba640 1 -- start start 2024-08-19T09:57:09.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.224+0000 7f02b81ba640 10 monclient: init 2024-08-19T09:57:09.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient: _add_conns ranks=[7,4,6,3,1,2,0,5,8] 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient(hunting): picked mon.noname-i con 0x7f02b00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient(hunting): picked mon.noname-c con 0x7f02b00a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient(hunting): picked mon.noname-d con 0x7f02b013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b00a3b40 con 0x7f02b00a6ba0 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b00a8130 con 0x7f02b013f5c0 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b006ede0 con 0x7f02b00a67c0 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b81ba640 10 monclient(hunting): authenticate will time out at 1085.301169s 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b5755640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f02b00a6ba0 0x7f02b00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37382/0 (socket says 172.21.15.1:37382) 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b5755640 1 -- 172.21.15.1:0/977173311 learned_addr learned my addr 172.21.15.1:0/977173311 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.225+0000 7f02b6757640 1 --1- 172.21.15.1:0/977173311 >> v1:172.21.15.1:6792/0 conn(0x7f02b013f5c0 0x7f02b01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36816/0 (socket says 172.21.15.1:36816) 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b5755640 1 -- 172.21.15.1:0/977173311 >> v1:172.21.15.1:6791/0 conn(0x7f02b00a6ba0 legacy=0x7f02b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b6757640 1 -- 172.21.15.1:0/977173311 >> v1:172.21.15.1:6792/0 conn(0x7f02b013f5c0 legacy=0x7f02b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912430530 0 0) 0x7f02b00a8130 con 0x7f02b013f5c0 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): my global_id is 164121 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f029c003520 con 0x7f02b013f5c0 2024-08-19T09:57:09.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1253876322 0 0) 0x7f02b00a3b40 con 0x7f02b00a6ba0 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient(hunting): my global_id is 27166 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b00a8130 con 0x7f02b00a6ba0 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b5f56640 1 -- 172.21.15.1:0/977173311 >> v1:172.21.15.82:6792/0 conn(0x7f02b00a67c0 legacy=0x7f02b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 443727823 0 0) 0x7f029c003520 con 0x7f02b013f5c0 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 >> v1:172.21.15.82:6792/0 conn(0x7f02b00a67c0 legacy=0x7f02b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 >> v1:172.21.15.1:6791/0 conn(0x7f02b00a6ba0 legacy=0x7f02b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 monclient: found mon.noname-d 2024-08-19T09:57:09.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b00a92a0 con 0x7f02b013f5c0 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02a8001d90 con 0x7f02b013f5c0 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.227+0000 7f02b4f54640 10 monclient: dump: 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.234 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.235 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 5 monclient: authenticate success, global_id 164121 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02a80020a0 con 0x7f02b013f5c0 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b4f54640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b4f54640 1 -- 172.21.15.1:0/977173311 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02a8004ea0 con 0x7f02b013f5c0 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b4f54640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 4 set_mon_vals no callback set 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b4f54640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b4f54640 10 monclient: dump: 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.236 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 10 monclient: shutdown 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 1 -- 172.21.15.1:0/977173311 >> v1:172.21.15.1:6792/0 conn(0x7f02b013f5c0 legacy=0x7f02b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 1 -- 172.21.15.1:0/977173311 shutdown_connections 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 1 -- 172.21.15.1:0/977173311 >> 172.21.15.1:0/977173311 conn(0x7f02b00ac4b0 msgr2=0x7f02b00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 1 -- 172.21.15.1:0/977173311 shutdown_connections 2024-08-19T09:57:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.228+0000 7f02b81ba640 1 -- 172.21.15.1:0/977173311 wait complete. 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 10 monclient: monmap: 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 1 librados: starting msgr at 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 1 librados: starting objecter 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 1 Processor -- start 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 1 -- start start 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 1 librados: setting wanted keys 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 1 librados: calling monclient init 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 10 monclient: init 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.229+0000 7f02b81ba640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 10 monclient: _add_conns ranks=[1,4,8,5,3,2,0,6,7] 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 10 monclient(hunting): picked mon.noname-f con 0x7f02b00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 10 monclient(hunting): picked mon.noname-c con 0x7f02b00a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 10 monclient(hunting): picked mon.noname-e con 0x7f02b013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b013a880 con 0x7f02b00a67c0 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b0139ce0 con 0x7f02b00a6ba0 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b0142bf0 con 0x7f02b013f5c0 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b81ba640 10 monclient(hunting): authenticate will time out at 1085.306097s 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b5755640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f02b00a6ba0 0x7f02b00a0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37392/0 (socket says 172.21.15.1:37392) 2024-08-19T09:57:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.230+0000 7f02b5755640 1 -- 172.21.15.1:0/3694779551 learned_addr learned my addr 172.21.15.1:0/3694779551 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02b5755640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6791/0 conn(0x7f02b00a6ba0 legacy=0x7f02b00a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 client.?.objecter ms_handle_connect 0x7f02b00a6ba0 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 client.?.objecter ms_handle_connect 0x7f02b013f5c0 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02b6757640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6793/0 conn(0x7f02b013f5c0 legacy=0x7f02b01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02b5f56640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.82:6789/0 conn(0x7f02b00a67c0 legacy=0x7f02b009fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 client.?.objecter ms_handle_connect 0x7f02b00a67c0 2024-08-19T09:57:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2535421666 0 0) 0x7f02b0139ce0 con 0x7f02b00a6ba0 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): my global_id is 27175 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0294004eb0 con 0x7f02b00a6ba0 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656203603 0 0) 0x7f02b0142bf0 con 0x7f02b013f5c0 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): my global_id is 27710 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b0139ce0 con 0x7f02b013f5c0 2024-08-19T09:57:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1825827839 0 0) 0x7f02b013a880 con 0x7f02b00a67c0 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): my global_id is 164116 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b0142bf0 con 0x7f02b00a67c0 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 234660256 0 0) 0x7f0294004eb0 con 0x7f02b00a6ba0 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02b013a880 con 0x7f02b00a6ba0 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02a00047e0 con 0x7f02b00a6ba0 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): dump: 2024-08-19T09:57:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.251 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1072705700 0 0) 0x7f02b0139ce0 con 0x7f02b013f5c0 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0294004eb0 con 0x7f02b013f5c0 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02a8004ea0 con 0x7f02b013f5c0 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): dump: 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.252 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.253 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1808983086 0 0) 0x7f02b0142bf0 con 0x7f02b00a67c0 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02b0139ce0 con 0x7f02b00a67c0 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02ac001d00 con 0x7f02b00a67c0 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient(hunting): dump: 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.254 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 170777488 0 0) 0x7f02b013a880 con 0x7f02b00a6ba0 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6793/0 conn(0x7f02b013f5c0 legacy=0x7f02b01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.82:6789/0 conn(0x7f02b00a67c0 legacy=0x7f02b009fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 1 monclient: found mon.c 2024-08-19T09:57:09.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.232+0000 7f02a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b013a5a0 con 0x7f02b00a6ba0 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 5 monclient: authenticate success, global_id 27175 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 10 monclient: _renew_subs 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02b00a0a90 con 0x7f02b00a6ba0 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 10 client.27175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 10 monclient: _renew_subs 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f02b01d7b50 con 0x7f02b00a6ba0 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 1 librados: init done 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02a0004c30 con 0x7f02b00a6ba0 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02a0005160 con 0x7f02b00a6ba0 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a7fff640 4 set_mon_vals no callback set 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 10 monclient: dump: 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.258 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f02a0005300 con 0x7f02b00a6ba0 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.233+0000 7f02b81ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f02980065c0 con 0x7f0298002860 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(294..294 src has 1..294) ==== 5875+0+0 (unknown 2685745609 0 0) 0x7f02a004e450 con 0x7f02b00a6ba0 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02a67fc640 10 client.27175.objecter ms_dispatch 0x7f02b0000b90 osd_map(294..294 src has 1..294) 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02a67fc640 3 client.27175.objecter handle_osd_map got epochs [294,294] > 0 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02a67fc640 3 client.27175.objecter handle_osd_map decoding full epoch 294 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02a67fc640 20 client.27175.objecter dump_active .. 0 homeless 2024-08-19T09:57:09.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.234+0000 7f02b6757640 10 monclient: get_auth_request con 0x7f0294041800 auth_method 0 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 client.27175.objecter ms_handle_connect 0x7f0298002860 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 client.27175.objecter resend_mon_ops 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02b5f56640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6790/0 conn(0x7f0298002860 legacy=0x7f0298004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148590302 0 0) 0x7f02980065c0 con 0x7f0298002860 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient(hunting): my global_id is 164108 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b013a880 con 0x7f0298002860 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3843388923 0 0) 0x7f02b013a880 con 0x7f0298002860 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02980065c0 con 0x7f0298002860 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02ac003ce0 con 0x7f0298002860 2024-08-19T09:57:09.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 monclient: dump: 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.261 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 10 client.27175.objecter ms_handle_connect 0x7f0294041800 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 898053159 0 0) 0x7f02980065c0 con 0x7f0298002860 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.236+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f029800a160 con 0x7f0298002860 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.237+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f02ac003fe0 con 0x7f0298002860 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.237+0000 7f02a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.237+0000 7f02a67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.237+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6790/0 conn(0x7f0298002860 legacy=0x7f0298004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.237+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6790/0 conn(0x7f0298002860 legacy=0x7f0298004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.243+0000 7f02b81ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.243+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f029800c4f0 con 0x7f0298009110 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02b5f56640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6790/0 conn(0x7f0298009110 legacy=0x7f0298009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 client.27175.objecter ms_handle_connect 0x7f0298009110 2024-08-19T09:57:09.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 client.27175.objecter resend_mon_ops 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048532847 0 0) 0x7f029800c4f0 con 0x7f0298009110 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient(hunting): my global_id is 164117 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02980065c0 con 0x7f0298009110 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1713779756 0 0) 0x7f02980065c0 con 0x7f0298009110 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f029800c4f0 con 0x7f0298009110 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f02ac004490 con 0x7f0298009110 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 10 monclient: dump: 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.265 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2430445554 0 0) 0x7f029800c4f0 con 0x7f0298009110 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.245+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0298009ce0 con 0x7f0298009110 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 1829055893) 0x7f02ac005ea0 con 0x7f0298009110 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02a67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6790/0 conn(0x7f0298009110 legacy=0x7f0298009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02a67fc640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6790/0 conn(0x7f0298009110 legacy=0x7f0298009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16061, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.268 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.269 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.270 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:09.271 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.272 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.278 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.279 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.280 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:09.281 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.282 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.283 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.284 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.286 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.287 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.289 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.289 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:09.289 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 10 librados: watch_flush enter 2024-08-19T09:57:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 10 librados: watch_flush exit 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 20 client.27175.objecter shutdown clearing up homeless session... 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 10 client.27175.objecter successfully canceled tick 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0294041800 legacy=0x7f0294043cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 10 monclient: shutdown 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 >> v1:172.21.15.1:6791/0 conn(0x7f02b00a6ba0 legacy=0x7f02b00a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b6757640 1 -- 172.21.15.1:0/3694779551 reap_dead start 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b6757640 1 -- 172.21.15.1:0/3694779551 reap_dead start 2024-08-19T09:57:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 shutdown_connections 2024-08-19T09:57:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 >> 172.21.15.1:0/3694779551 conn(0x7f02b00ac4b0 msgr2=0x7f02b00a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 shutdown_connections 2024-08-19T09:57:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 -- 172.21.15.1:0/3694779551 wait complete. 2024-08-19T09:57:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.246+0000 7f02b81ba640 1 librados: shutdown 2024-08-19T09:57:09.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:09.293 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:57:09.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:57:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.391+0000 7fac203f7640 10 monclient: get_monmap_and_config 2024-08-19T09:57:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.391+0000 7fac203f7640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.391+0000 7fac203f7640 10 monclient: monmap: 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:09.392386+0000 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:09.392386+0000 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.394 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 1 Processor -- start 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 1 -- start start 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient: init 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient: _add_conns ranks=[8,6,2,1,3,5,4,7,0] 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient(hunting): picked mon.noname-e con 0x7fac180a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient(hunting): picked mon.noname-d con 0x7fac180a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient(hunting): picked mon.noname-b con 0x7fac1813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac180a3b40 con 0x7fac1813f5c0 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac180a8130 con 0x7fac180a6ba0 2024-08-19T09:57:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1806ede0 con 0x7fac180a67c0 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac203f7640 10 monclient(hunting): authenticate will time out at 1085.468751s 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac1e193640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fac180a67c0 0x7fac180a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52878/0 (socket says 172.21.15.1:52878) 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac1e193640 1 -- 172.21.15.1:0/3819996149 learned_addr learned my addr 172.21.15.1:0/3819996149 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.392+0000 7fac1d992640 1 --1- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6792/0 conn(0x7fac180a6ba0 0x7fac180a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36828/0 (socket says 172.21.15.1:36828) 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1e994640 1 -- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6790/0 conn(0x7fac1813f5c0 legacy=0x7fac181419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d992640 1 -- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6792/0 conn(0x7fac180a6ba0 legacy=0x7fac180a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1e193640 1 -- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6793/0 conn(0x7fac180a67c0 legacy=0x7fac180a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13894565 0 0) 0x7fac180a3b40 con 0x7fac1813f5c0 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): my global_id is 164126 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac00003520 con 0x7fac1813f5c0 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034775664 0 0) 0x7fac180a8130 con 0x7fac180a6ba0 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): my global_id is 164130 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac180a3b40 con 0x7fac180a6ba0 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 742097483 0 0) 0x7fac1806ede0 con 0x7fac180a67c0 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient(hunting): my global_id is 27719 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac180a8130 con 0x7fac180a67c0 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2200092391 0 0) 0x7fac00003520 con 0x7fac1813f5c0 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d992640 1 --1- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6792/0 conn(0x7fac180a6ba0 0x7fac180a6fb0 :-1 s=OPENED pgs=3330 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fac0c001970 mon_map magic: 0 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6793/0 conn(0x7fac180a67c0 legacy=0x7fac180a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6792/0 conn(0x7fac180a6ba0 legacy=0x7fac180a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 monclient: found mon.noname-b 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac180a92a0 con 0x7fac1813f5c0 2024-08-19T09:57:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac08001eb0 con 0x7fac1813f5c0 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac203f7640 5 monclient: authenticate success, global_id 164126 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: dump: 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac203f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac0c001970 con 0x7fac180a6ba0 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac08003990 con 0x7fac1813f5c0 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 1 -- 172.21.15.1:0/3819996149 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac08003f00 con 0x7fac1813f5c0 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac203f7640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac203f7640 4 set_mon_vals no callback set 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.395+0000 7fac1d191640 10 monclient: dump: 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.407 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 10 monclient: shutdown 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 1 -- 172.21.15.1:0/3819996149 >> v1:172.21.15.1:6790/0 conn(0x7fac1813f5c0 legacy=0x7fac181419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 1 -- 172.21.15.1:0/3819996149 shutdown_connections 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 1 -- 172.21.15.1:0/3819996149 >> 172.21.15.1:0/3819996149 conn(0x7fac180ac4b0 msgr2=0x7fac180a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 1 -- 172.21.15.1:0/3819996149 shutdown_connections 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 1 -- 172.21.15.1:0/3819996149 wait complete. 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.396+0000 7fac203f7640 10 monclient: monmap: 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 librados: starting msgr at 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 librados: starting objecter 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 Processor -- start 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 -- start start 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 librados: setting wanted keys 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 librados: calling monclient init 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient: init 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient: _add_conns ranks=[8,2,4,7,3,5,6,1,0] 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient(hunting): picked mon.noname-e con 0x7fac180a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient(hunting): picked mon.noname-b con 0x7fac180a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient(hunting): picked mon.noname-c con 0x7fac1813f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac181429d0 con 0x7fac180a6ba0 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1813c580 con 0x7fac1813f5c0 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1813c860 con 0x7fac180a67c0 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac203f7640 10 monclient(hunting): authenticate will time out at 1085.473856s 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac1e193640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fac180a67c0 0x7fac18139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52888/0 (socket says 172.21.15.1:52888) 2024-08-19T09:57:09.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.397+0000 7fac1e193640 1 -- 172.21.15.1:0/1941515437 learned_addr learned my addr 172.21.15.1:0/1941515437 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac1d992640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6790/0 conn(0x7fac180a6ba0 legacy=0x7fac1813a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac1e994640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fac1813f5c0 legacy=0x7fac181d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 client.?.objecter ms_handle_connect 0x7fac180a6ba0 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 client.?.objecter ms_handle_connect 0x7fac180a67c0 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 client.?.objecter ms_handle_connect 0x7fac1813f5c0 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac1e193640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6793/0 conn(0x7fac180a67c0 legacy=0x7fac18139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744487055 0 0) 0x7fac0c001ca0 con 0x7fac180a6ba0 2024-08-19T09:57:09.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): my global_id is 164135 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac04004eb0 con 0x7fac180a6ba0 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 433051634 0 0) 0x7fac1813c580 con 0x7fac1813f5c0 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): my global_id is 27184 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac0c001ca0 con 0x7fac1813f5c0 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216420044 0 0) 0x7fac1813c860 con 0x7fac180a67c0 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): my global_id is 27728 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac1813c580 con 0x7fac180a67c0 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1311096803 0 0) 0x7fac04004eb0 con 0x7fac180a6ba0 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac1813c860 con 0x7fac180a6ba0 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac0c0018d0 con 0x7fac180a6ba0 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4070433608 0 0) 0x7fac0c001ca0 con 0x7fac1813f5c0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac04004eb0 con 0x7fac1813f5c0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 568767584 0 0) 0x7fac1813c580 con 0x7fac180a67c0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac0c001ca0 con 0x7fac180a67c0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac08003e80 con 0x7fac1813f5c0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.422 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac100047e0 con 0x7fac180a67c0 2024-08-19T09:57:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.424 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2138157582 0 0) 0x7fac1813c860 con 0x7fac180a6ba0 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6793/0 conn(0x7fac180a67c0 legacy=0x7fac18139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fac1813f5c0 legacy=0x7fac181d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 monclient: found mon.b 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1813b060 con 0x7fac180a6ba0 2024-08-19T09:57:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 5 monclient: authenticate success, global_id 164135 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 10 monclient: _renew_subs 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac180a2880 con 0x7fac180a6ba0 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 10 client.164135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 10 monclient: _renew_subs 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fac180a2040 con 0x7fac180a6ba0 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac203f7640 1 librados: init done 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac0c004dd0 con 0x7fac180a6ba0 2024-08-19T09:57:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac0c005300 con 0x7fac180a6ba0 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac1c990640 4 set_mon_vals no callback set 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.400+0000 7fac16ffd640 10 monclient: dump: 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.401+0000 7fac203f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.401+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fabf80065c0 con 0x7fabf8002860 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.401+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fac0c0117e0 con 0x7fac180a6ba0 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.402+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(294..294 src has 1..294) ==== 5875+0+0 (unknown 2685745609 0 0) 0x7fac0c04e610 con 0x7fac180a6ba0 2024-08-19T09:57:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.402+0000 7fac16ffd640 10 client.164135.objecter ms_dispatch 0x7fac18000b90 osd_map(294..294 src has 1..294) 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.402+0000 7fac16ffd640 3 client.164135.objecter handle_osd_map got epochs [294,294] > 0 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.402+0000 7fac16ffd640 3 client.164135.objecter handle_osd_map decoding full epoch 294 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.402+0000 7fac16ffd640 20 client.164135.objecter dump_active .. 0 homeless 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.402+0000 7fac1e994640 10 monclient: get_auth_request con 0x7fac04041930 auth_method 0 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.403+0000 7fac16ffd640 10 client.164135.objecter ms_handle_connect 0x7fabf8002860 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.403+0000 7fac1e193640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fabf8002860 legacy=0x7fabf8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.403+0000 7fac16ffd640 10 client.164135.objecter resend_mon_ops 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101634408 0 0) 0x7fabf80065c0 con 0x7fabf8002860 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient(hunting): my global_id is 27193 2024-08-19T09:57:09.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac1813c860 con 0x7fabf8002860 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 737479190 0 0) 0x7fac1813c860 con 0x7fabf8002860 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabf80065c0 con 0x7fabf8002860 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac10004eb0 con 0x7fabf8002860 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient: dump: 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3061047845 0 0) 0x7fabf80065c0 con 0x7fabf8002860 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabf800a160 con 0x7fabf8002860 2024-08-19T09:57:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 client.164135.objecter ms_handle_connect 0x7fac04041930 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fac10006370 con 0x7fabf8002860 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fabf8002860 legacy=0x7fabf8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.404+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fabf8002860 legacy=0x7fabf8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.410+0000 7fac203f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.410+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fabf8009e70 con 0x7fabf8009130 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 client.164135.objecter ms_handle_connect 0x7fabf8009130 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac1e193640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fabf8009130 legacy=0x7fabf80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 client.164135.objecter resend_mon_ops 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790663108 0 0) 0x7fabf8009e70 con 0x7fabf8009130 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient(hunting): my global_id is 27202 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabf80065c0 con 0x7fabf8009130 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3050615224 0 0) 0x7fabf80065c0 con 0x7fabf8009130 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabf8009e70 con 0x7fabf8009130 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fac100054a0 con 0x7fabf8009130 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient: dump: 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.433 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1584101506 0 0) 0x7fabf8009e70 con 0x7fabf8009130 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fabf800c3f0 con 0x7fabf8009130 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2712809303) 0x7fac10005890 con 0x7fabf8009130 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fabf8009130 legacy=0x7fabf80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.413+0000 7fac16ffd640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6791/0 conn(0x7fabf8009130 legacy=0x7fabf80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 435685, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:09.439 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:09.456 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 10 librados: watch_flush enter 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 10 librados: watch_flush exit 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 20 client.164135.objecter shutdown clearing up homeless session... 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 10 client.164135.objecter successfully canceled tick 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.82:6812/2549104415 conn(0x7fac04041930 legacy=0x7fac04043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 10 monclient: shutdown 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 >> v1:172.21.15.1:6790/0 conn(0x7fac180a6ba0 legacy=0x7fac1813a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac1e994640 1 -- 172.21.15.1:0/1941515437 reap_dead start 2024-08-19T09:57:09.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac1e994640 1 -- 172.21.15.1:0/1941515437 reap_dead start 2024-08-19T09:57:09.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 shutdown_connections 2024-08-19T09:57:09.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 >> 172.21.15.1:0/1941515437 conn(0x7fac180ac4b0 msgr2=0x7fac18141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 shutdown_connections 2024-08-19T09:57:09.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 -- 172.21.15.1:0/1941515437 wait complete. 2024-08-19T09:57:09.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.414+0000 7fac203f7640 1 librados: shutdown 2024-08-19T09:57:09.459 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:09.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:57:09.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:57:09.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.558+0000 7f7ff6832640 10 monclient: get_monmap_and_config 2024-08-19T09:57:09.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.558+0000 7f7ff6832640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.558+0000 7f7ff6832640 10 monclient: monmap: 2024-08-19T09:57:09.559 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.559 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.559 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:09.559397+0000 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:09.559397+0000 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.560 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.561 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 1 Processor -- start 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 1 -- start start 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient: init 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient: _add_conns ranks=[8,2,6,5,3,4,0,7,1] 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient(hunting): picked mon.noname-e con 0x7f7ff0139580 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:09.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff00a4500 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient(hunting): picked mon.noname-d con 0x7f7ff006f250 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff00a77d0 con 0x7f7ff00a4500 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff00a3330 con 0x7f7ff006f250 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff00a3bb0 con 0x7f7ff0139580 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff6832640 10 monclient(hunting): authenticate will time out at 1085.635749s 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff4dcf640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f7ff006f250 0x7f7ff0141ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36836/0 (socket says 172.21.15.1:36836) 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7ff4dcf640 1 -- 172.21.15.1:0/184551212 learned_addr learned my addr 172.21.15.1:0/184551212 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7fef7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7ff00a4500 0x7f7ff013e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:47986/0 (socket says 172.21.15.1:47986) 2024-08-19T09:57:09.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.559+0000 7f7feffff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f7ff0139580 0x7f7ff0139960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52902/0 (socket says 172.21.15.1:52902) 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7ff4dcf640 1 -- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6792/0 conn(0x7f7ff006f250 legacy=0x7f7ff0141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feffff640 1 -- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6793/0 conn(0x7f7ff0139580 legacy=0x7f7ff0139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7fef7fe640 1 -- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6790/0 conn(0x7f7ff00a4500 legacy=0x7f7ff013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901531516 0 0) 0x7f7ff00a3330 con 0x7f7ff006f250 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): my global_id is 164139 2024-08-19T09:57:09.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fd8003520 con 0x7f7ff006f250 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100294320 0 0) 0x7f7ff00a3bb0 con 0x7f7ff0139580 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): my global_id is 27737 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ff00a3330 con 0x7f7ff0139580 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349840523 0 0) 0x7f7ff00a77d0 con 0x7f7ff00a4500 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient(hunting): my global_id is 164144 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ff00a3bb0 con 0x7f7ff00a4500 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7ff4dcf640 1 --1- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6792/0 conn(0x7f7ff006f250 0x7f7ff0141ef0 :-1 s=OPENED pgs=3331 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f7fe8001e60 mon_map magic: 0 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2116660191 0 0) 0x7f7ff00a3330 con 0x7f7ff0139580 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6792/0 conn(0x7f7ff006f250 legacy=0x7f7ff0141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6790/0 conn(0x7f7ff00a4500 legacy=0x7f7ff013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 monclient: found mon.noname-e 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:57:09.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff00a95b0 con 0x7f7ff0139580 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe8001e60 con 0x7f7ff006f250 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe0001ac0 con 0x7f7ff0139580 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: dump: 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.576 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7ff6832640 5 monclient: authenticate success, global_id 27737 2024-08-19T09:57:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7ff6832640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fe00047e0 con 0x7f7ff0139580 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 1 -- 172.21.15.1:0/184551212 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe0004cd0 con 0x7f7ff0139580 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7feeffd640 10 monclient: dump: 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7ff6832640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.562+0000 7f7ff6832640 4 set_mon_vals no callback set 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 10 monclient: shutdown 2024-08-19T09:57:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 -- 172.21.15.1:0/184551212 >> v1:172.21.15.1:6793/0 conn(0x7f7ff0139580 legacy=0x7f7ff0139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 -- 172.21.15.1:0/184551212 shutdown_connections 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 -- 172.21.15.1:0/184551212 >> 172.21.15.1:0/184551212 conn(0x7f7ff00ac5d0 msgr2=0x7f7ff00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 -- 172.21.15.1:0/184551212 shutdown_connections 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 -- 172.21.15.1:0/184551212 wait complete. 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 10 monclient: monmap: 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 librados: starting msgr at 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.563+0000 7f7ff6832640 1 librados: starting objecter 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 Processor -- start 2024-08-19T09:57:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 -- start start 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 librados: setting wanted keys 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 librados: calling monclient init 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient: init 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient: _add_conns ranks=[2,6,7,0,5,8,1,3,4] 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff006f250 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient(hunting): picked mon.noname-d con 0x7f7ff0139580 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient(hunting): picked mon.noname-i con 0x7f7ff01ce790 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff01386a0 con 0x7f7ff006f250 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff0138980 con 0x7f7ff0139580 2024-08-19T09:57:09.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff00a6fc0 con 0x7f7ff01ce790 2024-08-19T09:57:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7ff6832640 10 monclient(hunting): authenticate will time out at 1085.640804s 2024-08-19T09:57:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7fef7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f7ff0139580 0x7f7ff01d2f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36852/0 (socket says 172.21.15.1:36852) 2024-08-19T09:57:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7feffff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7ff006f250 0x7f7ff01ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:47998/0 (socket says 172.21.15.1:47998) 2024-08-19T09:57:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.564+0000 7f7fef7fe640 1 -- 172.21.15.1:0/1011705217 learned_addr learned my addr 172.21.15.1:0/1011705217 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.566+0000 7f7fef7fe640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7ff0139580 legacy=0x7f7ff01d2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.566+0000 7f7fecff9640 10 client.?.objecter ms_handle_connect 0x7f7ff0139580 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.566+0000 7f7feffff640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6790/0 conn(0x7f7ff006f250 legacy=0x7f7ff01ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 client.?.objecter ms_handle_connect 0x7f7ff006f250 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826832163 0 0) 0x7f7ff0138980 con 0x7f7ff0139580 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): my global_id is 164148 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fdc004ee0 con 0x7f7ff0139580 2024-08-19T09:57:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3928570258 0 0) 0x7f7ff01386a0 con 0x7f7ff006f250 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): my global_id is 164153 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ff0138980 con 0x7f7ff006f250 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff4dcf640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.82:6792/0 conn(0x7f7ff01ce790 legacy=0x7f7ff01d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 client.?.objecter ms_handle_connect 0x7f7ff01ce790 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1679539040 0 0) 0x7f7fdc004ee0 con 0x7f7ff0139580 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ff01386a0 con 0x7f7ff0139580 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe40047e0 con 0x7f7ff0139580 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): dump: 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1826281033 0 0) 0x7f7ff0138980 con 0x7f7ff006f250 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fdc004ee0 con 0x7f7ff006f250 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe0001b20 con 0x7f7ff006f250 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): dump: 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2305183660 0 0) 0x7f7ff00a6fc0 con 0x7f7ff01ce790 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient(hunting): my global_id is 144421 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ff0138980 con 0x7f7ff01ce790 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3166422341 0 0) 0x7f7ff01386a0 con 0x7f7ff0139580 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.82:6792/0 conn(0x7f7ff01ce790 legacy=0x7f7ff01d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6790/0 conn(0x7f7ff006f250 legacy=0x7f7ff01ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 monclient: found mon.d 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff0136620 con 0x7f7ff0139580 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 5 monclient: authenticate success, global_id 164148 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 10 monclient: _renew_subs 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff0137550 con 0x7f7ff0139580 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 10 client.164148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:09.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 10 monclient: _renew_subs 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ff0137180 con 0x7f7ff0139580 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7ff6832640 1 librados: init done 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fe4004c30 con 0x7f7ff0139580 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe4005160 con 0x7f7ff0139580 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fee7fc640 4 set_mon_vals no callback set 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.567+0000 7f7fecff9640 10 monclient: dump: 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7ff6832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7fd00065c0 con 0x7f7fd0002860 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7fe4005400 con 0x7f7ff0139580 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(294..294 src has 1..294) ==== 5875+0+0 (unknown 2685745609 0 0) 0x7f7fe404e390 con 0x7f7ff0139580 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7fecff9640 10 client.164148.objecter ms_dispatch 0x7f7ff0000b90 osd_map(294..294 src has 1..294) 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7fecff9640 3 client.164148.objecter handle_osd_map got epochs [294,294] > 0 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.568+0000 7f7fecff9640 3 client.164148.objecter handle_osd_map decoding full epoch 294 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.569+0000 7f7fecff9640 20 client.164148.objecter dump_active .. 0 homeless 2024-08-19T09:57:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.569+0000 7f7ff4dcf640 10 monclient: get_auth_request con 0x7f7fdc041640 auth_method 0 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 10 client.164148.objecter ms_handle_connect 0x7f7fd0002860 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7feffff640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7fd0002860 legacy=0x7f7fd0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 10 client.164148.objecter resend_mon_ops 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822208357 0 0) 0x7f7fd00065c0 con 0x7f7fd0002860 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 10 monclient(hunting): my global_id is 164157 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.570+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ff01386a0 con 0x7f7fd0002860 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 818038243 0 0) 0x7f7ff01386a0 con 0x7f7fd0002860 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fd00065c0 con 0x7f7fd0002860 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe00047e0 con 0x7f7fd0002860 2024-08-19T09:57:09.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 10 monclient: dump: 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3891365172 0 0) 0x7f7fd00065c0 con 0x7f7fd0002860 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fd000a160 con 0x7f7fd0002860 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 10 client.164148.objecter ms_handle_connect 0x7f7fdc041640 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7fe0005200 con 0x7f7fd0002860 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7fd0002860 legacy=0x7f7fd0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.571+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7fd0002860 legacy=0x7f7fd0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.577+0000 7f7ff6832640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.577+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7fd0009e70 con 0x7f7fd0009130 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 client.164148.objecter ms_handle_connect 0x7f7fd0009130 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7feffff640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7fd0009130 legacy=0x7f7fd00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 client.164148.objecter resend_mon_ops 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506886302 0 0) 0x7f7fd0009e70 con 0x7f7fd0009130 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient(hunting): my global_id is 164166 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fd00065c0 con 0x7f7fd0009130 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1117090340 0 0) 0x7f7fd00065c0 con 0x7f7fd0009130 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fd0009e70 con 0x7f7fd0009130 2024-08-19T09:57:09.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7fe0005a30 con 0x7f7fd0009130 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient: dump: 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.600 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1747734565 0 0) 0x7f7fd0009e70 con 0x7f7fd0009130 2024-08-19T09:57:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7fd000c3f0 con 0x7f7fd0009130 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3088667052) 0x7f7fe0007f20 con 0x7f7fd0009130 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7fd0009130 legacy=0x7f7fd00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.580+0000 7f7fecff9640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7fd0009130 legacy=0x7f7fd00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:09.602 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16398, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.604 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.606 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.607 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.608 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.609 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:09.615 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.616 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:09.622 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:09.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 10 librados: watch_flush enter 2024-08-19T09:57:09.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 10 librados: watch_flush exit 2024-08-19T09:57:09.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 20 client.164148.objecter shutdown clearing up homeless session... 2024-08-19T09:57:09.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 10 client.164148.objecter successfully canceled tick 2024-08-19T09:57:09.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7fdc041640 legacy=0x7f7fdc043b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 10 monclient: shutdown 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 >> v1:172.21.15.1:6792/0 conn(0x7f7ff0139580 legacy=0x7f7ff01d2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff4dcf640 1 -- 172.21.15.1:0/1011705217 reap_dead start 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff4dcf640 1 -- 172.21.15.1:0/1011705217 reap_dead start 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 shutdown_connections 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 >> 172.21.15.1:0/1011705217 conn(0x7f7ff00ac5d0 msgr2=0x7f7ff00aa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 shutdown_connections 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 -- 172.21.15.1:0/1011705217 wait complete. 2024-08-19T09:57:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.581+0000 7f7ff6832640 1 librados: shutdown 2024-08-19T09:57:09.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:09.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:57:09.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:57:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.735+0000 7eff18420640 10 monclient: get_monmap_and_config 2024-08-19T09:57:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.735+0000 7eff18420640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.735+0000 7eff18420640 10 monclient: monmap: 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:09.736427+0000 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:09.736427+0000 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.738 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.738 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.738 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.738 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 1 Processor -- start 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 1 -- start start 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient: init 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient: _add_conns ranks=[7,3,4,8,6,2,5,1,0] 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient(hunting): picked mon.noname-i con 0x7eff100a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient(hunting): picked mon.noname-g con 0x7eff100a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient(hunting): picked mon.noname-c con 0x7eff1013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff100a3b40 con 0x7eff100a6ba0 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff100a8130 con 0x7eff1013f5c0 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1006ede0 con 0x7eff100a67c0 2024-08-19T09:57:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff18420640 10 monclient(hunting): authenticate will time out at 1085.812805s 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff169bd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7eff1013f5c0 0x7eff101419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37432/0 (socket says 172.21.15.1:37432) 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.736+0000 7eff169bd640 1 -- 172.21.15.1:0/2623826729 learned_addr learned my addr 172.21.15.1:0/2623826729 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff169bd640 1 -- 172.21.15.1:0/2623826729 >> v1:172.21.15.1:6791/0 conn(0x7eff1013f5c0 legacy=0x7eff101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399305820 0 0) 0x7eff100a8130 con 0x7eff1013f5c0 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): my global_id is 27211 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efefc003520 con 0x7eff1013f5c0 2024-08-19T09:57:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff161bc640 1 -- 172.21.15.1:0/2623826729 >> v1:172.21.15.82:6792/0 conn(0x7eff100a67c0 legacy=0x7eff100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff159bb640 1 -- 172.21.15.1:0/2623826729 >> v1:172.21.15.82:6790/0 conn(0x7eff100a6ba0 legacy=0x7eff100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864805726 0 0) 0x7eff1006ede0 con 0x7eff100a67c0 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): my global_id is 144430 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff100a8130 con 0x7eff100a67c0 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217773467 0 0) 0x7eff100a3b40 con 0x7eff100a6ba0 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient(hunting): my global_id is 134886 2024-08-19T09:57:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff1006ede0 con 0x7eff100a6ba0 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1929677109 0 0) 0x7efefc003520 con 0x7eff1013f5c0 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 >> v1:172.21.15.82:6792/0 conn(0x7eff100a67c0 legacy=0x7eff100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 >> v1:172.21.15.82:6790/0 conn(0x7eff100a6ba0 legacy=0x7eff100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 monclient: found mon.noname-c 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff100a92a0 con 0x7eff1013f5c0 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff08001e60 con 0x7eff1013f5c0 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff18420640 5 monclient: authenticate success, global_id 27211 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff18420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: dump: 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff18420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff08004c20 con 0x7eff1013f5c0 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff18420640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 1 -- 172.21.15.1:0/2623826729 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff08005190 con 0x7eff1013f5c0 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff18420640 4 set_mon_vals no callback set 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.739+0000 7eff151ba640 10 monclient: dump: 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.757 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 10 monclient: shutdown 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 -- 172.21.15.1:0/2623826729 >> v1:172.21.15.1:6791/0 conn(0x7eff1013f5c0 legacy=0x7eff101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 -- 172.21.15.1:0/2623826729 shutdown_connections 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 -- 172.21.15.1:0/2623826729 >> 172.21.15.1:0/2623826729 conn(0x7eff100ac4b0 msgr2=0x7eff100a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 -- 172.21.15.1:0/2623826729 shutdown_connections 2024-08-19T09:57:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 -- 172.21.15.1:0/2623826729 wait complete. 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 10 monclient: monmap: 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 librados: starting msgr at 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.740+0000 7eff18420640 1 librados: starting objecter 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 Processor -- start 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 -- start start 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 librados: setting wanted keys 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 librados: calling monclient init 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient: init 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient: _add_conns ranks=[5,8,6,7,3,4,1,2,0] 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient(hunting): picked mon.noname-h con 0x7eff100a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient(hunting): picked mon.noname-e con 0x7eff100a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient(hunting): picked mon.noname-d con 0x7eff1013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1013bb00 con 0x7eff100a67c0 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff101cf020 con 0x7eff1013f5c0 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff10139ce0 con 0x7eff100a6ba0 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff18420640 10 monclient(hunting): authenticate will time out at 1085.817590s 2024-08-19T09:57:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff169bd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7eff1013f5c0 0x7eff101d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36872/0 (socket says 172.21.15.1:36872) 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.741+0000 7eff169bd640 1 -- 172.21.15.1:0/797812995 learned_addr learned my addr 172.21.15.1:0/797812995 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff159bb640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7eff100a6ba0 legacy=0x7eff101ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 client.?.objecter ms_handle_connect 0x7eff100a6ba0 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff169bd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6792/0 conn(0x7eff1013f5c0 legacy=0x7eff101d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 client.?.objecter ms_handle_connect 0x7eff1013f5c0 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490416255 0 0) 0x7eff10139ce0 con 0x7eff100a6ba0 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 10 monclient(hunting): my global_id is 27746 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff161bc640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.82:6791/0 conn(0x7eff100a67c0 legacy=0x7eff101ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.743+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efef4004eb0 con 0x7eff100a6ba0 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 client.?.objecter ms_handle_connect 0x7eff100a67c0 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776256926 0 0) 0x7eff101cf020 con 0x7eff1013f5c0 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): my global_id is 164175 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff10139ce0 con 0x7eff1013f5c0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709984837 0 0) 0x7eff1013bb00 con 0x7eff100a67c0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): my global_id is 26897 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff101cf020 con 0x7eff100a67c0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2274445444 0 0) 0x7efef4004eb0 con 0x7eff100a6ba0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff1013bb00 con 0x7eff100a6ba0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff00003610 con 0x7eff100a6ba0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 490323535 0 0) 0x7eff10139ce0 con 0x7eff1013f5c0 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efef4004eb0 con 0x7eff1013f5c0 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff080020b0 con 0x7eff1013f5c0 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.768 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1009628037 0 0) 0x7eff101cf020 con 0x7eff100a67c0 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff10139ce0 con 0x7eff100a67c0 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff0c003610 con 0x7eff100a67c0 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.770 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.771 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 573661940 0 0) 0x7eff1013bb00 con 0x7eff100a6ba0 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6792/0 conn(0x7eff1013f5c0 legacy=0x7eff101d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.82:6791/0 conn(0x7eff100a67c0 legacy=0x7eff101ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 monclient: found mon.e 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1013e690 con 0x7eff100a6ba0 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 5 monclient: authenticate success, global_id 27746 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 10 monclient: _renew_subs 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff1013d6b0 con 0x7eff100a6ba0 2024-08-19T09:57:09.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 10 client.27746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 10 monclient: _renew_subs 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7eff1013aa00 con 0x7eff100a6ba0 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff18420640 1 librados: init done 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff00003a60 con 0x7eff100a6ba0 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff00003f50 con 0x7eff100a6ba0 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff07fff640 4 set_mon_vals no callback set 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.744+0000 7eff06ffd640 10 monclient: dump: 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.775 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7eff00004220 con 0x7eff100a6ba0 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff18420640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efef80065c0 con 0x7efef8002860 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(295..295 src has 1..295) ==== 6280+0+0 (unknown 2052330141 0 0) 0x7eff0004e3c0 con 0x7eff100a6ba0 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff06ffd640 10 client.27746.objecter ms_dispatch 0x7eff10000b90 osd_map(295..295 src has 1..295) 2024-08-19T09:57:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff06ffd640 3 client.27746.objecter handle_osd_map got epochs [295,295] > 0 2024-08-19T09:57:09.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff06ffd640 3 client.27746.objecter handle_osd_map decoding full epoch 295 2024-08-19T09:57:09.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff06ffd640 20 client.27746.objecter dump_active .. 0 homeless 2024-08-19T09:57:09.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.745+0000 7eff169bd640 10 monclient: get_auth_request con 0x7efef4041ab0 auth_method 0 2024-08-19T09:57:09.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff161bc640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7efef8002860 legacy=0x7efef8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 10 client.27746.objecter ms_handle_connect 0x7efef8002860 2024-08-19T09:57:09.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 10 client.27746.objecter resend_mon_ops 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704474858 0 0) 0x7efef80065c0 con 0x7efef8002860 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 10 monclient(hunting): my global_id is 27755 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.747+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff1013bb00 con 0x7efef8002860 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 983013695 0 0) 0x7eff1013bb00 con 0x7efef8002860 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efef80065c0 con 0x7efef8002860 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff0c003e10 con 0x7efef8002860 2024-08-19T09:57:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 10 monclient: dump: 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3742433442 0 0) 0x7efef80065c0 con 0x7efef8002860 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efef800a160 con 0x7efef8002860 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 10 client.27746.objecter ms_handle_connect 0x7efef4041ab0 2024-08-19T09:57:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7eff0c004040 con 0x7efef8002860 2024-08-19T09:57:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7efef8002860 legacy=0x7efef8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.748+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7efef8002860 legacy=0x7efef8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.754+0000 7eff18420640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.754+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efef8009e50 con 0x7efef8009110 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.756+0000 7eff161bc640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7efef8009110 legacy=0x7efef80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.756+0000 7eff06ffd640 10 client.27746.objecter ms_handle_connect 0x7efef8009110 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.756+0000 7eff06ffd640 10 client.27746.objecter resend_mon_ops 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714258380 0 0) 0x7efef8009e50 con 0x7efef8009110 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient(hunting): my global_id is 27764 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efef80065c0 con 0x7efef8009110 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 148945704 0 0) 0x7efef80065c0 con 0x7efef8009110 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efef8009e50 con 0x7efef8009110 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7eff0c004470 con 0x7efef8009110 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient: dump: 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.786 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2033519493 0 0) 0x7efef8009e50 con 0x7efef8009110 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efef800c3a0 con 0x7efef8009110 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1090569443) 0x7eff0c005cf0 con 0x7efef8009110 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7efef8009110 legacy=0x7efef80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff06ffd640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7efef8009110 legacy=0x7efef80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.787 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 436030, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:09.792 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:09.793 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.794 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.795 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.796 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.797 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.798 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:09.799 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.800 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.805 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.808 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:09.810 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.811 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:09.812 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:09.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff18420640 10 librados: watch_flush enter 2024-08-19T09:57:09.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff18420640 10 librados: watch_flush exit 2024-08-19T09:57:09.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff18420640 20 client.27746.objecter shutdown clearing up homeless session... 2024-08-19T09:57:09.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.757+0000 7eff18420640 10 client.27746.objecter successfully canceled tick 2024-08-19T09:57:09.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.82:6812/2549104415 conn(0x7efef4041ab0 legacy=0x7efef4043f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 10 monclient: shutdown 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 >> v1:172.21.15.1:6793/0 conn(0x7eff100a6ba0 legacy=0x7eff101ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff169bd640 1 -- 172.21.15.1:0/797812995 reap_dead start 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff169bd640 1 -- 172.21.15.1:0/797812995 reap_dead start 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 shutdown_connections 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 >> 172.21.15.1:0/797812995 conn(0x7eff100ac4b0 msgr2=0x7eff10141c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 shutdown_connections 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 -- 172.21.15.1:0/797812995 wait complete. 2024-08-19T09:57:09.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.758+0000 7eff18420640 1 librados: shutdown 2024-08-19T09:57:09.815 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:09.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:57:09.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.915+0000 7f81f290d640 10 monclient: get_monmap_and_config 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.915+0000 7f81f290d640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.915+0000 7f81f290d640 10 monclient: monmap: 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:09.916079+0000 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:09.916079+0000 2024-08-19T09:57:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.917 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 1 Processor -- start 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 1 -- start start 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient: init 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient: _add_conns ranks=[6,5,7,8,4,1,2,0,3] 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient(hunting): picked mon.noname-d con 0x7f81ec0a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient(hunting): picked mon.noname-h con 0x7f81ec0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient(hunting): picked mon.noname-i con 0x7f81ec0a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec0a3c70 con 0x7f81ec0694b0 2024-08-19T09:57:09.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec0a33d0 con 0x7f81ec0a6830 2024-08-19T09:57:09.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec0a36b0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81f290d640 10 monclient(hunting): authenticate will time out at 1085.992598s 2024-08-19T09:57:09.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81ebfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f81ec0a6830 0x7f81ec0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36886/0 (socket says 172.21.15.1:36886) 2024-08-19T09:57:09.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.916+0000 7f81ebfff640 1 -- 172.21.15.1:0/3211049667 learned_addr learned my addr 172.21.15.1:0/3211049667 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.918+0000 7f81ebfff640 1 -- 172.21.15.1:0/3211049667 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0a6830 legacy=0x7f81ec0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 427194157 0 0) 0x7f81ec0a33d0 con 0x7f81ec0a6830 2024-08-19T09:57:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient(hunting): my global_id is 164184 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d8003520 con 0x7f81ec0a6830 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f0eaa640 1 -- 172.21.15.1:0/3211049667 >> v1:172.21.15.82:6792/0 conn(0x7f81ec0a46a0 legacy=0x7f81ec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81e3fff640 1 -- 172.21.15.1:0/3211049667 >> v1:172.21.15.82:6791/0 conn(0x7f81ec0694b0 legacy=0x7f81ec0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1962473478 0 0) 0x7f81d8003520 con 0x7f81ec0a6830 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 >> v1:172.21.15.82:6792/0 conn(0x7f81ec0a46a0 legacy=0x7f81ec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 >> v1:172.21.15.82:6791/0 conn(0x7f81ec0694b0 legacy=0x7f81ec0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 monclient: found mon.noname-d 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec06dbb0 con 0x7f81ec0a6830 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81dc001ae0 con 0x7f81ec0a6830 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 5 monclient: authenticate success, global_id 164184 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: dump: 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.932 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.932 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.932 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81dc0048a0 con 0x7f81ec0a6830 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 1 -- 172.21.15.1:0/3211049667 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81dc004e50 con 0x7f81ec0a6830 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 4 set_mon_vals no callback set 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81eb7fe640 10 monclient: dump: 2024-08-19T09:57:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 10 monclient: shutdown 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 1 -- 172.21.15.1:0/3211049667 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0a6830 legacy=0x7f81ec0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 1 -- 172.21.15.1:0/3211049667 shutdown_connections 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.919+0000 7f81f290d640 1 -- 172.21.15.1:0/3211049667 >> 172.21.15.1:0/3211049667 conn(0x7f81ec0a9020 msgr2=0x7f81ec068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 1 -- 172.21.15.1:0/3211049667 shutdown_connections 2024-08-19T09:57:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 1 -- 172.21.15.1:0/3211049667 wait complete. 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 10 monclient: build_initial_monmap 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 10 monclient: monmap: 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 1 librados: starting msgr at 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 1 librados: starting objecter 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.920+0000 7f81f290d640 1 Processor -- start 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 1 -- start start 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 1 librados: setting wanted keys 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 1 librados: calling monclient init 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient: init 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient: _add_conns ranks=[6,5,0,2,8,4,3,1,7] 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient(hunting): picked mon.noname-d con 0x7f81ec0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient(hunting): picked mon.noname-h con 0x7f81ec0a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient(hunting): picked mon.noname-a con 0x7f81ec0a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec0abb70 con 0x7f81ec0a6830 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec13bbf0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec138ed0 con 0x7f81ec0694b0 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81f290d640 10 monclient(hunting): authenticate will time out at 1085.997534s 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81ebfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 0x7f81ec1380b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36896/0 (socket says 172.21.15.1:36896) 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.921+0000 7f81ebfff640 1 -- 172.21.15.1:0/323250598 learned_addr learned my addr 172.21.15.1:0/323250598 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81ebfff640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 legacy=0x7f81ec1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e97fa640 10 client.?.objecter ms_handle_connect 0x7f81ec0694b0 2024-08-19T09:57:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e97fa640 10 client.?.objecter ms_handle_connect 0x7f81ec0a6830 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81f0eaa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6789/0 conn(0x7f81ec0a6830 legacy=0x7f81ec1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e97fa640 10 client.?.objecter ms_handle_connect 0x7f81ec0a46a0 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.923+0000 7f81e3fff640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6791/0 conn(0x7f81ec0a46a0 legacy=0x7f81ec1387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638385288 0 0) 0x7f81ec13bbf0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): my global_id is 26915 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81cc004eb0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572046662 0 0) 0x7f81ec138ed0 con 0x7f81ec0694b0 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): my global_id is 164193 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81ec13bbf0 con 0x7f81ec0694b0 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3353298759 0 0) 0x7f81e40019b0 con 0x7f81ec0a6830 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): my global_id is 164142 2024-08-19T09:57:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81ec138ed0 con 0x7f81ec0a6830 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81ebfff640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 legacy=0x7f81ec1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81ebfff640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 legacy=0x7f81ec1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81ebfff640 1 --1- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 0x7f81ec1380b0 :-1 s=OPENED pgs=3337 cs=1 l=1).handle_message read tag failed 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81ebfff640 1 --1- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 0x7f81ec1380b0 :-1 s=OPENED pgs=3337 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6792/0 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1704573974 0 0) 0x7f81cc004eb0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81e40019b0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81d4003610 con 0x7f81ec0a46a0 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): dump: 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.942 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 760357688 0 0) 0x7f81ec138ed0 con 0x7f81ec0a6830 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81cc004eb0 con 0x7f81ec0a6830 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81e4003cd0 con 0x7f81ec0a6830 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient(hunting): dump: 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.944 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2393158360 0 0) 0x7f81e40019b0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6792/0 conn(0x7f81ec0694b0 legacy=0x7f81ec1380b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.1:6789/0 conn(0x7f81ec0a6830 legacy=0x7f81ec1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: _finish_hunting 0 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 monclient: found mon.h 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec13bed0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: _finish_auth 0 2024-08-19T09:57:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: _check_auth_tickets 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 5 monclient: authenticate success, global_id 26915 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 10 monclient: _renew_subs 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f81ec142a00 con 0x7f81ec0a46a0 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 10 client.26915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 10 monclient: _renew_subs 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f81ec13b060 con 0x7f81ec0a46a0 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81f290d640 1 librados: init done 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81d4003b90 con 0x7f81ec0a46a0 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81d4004100 con 0x7f81ec0a46a0 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81eaffd640 4 set_mon_vals no callback set 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.924+0000 7f81e97fa640 10 monclient: dump: 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.948 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.949 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81f290d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f81d00065c0 con 0x7f81d0002860 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f81d40042a0 con 0x7f81ec0a46a0 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(295..295 src has 1..295) ==== 6280+0+0 (unknown 2052330141 0 0) 0x7f81d404e530 con 0x7f81ec0a46a0 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81e97fa640 10 client.26915.objecter ms_dispatch 0x7f81ec000b90 osd_map(295..295 src has 1..295) 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81e97fa640 3 client.26915.objecter handle_osd_map got epochs [295,295] > 0 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81e97fa640 3 client.26915.objecter handle_osd_map decoding full epoch 295 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81e97fa640 20 client.26915.objecter dump_active .. 0 homeless 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.925+0000 7f81f0eaa640 10 monclient: get_auth_request con 0x7f81cc0416d0 auth_method 0 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81ebfff640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6789/0 conn(0x7f81d0002860 legacy=0x7f81d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 10 client.26915.objecter ms_handle_connect 0x7f81d0002860 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 10 client.26915.objecter resend_mon_ops 2024-08-19T09:57:09.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846838400 0 0) 0x7f81d00065c0 con 0x7f81d0002860 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 10 monclient(hunting): my global_id is 164125 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.927+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81e40019b0 con 0x7f81d0002860 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3688978955 0 0) 0x7f81e40019b0 con 0x7f81d0002860 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81d00065c0 con 0x7f81d0002860 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81dc0048a0 con 0x7f81d0002860 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 10 monclient: dump: 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.951 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.952 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1814100095 0 0) 0x7f81d00065c0 con 0x7f81d0002860 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81d000a160 con 0x7f81d0002860 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 10 client.26915.objecter ms_handle_connect 0x7f81cc0416d0 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f81dc006ac0 con 0x7f81d0002860 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6789/0 conn(0x7f81d0002860 legacy=0x7f81d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.928+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6789/0 conn(0x7f81d0002860 legacy=0x7f81d0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.934+0000 7f81f290d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.934+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f81d0009e70 con 0x7f81d0009130 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81ebfff640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6789/0 conn(0x7f81d0009130 legacy=0x7f81d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 10 client.26915.objecter ms_handle_connect 0x7f81d0009130 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 10 client.26915.objecter resend_mon_ops 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142843176 0 0) 0x7f81d0009e70 con 0x7f81d0009130 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 10 monclient(hunting): my global_id is 164134 2024-08-19T09:57:09.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d00065c0 con 0x7f81d0009130 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1408715707 0 0) 0x7f81d00065c0 con 0x7f81d0009130 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.937+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81d0009e70 con 0x7f81d0009130 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f81dc0063b0 con 0x7f81d0009130 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 10 monclient: dump: 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:09.957 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1867927668 0 0) 0x7f81d0009e70 con 0x7f81d0009130 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f81d000c410 con 0x7f81d0009130 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3295963289) 0x7f81dc0066a0 con 0x7f81d0009130 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6789/0 conn(0x7f81d0009130 legacy=0x7f81d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81e97fa640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6789/0 conn(0x7f81d0009130 legacy=0x7f81d00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16750, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:09.979 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:09.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 10 librados: watch_flush enter 2024-08-19T09:57:09.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 10 librados: watch_flush exit 2024-08-19T09:57:09.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 20 client.26915.objecter shutdown clearing up homeless session... 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 10 client.26915.objecter successfully canceled tick 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6812/2549104415 conn(0x7f81cc0416d0 legacy=0x7f81cc043b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 10 monclient: shutdown 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 >> v1:172.21.15.82:6791/0 conn(0x7f81ec0a46a0 legacy=0x7f81ec1387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f0eaa640 1 -- 172.21.15.1:0/323250598 reap_dead start 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f0eaa640 1 -- 172.21.15.1:0/323250598 reap_dead start 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 shutdown_connections 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 >> 172.21.15.1:0/323250598 conn(0x7f81ec0a9020 msgr2=0x7f81ec0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 shutdown_connections 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.938+0000 7f81f290d640 1 -- 172.21.15.1:0/323250598 wait complete. 2024-08-19T09:57:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:09.939+0000 7f81f290d640 1 librados: shutdown 2024-08-19T09:57:09.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:09.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:57:09.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:57:10.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.082+0000 7fb2b63c5640 10 monclient: get_monmap_and_config 2024-08-19T09:57:10.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.082+0000 7fb2b63c5640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.082+0000 7fb2b63c5640 10 monclient: monmap: 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:10.083524+0000 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:10.083524+0000 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.084 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.085 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.085 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.085 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.085 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.083+0000 7fb2b63c5640 1 Processor -- start 2024-08-19T09:57:10.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.083+0000 7fb2b63c5640 1 -- start start 2024-08-19T09:57:10.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.083+0000 7fb2b63c5640 10 monclient: init 2024-08-19T09:57:10.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.083+0000 7fb2b63c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.083+0000 7fb2b63c5640 10 monclient: _add_conns ranks=[3,0,7,4,5,8,1,2,6] 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.083+0000 7fb2b63c5640 10 monclient(hunting): picked mon.noname-g con 0x7fb2b00a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb2b00a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 10 monclient(hunting): picked mon.noname-i con 0x7fb2b013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b00a3b40 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b00a8130 con 0x7fb2b00a67c0 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b006ede0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2b63c5640 10 monclient(hunting): authenticate will time out at 1086.160076s 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2a7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb2b00a6ba0 0x7fb2b00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38188/0 (socket says 172.21.15.1:38188) 2024-08-19T09:57:10.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.084+0000 7fb2a7fff640 1 -- 172.21.15.1:0/3433801751 learned_addr learned my addr 172.21.15.1:0/3433801751 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2a7fff640 1 -- 172.21.15.1:0/3433801751 >> v1:172.21.15.1:6789/0 conn(0x7fb2b00a6ba0 legacy=0x7fb2b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2affff640 1 -- 172.21.15.1:0/3433801751 >> v1:172.21.15.82:6790/0 conn(0x7fb2b00a67c0 legacy=0x7fb2b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2b4962640 1 -- 172.21.15.1:0/3433801751 >> v1:172.21.15.82:6792/0 conn(0x7fb2b013f5c0 legacy=0x7fb2b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553292796 0 0) 0x7fb2b00a3b40 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): my global_id is 164151 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb29c003520 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170429262 0 0) 0x7fb2b00a8130 con 0x7fb2b00a67c0 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): my global_id is 134895 2024-08-19T09:57:10.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b00a3b40 con 0x7fb2b00a67c0 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077343806 0 0) 0x7fb2b006ede0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 10 monclient(hunting): my global_id is 144448 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.086+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b00a8130 con 0x7fb2b013f5c0 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1522923094 0 0) 0x7fb29c003520 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 >> v1:172.21.15.82:6792/0 conn(0x7fb2b013f5c0 legacy=0x7fb2b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 >> v1:172.21.15.82:6790/0 conn(0x7fb2b00a67c0 legacy=0x7fb2b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 monclient: found mon.noname-a 2024-08-19T09:57:10.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:57:10.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b00a92a0 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb298001ae0 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 5 monclient: authenticate success, global_id 164151 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: dump: 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.095 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2980048a0 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 1 -- 172.21.15.1:0/3433801751 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb298004e10 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2af7fe640 10 monclient: dump: 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.097 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 4 set_mon_vals no callback set 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 10 monclient: shutdown 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 1 -- 172.21.15.1:0/3433801751 >> v1:172.21.15.1:6789/0 conn(0x7fb2b00a6ba0 legacy=0x7fb2b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 1 -- 172.21.15.1:0/3433801751 shutdown_connections 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 1 -- 172.21.15.1:0/3433801751 >> 172.21.15.1:0/3433801751 conn(0x7fb2b00ac4b0 msgr2=0x7fb2b00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 1 -- 172.21.15.1:0/3433801751 shutdown_connections 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.087+0000 7fb2b63c5640 1 -- 172.21.15.1:0/3433801751 wait complete. 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 10 monclient: monmap: 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.099 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 1 librados: starting msgr at 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 1 librados: starting objecter 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 1 Processor -- start 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 1 -- start start 2024-08-19T09:57:10.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 1 librados: setting wanted keys 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 1 librados: calling monclient init 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.088+0000 7fb2b63c5640 10 monclient: init 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient: _add_conns ranks=[3,5,4,7,8,2,6,0,1] 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient(hunting): picked mon.noname-g con 0x7fb2b00a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient(hunting): picked mon.noname-h con 0x7fb2b00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb2b013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b00a9cc0 con 0x7fb2b00a67c0 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b013ae00 con 0x7fb2b013f5c0 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b013bc70 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b63c5640 10 monclient(hunting): authenticate will time out at 1086.165288s 2024-08-19T09:57:10.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2affff640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fb2b00a67c0 0x7fb2b013c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:37182/0 (socket says 172.21.15.1:37182) 2024-08-19T09:57:10.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2affff640 1 -- 172.21.15.1:0/276997483 learned_addr learned my addr 172.21.15.1:0/276997483 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.089+0000 7fb2b4962640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb2b013f5c0 0x7fb2b01d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37440/0 (socket says 172.21.15.1:37440) 2024-08-19T09:57:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2b4962640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.1:6791/0 conn(0x7fb2b013f5c0 legacy=0x7fb2b01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 client.?.objecter ms_handle_connect 0x7fb2b013f5c0 2024-08-19T09:57:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2affff640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb2b00a67c0 legacy=0x7fb2b013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 client.?.objecter ms_handle_connect 0x7fb2b00a67c0 2024-08-19T09:57:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 client.?.objecter ms_handle_connect 0x7fb2b00a6ba0 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2a7fff640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6791/0 conn(0x7fb2b00a6ba0 legacy=0x7fb2b013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526097844 0 0) 0x7fb2b013ae00 con 0x7fb2b013f5c0 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): my global_id is 27220 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb290004eb0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2829144745 0 0) 0x7fb2b00a9cc0 con 0x7fb2b00a67c0 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): my global_id is 134904 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b013ae00 con 0x7fb2b00a67c0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719128765 0 0) 0x7fb2b013bc70 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): my global_id is 26924 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b00a9cc0 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 711324439 0 0) 0x7fb290004eb0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2b013bc70 con 0x7fb2b013f5c0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2a8002060 con 0x7fb2b013f5c0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): dump: 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1118399257 0 0) 0x7fb2b013ae00 con 0x7fb2b00a67c0 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb290004eb0 con 0x7fb2b00a67c0 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2a0003610 con 0x7fb2b00a67c0 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.091+0000 7fb2ad7fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient(hunting): dump: 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.113 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4092560468 0 0) 0x7fb2b00a9cc0 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2b013ae00 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb298004d90 con 0x7fb2b00a6ba0 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient(hunting): dump: 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3163240692 0 0) 0x7fb2b013bc70 con 0x7fb2b013f5c0 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6791/0 conn(0x7fb2b00a6ba0 legacy=0x7fb2b013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb2b00a67c0 legacy=0x7fb2b013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 monclient: found mon.c 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b013b3d0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 5 monclient: authenticate success, global_id 27220 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 10 monclient: _renew_subs 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b013e4e0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 10 client.27220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 10 monclient: _renew_subs 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2b013cf40 con 0x7fb2b013f5c0 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2b63c5640 1 librados: init done 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2a8003d50 con 0x7fb2b013f5c0 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2a8004280 con 0x7fb2b013f5c0 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2aeffd640 4 set_mon_vals no callback set 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.092+0000 7fb2ad7fa640 10 monclient: dump: 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.117 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2b63c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb2940065c0 con 0x7fb294002860 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb2a8004420 con 0x7fb2b013f5c0 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(295..295 src has 1..295) ==== 6280+0+0 (unknown 2052330141 0 0) 0x7fb2a804e7c0 con 0x7fb2b013f5c0 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2ad7fa640 10 client.27220.objecter ms_dispatch 0x7fb2b0000b90 osd_map(295..295 src has 1..295) 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2ad7fa640 3 client.27220.objecter handle_osd_map got epochs [295,295] > 0 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2ad7fa640 3 client.27220.objecter handle_osd_map decoding full epoch 295 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2ad7fa640 20 client.27220.objecter dump_active .. 0 homeless 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.093+0000 7fb2a7fff640 10 monclient: get_auth_request con 0x7fb2900419a0 auth_method 0 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 client.27220.objecter ms_handle_connect 0x7fb294002860 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 client.27220.objecter resend_mon_ops 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2affff640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb294002860 legacy=0x7fb294004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2718813768 0 0) 0x7fb2940065c0 con 0x7fb294002860 2024-08-19T09:57:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient(hunting): my global_id is 134913 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b013bc70 con 0x7fb294002860 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 client.27220.objecter ms_handle_connect 0x7fb2900419a0 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 608474890 0 0) 0x7fb2b013bc70 con 0x7fb294002860 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2940065c0 con 0x7fb294002860 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2a0003f20 con 0x7fb294002860 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.095+0000 7fb2ad7fa640 10 monclient: dump: 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 545029886 0 0) 0x7fb2940065c0 con 0x7fb294002860 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb29400a160 con 0x7fb294002860 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb2a0004150 con 0x7fb294002860 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb294002860 legacy=0x7fb294004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.096+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb294002860 legacy=0x7fb294004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.102+0000 7fb2b63c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.102+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb294009e70 con 0x7fb294009130 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2affff640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb294009130 legacy=0x7fb2940095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 client.27220.objecter ms_handle_connect 0x7fb294009130 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 client.27220.objecter resend_mon_ops 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3802958417 0 0) 0x7fb294009e70 con 0x7fb294009130 2024-08-19T09:57:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient(hunting): my global_id is 134922 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2940065c0 con 0x7fb294009130 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4265020883 0 0) 0x7fb2940065c0 con 0x7fb294009130 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb294009e70 con 0x7fb294009130 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2a0005b00 con 0x7fb294009130 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 10 monclient: dump: 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.123 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4108004188 0 0) 0x7fb294009e70 con 0x7fb294009130 2024-08-19T09:57:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.105+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb29400c3f0 con 0x7fb294009130 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2936453372) 0x7fb2a0006ff0 con 0x7fb294009130 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2ad7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2ad7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb294009130 legacy=0x7fb2940095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2ad7fa640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6790/0 conn(0x7fb294009130 legacy=0x7fb2940095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:10.125 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 287166, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:10.126 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.127 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.128 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:10.129 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.130 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.131 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:10.132 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:10.136 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:10.137 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:10.140 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.144 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:10.145 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:10.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 10 librados: watch_flush enter 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 10 librados: watch_flush exit 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 20 client.27220.objecter shutdown clearing up homeless session... 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 10 client.27220.objecter successfully canceled tick 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb2900419a0 legacy=0x7fb290043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 10 monclient: shutdown 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 >> v1:172.21.15.1:6791/0 conn(0x7fb2b013f5c0 legacy=0x7fb2b01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b4962640 1 -- 172.21.15.1:0/276997483 reap_dead start 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b4962640 1 -- 172.21.15.1:0/276997483 reap_dead start 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 shutdown_connections 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 >> 172.21.15.1:0/276997483 conn(0x7fb2b00ac4b0 msgr2=0x7fb2b013edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 shutdown_connections 2024-08-19T09:57:10.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 -- 172.21.15.1:0/276997483 wait complete. 2024-08-19T09:57:10.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.106+0000 7fb2b63c5640 1 librados: shutdown 2024-08-19T09:57:10.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:10.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:57:10.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.248+0000 7fb727838640 10 monclient: get_monmap_and_config 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.248+0000 7fb727838640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.248+0000 7fb727838640 10 monclient: monmap: 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:10.249739+0000 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:10.249739+0000 2024-08-19T09:57:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.251 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.249+0000 7fb727838640 1 Processor -- start 2024-08-19T09:57:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.249+0000 7fb727838640 1 -- start start 2024-08-19T09:57:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.249+0000 7fb727838640 10 monclient: init 2024-08-19T09:57:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient: _add_conns ranks=[2,6,3,4,0,7,5,8,1] 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient(hunting): picked mon.noname-b con 0x7fb7200a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient(hunting): picked mon.noname-d con 0x7fb7200a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient(hunting): picked mon.noname-g con 0x7fb72013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7200a3b40 con 0x7fb7200a67c0 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7200a8130 con 0x7fb72013f5c0 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb72006ede0 con 0x7fb7200a6ba0 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb727838640 10 monclient(hunting): authenticate will time out at 1086.326155s 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb724dd3640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fb7200a6ba0 0x7fb7200a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36900/0 (socket says 172.21.15.1:36900) 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb7255d4640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb7200a67c0 0x7fb7200a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48006/0 (socket says 172.21.15.1:48006) 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.250+0000 7fb724dd3640 1 -- 172.21.15.1:0/1213081583 learned_addr learned my addr 172.21.15.1:0/1213081583 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb7255d4640 1 -- 172.21.15.1:0/1213081583 >> v1:172.21.15.1:6790/0 conn(0x7fb7200a67c0 legacy=0x7fb7200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb724dd3640 1 -- 172.21.15.1:0/1213081583 >> v1:172.21.15.1:6792/0 conn(0x7fb7200a6ba0 legacy=0x7fb7200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb725dd5640 1 -- 172.21.15.1:0/1213081583 >> v1:172.21.15.82:6790/0 conn(0x7fb72013f5c0 legacy=0x7fb7201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258438695 0 0) 0x7fb7200a3b40 con 0x7fb7200a67c0 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): my global_id is 164162 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb708003520 con 0x7fb7200a67c0 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737295965 0 0) 0x7fb72006ede0 con 0x7fb7200a6ba0 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): my global_id is 164202 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7200a3b40 con 0x7fb7200a6ba0 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041081793 0 0) 0x7fb7200a8130 con 0x7fb72013f5c0 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient(hunting): my global_id is 134931 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb72006ede0 con 0x7fb72013f5c0 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 877871300 0 0) 0x7fb708003520 con 0x7fb7200a67c0 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb724dd3640 1 --1- 172.21.15.1:0/1213081583 >> v1:172.21.15.1:6792/0 conn(0x7fb7200a6ba0 0x7fb7200a6fb0 :-1 s=OPENED pgs=3338 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fb714001980 mon_map magic: 0 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 >> v1:172.21.15.1:6792/0 conn(0x7fb7200a6ba0 legacy=0x7fb7200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 >> v1:172.21.15.82:6790/0 conn(0x7fb72013f5c0 legacy=0x7fb7201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 monclient: found mon.noname-b 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7200a92a0 con 0x7fb7200a67c0 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb718001ae0 con 0x7fb7200a67c0 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb727838640 5 monclient: authenticate success, global_id 164162 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb727838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: dump: 2024-08-19T09:57:10.260 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb727838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb714001980 con 0x7fb7200a6ba0 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.252+0000 7fb71ffff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7180047e0 con 0x7fb7200a67c0 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb71ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb71ffff640 1 -- 172.21.15.1:0/1213081583 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb718004d50 con 0x7fb7200a67c0 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 4 set_mon_vals no callback set 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb71ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb71ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:10.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb71ffff640 10 monclient: dump: 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 10 monclient: shutdown 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 1 -- 172.21.15.1:0/1213081583 >> v1:172.21.15.1:6790/0 conn(0x7fb7200a67c0 legacy=0x7fb7200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 1 -- 172.21.15.1:0/1213081583 shutdown_connections 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 1 -- 172.21.15.1:0/1213081583 >> 172.21.15.1:0/1213081583 conn(0x7fb7200ac4b0 msgr2=0x7fb7200a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 1 -- 172.21.15.1:0/1213081583 shutdown_connections 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.253+0000 7fb727838640 1 -- 172.21.15.1:0/1213081583 wait complete. 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 10 monclient: monmap: 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:10.264 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.265 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 1 librados: starting msgr at 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 1 librados: starting objecter 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 1 Processor -- start 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 1 -- start start 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 1 librados: setting wanted keys 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 1 librados: calling monclient init 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 10 monclient: init 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.254+0000 7fb727838640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 10 monclient: _add_conns ranks=[2,3,0,1,8,4,6,5,7] 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 10 monclient(hunting): picked mon.noname-b con 0x7fb7200a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 10 monclient(hunting): picked mon.noname-g con 0x7fb7200a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:10.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 10 monclient(hunting): picked mon.noname-a con 0x7fb72013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb72013d150 con 0x7fb72013f5c0 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb72013d430 con 0x7fb7200a67c0 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7201cb0c0 con 0x7fb7200a6ba0 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb727838640 10 monclient(hunting): authenticate will time out at 1086.331092s 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb7255d4640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb7200a67c0 0x7fb7201ca2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48016/0 (socket says 172.21.15.1:48016) 2024-08-19T09:57:10.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.255+0000 7fb7255d4640 1 -- 172.21.15.1:0/2324304633 learned_addr learned my addr 172.21.15.1:0/2324304633 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb725dd5640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.1:6789/0 conn(0x7fb72013f5c0 legacy=0x7fb7201d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 client.?.objecter ms_handle_connect 0x7fb72013f5c0 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 client.?.objecter ms_handle_connect 0x7fb7200a67c0 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb7255d4640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.1:6790/0 conn(0x7fb7200a67c0 legacy=0x7fb7201ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784254269 0 0) 0x7fb72013d150 con 0x7fb72013f5c0 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): my global_id is 164160 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb70c004eb0 con 0x7fb72013f5c0 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1699775868 0 0) 0x7fb72013d430 con 0x7fb7200a67c0 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): my global_id is 164171 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb72013d150 con 0x7fb7200a67c0 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 client.?.objecter ms_handle_connect 0x7fb7200a6ba0 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb724dd3640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6790/0 conn(0x7fb7200a6ba0 legacy=0x7fb7201ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 110089440 0 0) 0x7fb70c004eb0 con 0x7fb72013f5c0 2024-08-19T09:57:10.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb72013d430 con 0x7fb72013f5c0 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb710002020 con 0x7fb72013f5c0 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): dump: 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.276 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 274642401 0 0) 0x7fb7201cb0c0 con 0x7fb7200a6ba0 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): my global_id is 134940 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb70c004eb0 con 0x7fb7200a6ba0 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3977527902 0 0) 0x7fb72013d150 con 0x7fb7200a67c0 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7201cb0c0 con 0x7fb7200a67c0 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb718004cd0 con 0x7fb7200a67c0 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient(hunting): dump: 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.278 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1682041090 0 0) 0x7fb72013d430 con 0x7fb72013f5c0 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6790/0 conn(0x7fb7200a6ba0 legacy=0x7fb7201ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.1:6790/0 conn(0x7fb7200a67c0 legacy=0x7fb7201ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 monclient: found mon.a 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72013c130 con 0x7fb72013f5c0 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.257+0000 7fb71dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 5 monclient: authenticate success, global_id 164160 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 10 monclient: _renew_subs 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7200a9fc0 con 0x7fb72013f5c0 2024-08-19T09:57:10.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb710003c40 con 0x7fb72013f5c0 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb7100041b0 con 0x7fb72013f5c0 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71dffb640 10 monclient: dump: 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 10 client.164160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:10.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 10 monclient: _renew_subs 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7201d72a0 con 0x7fb72013f5c0 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb71f7fe640 4 set_mon_vals no callback set 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 1 librados: init done 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.258+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7201429d0 con 0x7fb72009da50 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb710004350 con 0x7fb72013f5c0 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(295..295 src has 1..295) ==== 6280+0+0 (unknown 2052330141 0 0) 0x7fb71004e730 con 0x7fb72013f5c0 2024-08-19T09:57:10.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb71dffb640 10 client.164160.objecter ms_dispatch 0x7fb720000b90 osd_map(295..295 src has 1..295) 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb71dffb640 3 client.164160.objecter handle_osd_map got epochs [295,295] > 0 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb71dffb640 3 client.164160.objecter handle_osd_map decoding full epoch 295 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb71dffb640 20 client.164160.objecter dump_active .. 0 homeless 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.259+0000 7fb724dd3640 10 monclient: get_auth_request con 0x7fb70c041710 auth_method 0 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 client.164160.objecter ms_handle_connect 0x7fb72009da50 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 client.164160.objecter resend_mon_ops 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb7255d4640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6791/0 conn(0x7fb72009da50 legacy=0x7fb72009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786645237 0 0) 0x7fb7201429d0 con 0x7fb72009da50 2024-08-19T09:57:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient(hunting): my global_id is 26933 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb72013d430 con 0x7fb72009da50 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1423870248 0 0) 0x7fb72013d430 con 0x7fb72009da50 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7201429d0 con 0x7fb72009da50 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb7180047e0 con 0x7fb72009da50 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 monclient: dump: 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 10 client.164160.objecter ms_handle_connect 0x7fb70c041710 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2737933219 0 0) 0x7fb7201429d0 con 0x7fb72009da50 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.261+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7200008d0 con 0x7fb72009da50 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.262+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb718005460 con 0x7fb72009da50 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.262+0000 7fb71dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.262+0000 7fb71dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.262+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6791/0 conn(0x7fb72009da50 legacy=0x7fb72009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.262+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6791/0 conn(0x7fb72009da50 legacy=0x7fb72009de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.268+0000 7fb727838640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.268+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb72009e920 con 0x7fb72009e540 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.270+0000 7fb7255d4640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6791/0 conn(0x7fb72009e540 legacy=0x7fb7201db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.270+0000 7fb71dffb640 10 client.164160.objecter ms_handle_connect 0x7fb72009e540 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.270+0000 7fb71dffb640 10 client.164160.objecter resend_mon_ops 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 320670838 0 0) 0x7fb72009e920 con 0x7fb72009e540 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient(hunting): my global_id is 26942 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7201429d0 con 0x7fb72009e540 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3217728172 0 0) 0x7fb7201429d0 con 0x7fb72009e540 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb72009e920 con 0x7fb72009e540 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb718005c60 con 0x7fb72009e540 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:10.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient: dump: 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3075620908 0 0) 0x7fb72009e920 con 0x7fb72009e540 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7201ded40 con 0x7fb72009e540 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 1114744372) 0x7fb7180060e0 con 0x7fb72009e540 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6791/0 conn(0x7fb72009e540 legacy=0x7fb7201db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.271+0000 7fb71dffb640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6791/0 conn(0x7fb72009e540 legacy=0x7fb7201db490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 436536, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:10.312 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 10 librados: watch_flush enter 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 10 librados: watch_flush exit 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 20 client.164160.objecter shutdown clearing up homeless session... 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 10 client.164160.objecter successfully canceled tick 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb70c041710 legacy=0x7fb70c043bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 10 monclient: shutdown 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 >> v1:172.21.15.1:6789/0 conn(0x7fb72013f5c0 legacy=0x7fb7201d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb725dd5640 1 -- 172.21.15.1:0/2324304633 reap_dead start 2024-08-19T09:57:10.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb725dd5640 1 -- 172.21.15.1:0/2324304633 reap_dead start 2024-08-19T09:57:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 shutdown_connections 2024-08-19T09:57:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 >> 172.21.15.1:0/2324304633 conn(0x7fb7200ac4b0 msgr2=0x7fb7201419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 shutdown_connections 2024-08-19T09:57:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 -- 172.21.15.1:0/2324304633 wait complete. 2024-08-19T09:57:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.272+0000 7fb727838640 1 librados: shutdown 2024-08-19T09:57:10.315 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:10.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:57:10.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:57:10.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.414+0000 7f60e91f4640 10 monclient: get_monmap_and_config 2024-08-19T09:57:10.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.414+0000 7f60e91f4640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.414+0000 7f60e91f4640 10 monclient: monmap: 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:10.415097+0000 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:10.415097+0000 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.416 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.414+0000 7f60e91f4640 1 Processor -- start 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.414+0000 7f60e91f4640 1 -- start start 2024-08-19T09:57:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.414+0000 7f60e91f4640 10 monclient: init 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient: _add_conns ranks=[7,4,0,1,2,3,8,5,6] 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient(hunting): picked mon.noname-i con 0x7f60e40a6830 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient(hunting): picked mon.noname-c con 0x7f60e406f280 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient(hunting): picked mon.noname-a con 0x7f60e413f840 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e40a3860 con 0x7f60e413f840 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e40a3b40 con 0x7f60e406f280 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e40a7a30 con 0x7f60e40a6830 2024-08-19T09:57:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e91f4640 10 monclient(hunting): authenticate will time out at 1086.491388s 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e359e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f60e413f840 0x7f60e4141c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38200/0 (socket says 172.21.15.1:38200) 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60e359e640 1 -- 172.21.15.1:0/613472341 learned_addr learned my addr 172.21.15.1:0/613472341 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.415+0000 7f60da59c640 1 --1- 172.21.15.1:0/613472341 >> v1:172.21.15.1:6791/0 conn(0x7f60e406f280 0x7f60e40a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37444/0 (socket says 172.21.15.1:37444) 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e359e640 1 -- 172.21.15.1:0/613472341 >> v1:172.21.15.1:6789/0 conn(0x7f60e413f840 legacy=0x7f60e4141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60da59c640 1 -- 172.21.15.1:0/613472341 >> v1:172.21.15.1:6791/0 conn(0x7f60e406f280 legacy=0x7f60e40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e2d9d640 1 -- 172.21.15.1:0/613472341 >> v1:172.21.15.82:6792/0 conn(0x7f60e40a6830 legacy=0x7f60e40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70737558 0 0) 0x7f60e40a3860 con 0x7f60e413f840 2024-08-19T09:57:10.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): my global_id is 164169 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60d0003520 con 0x7f60e413f840 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133966718 0 0) 0x7f60e40a3b40 con 0x7f60e406f280 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 10 monclient(hunting): my global_id is 27229 2024-08-19T09:57:10.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.417+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e40a3860 con 0x7f60e406f280 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216732909 0 0) 0x7f60e40a7a30 con 0x7f60e40a6830 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient(hunting): my global_id is 144457 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e40a3b40 con 0x7f60e40a6830 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982264760 0 0) 0x7f60d0003520 con 0x7f60e413f840 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 >> v1:172.21.15.82:6792/0 conn(0x7f60e40a6830 legacy=0x7f60e40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 >> v1:172.21.15.1:6791/0 conn(0x7f60e406f280 legacy=0x7f60e40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 monclient: found mon.noname-a 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e40a2f20 con 0x7f60e413f840 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60dc001e60 con 0x7f60e413f840 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 5 monclient: authenticate success, global_id 164169 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: dump: 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.429 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60dc004c20 con 0x7f60e413f840 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 1 -- 172.21.15.1:0/613472341 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60dc005190 con 0x7f60e413f840 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 4 set_mon_vals no callback set 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:10.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e259c640 10 monclient: dump: 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 10 monclient: shutdown 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 1 -- 172.21.15.1:0/613472341 >> v1:172.21.15.1:6789/0 conn(0x7f60e413f840 legacy=0x7f60e4141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 1 -- 172.21.15.1:0/613472341 shutdown_connections 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 1 -- 172.21.15.1:0/613472341 >> 172.21.15.1:0/613472341 conn(0x7f60e40ac6d0 msgr2=0x7f60e40a8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 1 -- 172.21.15.1:0/613472341 shutdown_connections 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.418+0000 7f60e91f4640 1 -- 172.21.15.1:0/613472341 wait complete. 2024-08-19T09:57:10.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.419+0000 7f60e91f4640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.419+0000 7f60e91f4640 10 monclient: monmap: 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.433 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.419+0000 7f60e91f4640 1 librados: starting msgr at 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.419+0000 7f60e91f4640 1 librados: starting objecter 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.419+0000 7f60e91f4640 1 Processor -- start 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 1 -- start start 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 1 librados: setting wanted keys 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 1 librados: calling monclient init 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient: init 2024-08-19T09:57:10.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient: _add_conns ranks=[8,2,1,0,5,4,7,3,6] 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient(hunting): picked mon.noname-e con 0x7f60e406f280 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient(hunting): picked mon.noname-b con 0x7f60e40a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient(hunting): picked mon.noname-f con 0x7f60e413f840 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e413d740 con 0x7f60e413f840 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e41259a0 con 0x7f60e40a6830 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e41232a0 con 0x7f60e406f280 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e91f4640 10 monclient(hunting): authenticate will time out at 1086.496628s 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e2d9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f60e406f280 0x7f60e4122480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52928/0 (socket says 172.21.15.1:52928) 2024-08-19T09:57:10.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.420+0000 7f60e2d9d640 1 -- 172.21.15.1:0/1379062737 learned_addr learned my addr 172.21.15.1:0/1379062737 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60e2d9d640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.1:6793/0 conn(0x7f60e406f280 legacy=0x7f60e4122480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60da59c640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.1:6790/0 conn(0x7f60e40a6830 legacy=0x7f60e4122b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60dbfff640 10 client.?.objecter ms_handle_connect 0x7f60e406f280 2024-08-19T09:57:10.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60dbfff640 10 client.?.objecter ms_handle_connect 0x7f60e40a6830 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456032545 0 0) 0x7f60e41259a0 con 0x7f60e40a6830 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): my global_id is 164180 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.422+0000 7f60e359e640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6789/0 conn(0x7f60e413f840 legacy=0x7f60e41d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60c4004ee0 con 0x7f60e40a6830 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 client.?.objecter ms_handle_connect 0x7f60e413f840 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907246468 0 0) 0x7f60e41232a0 con 0x7f60e406f280 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): my global_id is 27773 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e41259a0 con 0x7f60e406f280 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040476007 0 0) 0x7f60e413d740 con 0x7f60e413f840 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): my global_id is 164143 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e41232a0 con 0x7f60e413f840 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1602200250 0 0) 0x7f60c4004ee0 con 0x7f60e40a6830 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60e413d740 con 0x7f60e40a6830 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60cc0047e0 con 0x7f60e40a6830 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): dump: 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.445 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.446 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2765571014 0 0) 0x7f60e41259a0 con 0x7f60e406f280 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60c4004ee0 con 0x7f60e406f280 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60d4003610 con 0x7f60e406f280 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:10.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient(hunting): dump: 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.448 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3844129578 0 0) 0x7f60e413d740 con 0x7f60e40a6830 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.1:6793/0 conn(0x7f60e406f280 legacy=0x7f60e4122480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6789/0 conn(0x7f60e413f840 legacy=0x7f60e41d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 monclient: found mon.b 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e40a9d30 con 0x7f60e40a6830 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 5 monclient: authenticate success, global_id 164180 2024-08-19T09:57:10.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 10 monclient: _renew_subs 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60e4142c50 con 0x7f60e40a6830 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 10 client.164180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 10 monclient: _renew_subs 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f60e4123640 con 0x7f60e40a6830 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e91f4640 1 librados: init done 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60cc004c30 con 0x7f60e40a6830 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60cc005120 con 0x7f60e40a6830 2024-08-19T09:57:10.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60e1d9b640 4 set_mon_vals no callback set 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.423+0000 7f60dbfff640 10 monclient: dump: 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.452 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60e91f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f60c80065c0 con 0x7f60c8002860 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f60cc0053c0 con 0x7f60e40a6830 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(295..295 src has 1..295) ==== 6280+0+0 (unknown 2052330141 0 0) 0x7f60cc04e540 con 0x7f60e40a6830 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60dbfff640 10 client.164180.objecter ms_dispatch 0x7f60e4000b90 osd_map(295..295 src has 1..295) 2024-08-19T09:57:10.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60dbfff640 3 client.164180.objecter handle_osd_map got epochs [295,295] > 0 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60dbfff640 3 client.164180.objecter handle_osd_map decoding full epoch 295 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60dbfff640 20 client.164180.objecter dump_active .. 0 homeless 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.424+0000 7f60e359e640 10 monclient: get_auth_request con 0x7f60c4041620 auth_method 0 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 10 client.164180.objecter ms_handle_connect 0x7f60c8002860 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 10 client.164180.objecter resend_mon_ops 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60e2d9d640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6792/0 conn(0x7f60c8002860 legacy=0x7f60c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914523071 0 0) 0x7f60c80065c0 con 0x7f60c8002860 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 10 monclient(hunting): my global_id is 144466 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.426+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e413d740 con 0x7f60c8002860 2024-08-19T09:57:10.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 10 client.164180.objecter ms_handle_connect 0x7f60c4041620 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2116563408 0 0) 0x7f60e413d740 con 0x7f60c8002860 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60c80065c0 con 0x7f60c8002860 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60d4003fe0 con 0x7f60c8002860 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 10 monclient: dump: 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.455 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2437654920 0 0) 0x7f60c80065c0 con 0x7f60c8002860 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60c800a160 con 0x7f60c8002860 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f60d4004290 con 0x7f60c8002860 2024-08-19T09:57:10.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6792/0 conn(0x7f60c8002860 legacy=0x7f60c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.427+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6792/0 conn(0x7f60c8002860 legacy=0x7f60c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.433+0000 7f60e91f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.433+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f60c8009e50 con 0x7f60c8009110 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60e2d9d640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6792/0 conn(0x7f60c8009110 legacy=0x7f60c80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 client.164180.objecter ms_handle_connect 0x7f60c8009110 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 client.164180.objecter resend_mon_ops 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392729403 0 0) 0x7f60c8009e50 con 0x7f60c8009110 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient(hunting): my global_id is 144475 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60c80065c0 con 0x7f60c8009110 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2197704335 0 0) 0x7f60c80065c0 con 0x7f60c8009110 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60c8009e50 con 0x7f60c8009110 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60d4004440 con 0x7f60c8009110 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 10 monclient: dump: 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 262240143 0 0) 0x7f60c8009e50 con 0x7f60c8009110 2024-08-19T09:57:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.436+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60c800c3a0 con 0x7f60c8009110 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7851 (unknown 0 0 2875685048) 0x7f60d4005e60 con 0x7f60c8009110 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60dbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6792/0 conn(0x7f60c8009110 legacy=0x7f60c80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60dbfff640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6792/0 conn(0x7f60c8009110 legacy=0x7f60c80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:10.460 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 154815, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:10.461 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.462 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:10.464 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:10.464 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.465 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.465 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.465 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.465 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:10.466 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:10.467 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:10.472 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:10.473 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.475 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:10.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.478 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:10.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.482 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.483 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:10.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:10.485 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:10.485 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:10.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 10 librados: watch_flush enter 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 10 librados: watch_flush exit 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 20 client.164180.objecter shutdown clearing up homeless session... 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 10 client.164180.objecter successfully canceled tick 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.82:6812/2549104415 conn(0x7f60c4041620 legacy=0x7f60c4043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 10 monclient: shutdown 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 >> v1:172.21.15.1:6790/0 conn(0x7f60e40a6830 legacy=0x7f60e4122b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e359e640 1 -- 172.21.15.1:0/1379062737 reap_dead start 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e359e640 1 -- 172.21.15.1:0/1379062737 reap_dead start 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 shutdown_connections 2024-08-19T09:57:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 >> 172.21.15.1:0/1379062737 conn(0x7f60e40ac6d0 msgr2=0x7f60e4141e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 shutdown_connections 2024-08-19T09:57:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 -- 172.21.15.1:0/1379062737 wait complete. 2024-08-19T09:57:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.437+0000 7f60e91f4640 1 librados: shutdown 2024-08-19T09:57:10.488 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:10.488 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T09:57:10.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T09:57:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.585+0000 7fbc4ac84640 10 monclient: get_monmap_and_config 2024-08-19T09:57:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.585+0000 7fbc4ac84640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.586+0000 7fbc4ac84640 10 monclient: monmap: 2024-08-19T09:57:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:10.586907+0000 2024-08-19T09:57:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:10.586907+0000 2024-08-19T09:57:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.589 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.586+0000 7fbc4ac84640 1 Processor -- start 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.586+0000 7fbc4ac84640 1 -- start start 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.586+0000 7fbc4ac84640 10 monclient: init 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient: _add_conns ranks=[1,2,6,3,0,5,7,4,8] 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient(hunting): picked mon.noname-f con 0x7fbc440a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient(hunting): picked mon.noname-b con 0x7fbc440a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient(hunting): picked mon.noname-d con 0x7fbc4413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc440a3b40 con 0x7fbc440a67c0 2024-08-19T09:57:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc440a8130 con 0x7fbc440a6ba0 2024-08-19T09:57:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4406ede0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc4ac84640 10 monclient(hunting): authenticate will time out at 1086.663290s 2024-08-19T09:57:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc49221640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fbc4413f5c0 0x7fbc441419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36904/0 (socket says 172.21.15.1:36904) 2024-08-19T09:57:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.587+0000 7fbc49221640 1 -- 172.21.15.1:0/2941571336 learned_addr learned my addr 172.21.15.1:0/2941571336 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc49221640 1 -- 172.21.15.1:0/2941571336 >> v1:172.21.15.1:6792/0 conn(0x7fbc4413f5c0 legacy=0x7fbc441419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc43fff640 1 -- 172.21.15.1:0/2941571336 >> v1:172.21.15.1:6790/0 conn(0x7fbc440a6ba0 legacy=0x7fbc440a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc48a20640 1 -- 172.21.15.1:0/2941571336 >> v1:172.21.15.82:6789/0 conn(0x7fbc440a67c0 legacy=0x7fbc440a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1419378690 0 0) 0x7fbc4406ede0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc437fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc437fe640 10 monclient(hunting): my global_id is 164211 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.589+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc2c003520 con 0x7fbc4413f5c0 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045873407 0 0) 0x7fbc440a8130 con 0x7fbc440a6ba0 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): my global_id is 164189 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc4406ede0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998387229 0 0) 0x7fbc440a3b40 con 0x7fbc440a67c0 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient(hunting): my global_id is 164152 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc440a8130 con 0x7fbc440a67c0 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 549151995 0 0) 0x7fbc2c003520 con 0x7fbc4413f5c0 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 >> v1:172.21.15.1:6790/0 conn(0x7fbc440a6ba0 legacy=0x7fbc440a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 >> v1:172.21.15.82:6789/0 conn(0x7fbc440a67c0 legacy=0x7fbc440a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 monclient: found mon.noname-d 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc440a92a0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc3c001ad0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: dump: 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 5 monclient: authenticate success, global_id 164211 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc3c0048a0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 1 -- 172.21.15.1:0/2941571336 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc3c004e10 con 0x7fbc4413f5c0 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 4 set_mon_vals no callback set 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc437fe640 10 monclient: dump: 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 10 monclient: shutdown 2024-08-19T09:57:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 1 -- 172.21.15.1:0/2941571336 >> v1:172.21.15.1:6792/0 conn(0x7fbc4413f5c0 legacy=0x7fbc441419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 1 -- 172.21.15.1:0/2941571336 shutdown_connections 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 1 -- 172.21.15.1:0/2941571336 >> 172.21.15.1:0/2941571336 conn(0x7fbc440ac4b0 msgr2=0x7fbc440a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 1 -- 172.21.15.1:0/2941571336 shutdown_connections 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.590+0000 7fbc4ac84640 1 -- 172.21.15.1:0/2941571336 wait complete. 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 10 monclient: build_initial_monmap 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 10 monclient: monmap: 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 1 librados: starting msgr at 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 1 librados: starting objecter 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 1 Processor -- start 2024-08-19T09:57:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 1 -- start start 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 1 librados: setting wanted keys 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 1 librados: calling monclient init 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.591+0000 7fbc4ac84640 10 monclient: init 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient: _add_conns ranks=[1,3,5,7,2,0,4,8,6] 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient(hunting): picked mon.noname-f con 0x7fbc440a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient(hunting): picked mon.noname-g con 0x7fbc440a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient(hunting): picked mon.noname-h con 0x7fbc4413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4413c4d0 con 0x7fbc440a67c0 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc440a2a30 con 0x7fbc440a6ba0 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc440aa240 con 0x7fbc4413f5c0 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc4ac84640 10 monclient(hunting): authenticate will time out at 1086.668610s 2024-08-19T09:57:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc48a20640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fbc440a67c0 0x7fbc44139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:38384/0 (socket says 172.21.15.1:38384) 2024-08-19T09:57:10.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.592+0000 7fbc48a20640 1 -- 172.21.15.1:0/1260022511 learned_addr learned my addr 172.21.15.1:0/1260022511 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc43fff640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6790/0 conn(0x7fbc440a6ba0 legacy=0x7fbc4413a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc417fa640 10 client.?.objecter ms_handle_connect 0x7fbc440a6ba0 2024-08-19T09:57:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc417fa640 10 client.?.objecter ms_handle_connect 0x7fbc440a67c0 2024-08-19T09:57:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc48a20640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6789/0 conn(0x7fbc440a67c0 legacy=0x7fbc44139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc417fa640 10 client.?.objecter ms_handle_connect 0x7fbc4413f5c0 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.594+0000 7fbc49221640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6791/0 conn(0x7fbc4413f5c0 legacy=0x7fbc441d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376032298 0 0) 0x7fbc440a2a30 con 0x7fbc440a6ba0 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): my global_id is 134949 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc30004eb0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3389239738 0 0) 0x7fbc4413c4d0 con 0x7fbc440a67c0 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): my global_id is 164161 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc440a2a30 con 0x7fbc440a67c0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286258380 0 0) 0x7fbc440aa240 con 0x7fbc4413f5c0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): my global_id is 26951 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc4413c4d0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3879579912 0 0) 0x7fbc30004eb0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc440aa240 con 0x7fbc440a6ba0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc38003990 con 0x7fbc440a6ba0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:57:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): dump: 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.613 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 601677726 0 0) 0x7fbc4413c4d0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc30004eb0 con 0x7fbc4413f5c0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3150900669 0 0) 0x7fbc440a2a30 con 0x7fbc440a67c0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc4413c4d0 con 0x7fbc440a67c0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc3c001d20 con 0x7fbc4413f5c0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): dump: 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.615 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc34003610 con 0x7fbc440a67c0 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient(hunting): dump: 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 535372443 0 0) 0x7fbc440aa240 con 0x7fbc440a6ba0 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6791/0 conn(0x7fbc4413f5c0 legacy=0x7fbc441d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6789/0 conn(0x7fbc440a67c0 legacy=0x7fbc44139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: _finish_hunting 0 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 monclient: found mon.g 2024-08-19T09:57:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4413c1f0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: _finish_auth 0 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: _check_auth_tickets 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 5 monclient: authenticate success, global_id 134949 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 10 monclient: _renew_subs 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4413a740 con 0x7fbc440a6ba0 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 10 client.134949.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 10 monclient: _renew_subs 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc4413f1e0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc4ac84640 1 librados: init done 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc38003ea0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc38004390 con 0x7fbc440a6ba0 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc42ffd640 4 set_mon_vals no callback set 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.595+0000 7fbc417fa640 10 monclient: dump: 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.596+0000 7fbc4ac84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.596+0000 7fbc4ac84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.596+0000 7fbc4ac84640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.596+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc24005080 con 0x7fbc440a6ba0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbc38004530 con 0x7fbc440a6ba0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(295..295 src has 1..295) ==== 6280+0+0 (unknown 2052330141 0 0) 0x7fbc3804e8d0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc417fa640 10 client.134949.objecter ms_dispatch 0x7fbc44000b90 osd_map(295..295 src has 1..295) 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc417fa640 3 client.134949.objecter handle_osd_map got epochs [295,295] > 0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc417fa640 3 client.134949.objecter handle_osd_map decoding full epoch 295 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc417fa640 20 client.134949.objecter dump_active .. 0 homeless 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.597+0000 7fbc48a20640 10 monclient: get_auth_request con 0x7fbc30041a90 auth_method 0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.599+0000 7fbc417fa640 10 client.134949.objecter ms_handle_connect 0x7fbc30041a90 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.601+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fbc380178d0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.601+0000 7fbc417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:10.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.601+0000 7fbc417fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.855+0000 7fbc4ac84640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T09:57:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.855+0000 7fbc4ac84640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T09:57:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.855+0000 7fbc4ac84640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.855+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fbc24005c40 con 0x7fbc440a6ba0 2024-08-19T09:57:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.857+0000 7fbc417fa640 1 -- 172.21.15.1:0/1260022511 <== mon.3 v1:172.21.15.82:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fbc380171d0 con 0x7fbc440a6ba0 2024-08-19T09:57:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.857+0000 7fbc417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T09:57:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.857+0000 7fbc417fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.860+0000 7fbc4ac84640 10 librados: watch_flush enter 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.860+0000 7fbc4ac84640 10 librados: watch_flush exit 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.860+0000 7fbc4ac84640 20 client.134949.objecter shutdown clearing up homeless session... 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.860+0000 7fbc4ac84640 10 client.134949.objecter successfully canceled tick 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbc30041a90 legacy=0x7fbc30043f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 10 monclient: shutdown 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 >> v1:172.21.15.82:6790/0 conn(0x7fbc440a6ba0 legacy=0x7fbc4413a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 shutdown_connections 2024-08-19T09:57:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 >> 172.21.15.1:0/1260022511 conn(0x7fbc440ac4b0 msgr2=0x7fbc44141bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 shutdown_connections 2024-08-19T09:57:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 -- 172.21.15.1:0/1260022511 wait complete. 2024-08-19T09:57:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:10.861+0000 7fbc4ac84640 1 librados: shutdown 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=11,logm=55} crc {auth=567219510,config=2586702516,health=1261236759,kv=3855294130,logm=2882949104}) 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=458893466,config=2586702516,health=1357678568,kv=178558022,logm=2724106134}) 2024-08-19T09:57:10.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=89643662,config=2586702516,health=4180235331,kv=1455257465,logm=3064522619}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=21,config=2,health=11,kv=10,logm=56} crc {auth=4142383769,config=2586702516,health=249806387,kv=2657046684,logm=1417356489}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=3145369119,config=2586702516,health=1995790572,kv=1455257465,logm=2383288992}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=1215183272,config=2586702516,health=2572944309,kv=2971667195,logm=185418457}) 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=23,config=2,health=11,kv=12,logm=52} crc {auth=2654141596,config=2586702516,health=433126095,kv=1455257465,logm=1624670161}) 2024-08-19T09:57:10.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1928868192,config=2586702516,health=1995790572,kv=3796480173,logm=1743009657}) 2024-08-19T09:57:10.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.867+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=23,config=1,health=12,kv=12,logm=52} crc {auth=3460936913,config=1747058506,health=1995790572,kv=1455257465,logm=2514694241}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866549697}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878638053}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2624787908}) 2024-08-19T09:57:10.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3013513749}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=879164172}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3620830353}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2180424054}) 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179987417}) 2024-08-19T09:57:10.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.869+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4193712035}) 2024-08-19T09:57:10.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP::163:head 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:5d165639:::164:head 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f43765fc:::165:head 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b4c720e9:::166:head 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e694b040:::167:head 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:afa38db2:::168:head 2024-08-19T09:57:10.880 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:77ba9f53:::169:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:87495034:::170:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7c96bf0e:::171:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:dbe346cc:::172:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:e943ec24:::173:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f97a9c0c:::174:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:6f26e74d:::175:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4f95e106:::176:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:0e6f2f8f:::177:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:05db05f1:::178:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:38a78d66:::179:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:d095610b:::180:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:a1a9d709:::181:head 2024-08-19T09:57:10.881 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:1e5d39db:::182:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:f7df4fb9:::183:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:03a7f161:::184:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ba70721e:::185:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:28e5662d:::186:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:973d52de:::187:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:4303eb1c:::188:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:b990b48e:::189:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:29b8165b:::190:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:3547f197:::191:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:7e260936:::192:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:1abec7b1:::193:head 2024-08-19T09:57:10.882 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:10fdda93:::194:head 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:15817eea:::195:head 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:770bab57:::196:head 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:ed9e13e7:::197:head 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:71471a8f:::198:head 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: checking for 269:10fb1d02:::199:head 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8044 ms) 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,0 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: first is 1724061370 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,0 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,0 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,0 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,0 2024-08-19T09:57:10.883 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,1724061375,1724061376,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,1724061375,1724061376,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,1724061375,1724061376,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,1724061375,1724061376,1724061378,1724061379,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,1724061375,1724061376,1724061378,1724061379,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061370,1724061372,1724061373,1724061375,1724061376,1724061378,1724061379,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724061373,1724061375,1724061376,1724061378,1724061379,1724061381,1724061382,0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: first now 1724061373, trimmed 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20954 ms) 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: foo0 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-08-19T09:57:10.884 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14040 ms) 2024-08-19T09:57:10.885 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099870516}) 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568624541}) 2024-08-19T09:57:10.886 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1851981892}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.884+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1438695211}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4194954172}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=541941036}) 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.888 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2862284615}) 2024-08-19T09:57:10.888 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.888 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798704322}) 2024-08-19T09:57:10.888 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.885+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=379093725}) 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522184569}) 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093294770}) 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1268029090}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3312229416}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=893054523}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3834959504}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=778704171}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841862954}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.892+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3552257634}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.893+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.893+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.893+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227397199}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793607450}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2685209690}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414709031}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=996775602}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3966580411}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1859880676}) 2024-08-19T09:57:10.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252262988}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.894+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2770883345}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483409280}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246144066}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3497091215}) 2024-08-19T09:57:10.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3696896217}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4234315662}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2817928414}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=210517229}) 2024-08-19T09:57:10.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824707532}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.895+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3225663770}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4056375322}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965522943}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=399790620}) 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.896+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4277878021}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3945924717}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2621674262}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3973517485}) 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038632859}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.897+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1262645771}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2348626748}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=531605248}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3490873158}) 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3417592599}) 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=45790774}) 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2128582430}) 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907497590}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152661318}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.898+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273482225}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3270476838}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056280682}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.899+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4077557458}) 2024-08-19T09:57:10.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=94947674}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1937086901}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1368178187}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997346627}) 2024-08-19T09:57:10.937 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585651407}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.900+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=770719047}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219664509}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2683569990}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2781090540}) 2024-08-19T09:57:10.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2116667905}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2731620684}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3682397814}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1990994758}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993405426}) 2024-08-19T09:57:10.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.901+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3129980239}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96748965}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1711969829}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=101300829}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2567483328}) 2024-08-19T09:57:10.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3340723713}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=301570629}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1027970979}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687357834}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2462666412}) 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.903+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.903+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.903+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3242635130}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.903+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399918859}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2360660292}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2257774917}) 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2228247749}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=555448489}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4282690587}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735425393}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.904+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1217254158}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182531713}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2920487758}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317412502}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1272000020}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1487090213}) 2024-08-19T09:57:10.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=668853801}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2934507839}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434960}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.905+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2331872252}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605912615}) 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810819306}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=328615976}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3304171081}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=637486234}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2631269899}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3020843524}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183332929}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=695068297}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983469142}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2427623238}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2783725680}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=772368341}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2277968193}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2353565507}) 2024-08-19T09:57:10.948 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1649175452}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022313518}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.908+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3829876183}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=183538949}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166122658}) 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.949 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2809305321}) 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1152543943}) 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1030979051}) 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.950 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3495290325}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3508648859}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399440931}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.909+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2259301403}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927996944}) 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610060027}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1238744998}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1367552336}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2098189381}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1979310787}) 2024-08-19T09:57:10.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819871624}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774544805}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2649331245}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586244085}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428869641}) 2024-08-19T09:57:10.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1979901973}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2434209883}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1248054295}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1437071157}) 2024-08-19T09:57:10.954 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1870914539}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174790208}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.912+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806955336}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=522250113}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869067954}) 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.955 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2922301036}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542494427}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2225523019}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1368901544}) 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4173691008}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355352697}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.913+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=254867577}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3524039271}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511225376}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2415320214}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1162935548}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2530618879}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2172071008}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4052406545}) 2024-08-19T09:57:10.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868389456}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.914+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=974735329}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231488809}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348606184}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132771205}) 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.959 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=939029541}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1941551728}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2717247571}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3876095547}) 2024-08-19T09:57:10.960 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519033384}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.915+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3323135256}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3120696283}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579025323}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3213119210}) 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2438326735}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3856365729}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3419705766}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1154598324}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385334985}) 2024-08-19T09:57:10.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:10.917+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=881760220}) 2024-08-19T09:57:11.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285738084}) 2024-08-19T09:57:11.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291041181}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1296092382}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1271379842}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1761462421}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2998067771}) 2024-08-19T09:57:11.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4120559366}) 2024-08-19T09:57:11.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098619156}) 2024-08-19T09:57:11.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.218+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=562415218}) 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=698464335}) 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140488535}) 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2736465505}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1838220128}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=483477255}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.428+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=274530438}) 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.429+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.429+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.429+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2360206003}) 2024-08-19T09:57:11.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.429+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.429+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236609502}) 2024-08-19T09:57:11.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.429+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3424593296}) 2024-08-19T09:57:11.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3003451006}) 2024-08-19T09:57:11.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656471070}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2175679081}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2866842252}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3858199734}) 2024-08-19T09:57:11.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2886598655}) 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=934715869}) 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457371722}) 2024-08-19T09:57:11.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.802+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=262020700}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127280968}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=947188330}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4187580921}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3986322065}) 2024-08-19T09:57:11.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2730639188}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3984063122}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2523158776}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598575714}) 2024-08-19T09:57:11.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.804+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1969753779}) 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707233633}) 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841066712}) 2024-08-19T09:57:11.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2985301107}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=874682266}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.862+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1307213597}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2587154039}) 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4249530582}) 2024-08-19T09:57:11.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660644449}) 2024-08-19T09:57:11.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.863+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1203384030}) 2024-08-19T09:57:11.877 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:57:11.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580322266}) 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191810191}) 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3756741695}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=562482212}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2570924882}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3050511382}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.893 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=306579985}) 2024-08-19T09:57:11.894 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.894 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356845803}) 2024-08-19T09:57:11.894 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.890+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3270163922}) 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345608928}) 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1770088982}) 2024-08-19T09:57:11.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=422756839}) 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2031654632}) 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2968864066}) 2024-08-19T09:57:11.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1689337409}) 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3272606250}) 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745890371}) 2024-08-19T09:57:11.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.902+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3455628265}) 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3616065231}) 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2876060842}) 2024-08-19T09:57:11.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2655302110}) 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2995292518}) 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=559033251}) 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=122258766}) 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=872120074}) 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130916926}) 2024-08-19T09:57:11.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:11.918+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3963598810}) 2024-08-19T09:57:12.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.004+0000 7f0dee741640 10 monclient: get_monmap_and_config 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.004+0000 7f0dee741640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.004+0000 7f0dee741640 10 monclient: monmap: 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:12.005745+0000 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:12.005745+0000 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.007 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.005+0000 7f0dee741640 1 Processor -- start 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.005+0000 7f0dee741640 1 -- start start 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.005+0000 7f0dee741640 10 monclient: init 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient: _add_conns ranks=[2,6,5,8,1,3,7,0,4] 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient(hunting): picked mon.noname-b con 0x7f0de80a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient(hunting): picked mon.noname-d con 0x7f0de80a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient(hunting): picked mon.noname-h con 0x7f0de813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:12.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de80a3b40 con 0x7f0de80a67c0 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de80a8130 con 0x7f0de813f5c0 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de806ede0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0dee741640 10 monclient(hunting): authenticate will time out at 1088.082273s 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0ddffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f0de80a6ba0 0x7f0de80a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36912/0 (socket says 172.21.15.1:36912) 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0de7fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0de80a67c0 0x7f0de80a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48040/0 (socket says 172.21.15.1:48040) 2024-08-19T09:57:12.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.006+0000 7f0ddffff640 1 -- 172.21.15.1:0/3806592358 learned_addr learned my addr 172.21.15.1:0/3806592358 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0ddffff640 1 -- 172.21.15.1:0/3806592358 >> v1:172.21.15.1:6792/0 conn(0x7f0de80a6ba0 legacy=0x7f0de80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de7fff640 1 -- 172.21.15.1:0/3806592358 >> v1:172.21.15.1:6790/0 conn(0x7f0de80a67c0 legacy=0x7f0de80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0deccde640 1 -- 172.21.15.1:0/3806592358 >> v1:172.21.15.82:6791/0 conn(0x7f0de813f5c0 legacy=0x7f0de81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705174451 0 0) 0x7f0de806ede0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): my global_id is 164220 2024-08-19T09:57:12.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dd4003520 con 0x7f0de80a6ba0 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122141250 0 0) 0x7f0de80a3b40 con 0x7f0de80a67c0 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 10 monclient(hunting): my global_id is 164198 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.008+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0de806ede0 con 0x7f0de80a67c0 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4057116533 0 0) 0x7f0de80a8130 con 0x7f0de813f5c0 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient(hunting): my global_id is 26960 2024-08-19T09:57:12.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0de80a3b40 con 0x7f0de813f5c0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3051151827 0 0) 0x7f0dd4003520 con 0x7f0de80a6ba0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 >> v1:172.21.15.82:6791/0 conn(0x7f0de813f5c0 legacy=0x7f0de81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 >> v1:172.21.15.1:6790/0 conn(0x7f0de80a67c0 legacy=0x7f0de80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 monclient: found mon.noname-d 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de80a92a0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dd0001ad0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 5 monclient: authenticate success, global_id 164220 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: dump: 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0dd00048a0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 1 -- 172.21.15.1:0/3806592358 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dd0004dd0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 4 set_mon_vals no callback set 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0de77fe640 10 monclient: dump: 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 10 monclient: shutdown 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 1 -- 172.21.15.1:0/3806592358 >> v1:172.21.15.1:6792/0 conn(0x7f0de80a6ba0 legacy=0x7f0de80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 1 -- 172.21.15.1:0/3806592358 shutdown_connections 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 1 -- 172.21.15.1:0/3806592358 >> 172.21.15.1:0/3806592358 conn(0x7f0de80ac4b0 msgr2=0x7f0de80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 1 -- 172.21.15.1:0/3806592358 shutdown_connections 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.009+0000 7f0dee741640 1 -- 172.21.15.1:0/3806592358 wait complete. 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 10 monclient: monmap: 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.026 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 1 librados: starting msgr at 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 1 librados: starting objecter 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 1 Processor -- start 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 1 -- start start 2024-08-19T09:57:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 1 librados: setting wanted keys 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 1 librados: calling monclient init 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.010+0000 7f0dee741640 10 monclient: init 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient: _add_conns ranks=[4,5,2,8,1,7,0,6,3] 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient(hunting): picked mon.noname-c con 0x7f0de80a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient(hunting): picked mon.noname-h con 0x7f0de80a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient(hunting): picked mon.noname-b con 0x7f0de813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de80a28a0 con 0x7f0de813f5c0 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de80a21a0 con 0x7f0de80a67c0 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de8142bd0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0dee741640 10 monclient(hunting): authenticate will time out at 1088.087344s 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0de7fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f0de80a67c0 0x7f0de809ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37450/0 (socket says 172.21.15.1:37450) 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.011+0000 7f0de7fff640 1 -- 172.21.15.1:0/234762873 learned_addr learned my addr 172.21.15.1:0/234762873 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0deccde640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.1:6790/0 conn(0x7f0de813f5c0 legacy=0x7f0de81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 client.?.objecter ms_handle_connect 0x7f0de813f5c0 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de7fff640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.1:6791/0 conn(0x7f0de80a67c0 legacy=0x7f0de809ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 client.?.objecter ms_handle_connect 0x7f0de80a67c0 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 client.?.objecter ms_handle_connect 0x7f0de80a6ba0 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0ddffff640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.82:6791/0 conn(0x7f0de80a6ba0 legacy=0x7f0de80a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688336597 0 0) 0x7f0de80a28a0 con 0x7f0de813f5c0 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): my global_id is 164207 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dc8004eb0 con 0x7f0de813f5c0 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2483420613 0 0) 0x7f0dd8001500 con 0x7f0de80a67c0 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): my global_id is 27238 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0de80a28a0 con 0x7f0de80a67c0 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421409392 0 0) 0x7f0de8142bd0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 10 monclient(hunting): my global_id is 26969 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dd8001500 con 0x7f0de80a6ba0 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.013+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3639957646 0 0) 0x7f0dc8004eb0 con 0x7f0de813f5c0 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0de8142bd0 con 0x7f0de813f5c0 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0de00020b0 con 0x7f0de813f5c0 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): dump: 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.031 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2612097285 0 0) 0x7f0de80a28a0 con 0x7f0de80a67c0 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dc8004eb0 con 0x7f0de80a67c0 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dd8004b20 con 0x7f0de80a67c0 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): dump: 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 608396705 0 0) 0x7f0dd8001500 con 0x7f0de80a6ba0 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0de80a28a0 con 0x7f0de80a6ba0 2024-08-19T09:57:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0dd0004d90 con 0x7f0de80a6ba0 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient(hunting): dump: 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3739396862 0 0) 0x7f0de8142bd0 con 0x7f0de813f5c0 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.82:6791/0 conn(0x7f0de80a6ba0 legacy=0x7f0de80a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.1:6791/0 conn(0x7f0de80a67c0 legacy=0x7f0de809ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 monclient: found mon.b 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de80a1640 con 0x7f0de813f5c0 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 5 monclient: authenticate success, global_id 164207 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 10 monclient: _renew_subs 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0de813a840 con 0x7f0de813f5c0 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 10 client.164207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 10 monclient: _renew_subs 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0de8139b50 con 0x7f0de813f5c0 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0dee741640 1 librados: init done 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0de0003e00 con 0x7f0de813f5c0 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0de00042f0 con 0x7f0de813f5c0 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de6ffd640 4 set_mon_vals no callback set 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.014+0000 7f0de57fa640 10 monclient: dump: 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.039 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0de0004580 con 0x7f0de813f5c0 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0dee741640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0dee741640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0dee741640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dcc005080 con 0x7f0de813f5c0 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(297..297 src has 1..297) ==== 5854+0+0 (unknown 1085451124 0 0) 0x7f0de004e700 con 0x7f0de813f5c0 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de57fa640 10 client.164207.objecter ms_dispatch 0x7f0de8000b90 osd_map(297..297 src has 1..297) 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de57fa640 3 client.164207.objecter handle_osd_map got epochs [297,297] > 0 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de57fa640 3 client.164207.objecter handle_osd_map decoding full epoch 297 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de57fa640 20 client.164207.objecter dump_active .. 0 homeless 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.015+0000 7f0de7fff640 10 monclient: get_auth_request con 0x7f0dc8041ae0 auth_method 0 2024-08-19T09:57:12.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.017+0000 7f0de57fa640 10 client.164207.objecter ms_handle_connect 0x7f0dc8041ae0 2024-08-19T09:57:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.019+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f0de0017910 con 0x7f0de813f5c0 2024-08-19T09:57:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.019+0000 7f0de57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.019+0000 7f0de57fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:12.112 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196866992}) 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835346112}) 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1765260054}) 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.113 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4197976193}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3392234735}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2300019006}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.114 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1890432136}) 2024-08-19T09:57:12.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574501521}) 2024-08-19T09:57:12.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.110+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3280630758}) 2024-08-19T09:57:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0dee741640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:57:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0dee741640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:57:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0dee741640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0dcc005370 con 0x7f0de813f5c0 2024-08-19T09:57:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0de57fa640 1 -- 172.21.15.1:0/234762873 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 3303099812) 0x7f0de0017210 con 0x7f0de813f5c0 2024-08-19T09:57:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0de57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:57:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.275+0000 7f0de57fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:12.277 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":66,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 10 librados: watch_flush enter 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 10 librados: watch_flush exit 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 20 client.164207.objecter shutdown clearing up homeless session... 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 10 client.164207.objecter successfully canceled tick 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0dc8041ae0 legacy=0x7f0dc8043fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 10 monclient: shutdown 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 >> v1:172.21.15.1:6790/0 conn(0x7f0de813f5c0 legacy=0x7f0de81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 shutdown_connections 2024-08-19T09:57:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 >> 172.21.15.1:0/234762873 conn(0x7f0de80ac4b0 msgr2=0x7f0de80a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 shutdown_connections 2024-08-19T09:57:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.277+0000 7f0dee741640 1 -- 172.21.15.1:0/234762873 wait complete. 2024-08-19T09:57:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.278+0000 7f0dee741640 1 librados: shutdown 2024-08-19T09:57:12.289 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:57:12.289 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T09:57:12.289 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:57:12.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:57:12.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96799055}) 2024-08-19T09:57:12.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909761336}) 2024-08-19T09:57:12.403 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.403 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.403 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3696138028}) 2024-08-19T09:57:12.403 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.403 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.403 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1107942821}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1067196216}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2812845566}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.390+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.391+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.391+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3575275841}) 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.391+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.404 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.391+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801057133}) 2024-08-19T09:57:12.405 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.391+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=610287510}) 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713677720}) 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99613149}) 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.406 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1710429522}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1779796947}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1836257360}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.407 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1533622778}) 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3778985763}) 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582792484}) 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.392+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3477997533}) 2024-08-19T09:57:12.408 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.409 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.409 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279008060}) 2024-08-19T09:57:12.409 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.409 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546484438}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3858749581}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2345108359}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.410 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1503651246}) 2024-08-19T09:57:12.411 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.411 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.411 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3912656158}) 2024-08-19T09:57:12.411 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.411 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3680802227}) 2024-08-19T09:57:12.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701165873}) 2024-08-19T09:57:12.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.393+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3306560800}) 2024-08-19T09:57:12.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059204569}) 2024-08-19T09:57:12.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1147833443}) 2024-08-19T09:57:12.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1094604226}) 2024-08-19T09:57:12.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1637940633}) 2024-08-19T09:57:12.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2063294434}) 2024-08-19T09:57:12.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=256367941}) 2024-08-19T09:57:12.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.394+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3019736950}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.395+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.395+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127859532}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.395+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4150762907}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407400598}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867334908}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4136961139}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1686157000}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=548726190}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4006019776}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3470026157}) 2024-08-19T09:57:12.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830018208}) 2024-08-19T09:57:12.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.396+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=204235617}) 2024-08-19T09:57:12.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3609446223}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1321484896}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2844379091}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1046169006}) 2024-08-19T09:57:12.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2162049208}) 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4273677880}) 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604134434}) 2024-08-19T09:57:12.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562101997}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.405+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2462772692}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755711434}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=392712350}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3279681373}) 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700317051}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.406+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4198115523}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1836396786}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3474202426}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626951342}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.407+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3520314009}) 2024-08-19T09:57:12.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616208172}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016253342}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1354767307}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=314631580}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2364566646}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1014379571}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=654074551}) 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.439 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797072065}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.408+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3670935313}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=375883728}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475010711}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=169016510}) 2024-08-19T09:57:12.440 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=464835874}) 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4227459145}) 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.441 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3853329043}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2497696520}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909866247}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.409+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2188630041}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2109179517}) 2024-08-19T09:57:12.442 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1028794609}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=886489970}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=400983125}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.443 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1628011142}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3763146100}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3244593297}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764073921}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.410+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=215914149}) 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.411+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.444 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.411+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.411+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626030908}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.411+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.411+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2858113373}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1330321546}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2585827658}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.445 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=193595652}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2352577043}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1926165148}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058050233}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.412+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2155068345}) 2024-08-19T09:57:12.446 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269217112}) 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2155504099}) 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1587795352}) 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.447 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.448 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3246927083}) 2024-08-19T09:57:12.448 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.448 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.448 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4096170651}) 2024-08-19T09:57:12.448 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.448 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.449 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2652054548}) 2024-08-19T09:57:12.449 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.449 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.449 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=458981920}) 2024-08-19T09:57:12.450 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.450 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333652163}) 2024-08-19T09:57:12.450 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.413+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1800170489}) 2024-08-19T09:57:12.450 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.450 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.451 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2846824542}) 2024-08-19T09:57:12.451 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.451 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.451 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713838036}) 2024-08-19T09:57:12.451 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.451 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3898963799}) 2024-08-19T09:57:12.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4253846910}) 2024-08-19T09:57:12.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4287198589}) 2024-08-19T09:57:12.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=727320838}) 2024-08-19T09:57:12.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1379894663}) 2024-08-19T09:57:12.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002690002}) 2024-08-19T09:57:12.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.414+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3134023021}) 2024-08-19T09:57:12.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232175756}) 2024-08-19T09:57:12.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2219986103}) 2024-08-19T09:57:12.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=670078914}) 2024-08-19T09:57:12.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=342018281}) 2024-08-19T09:57:12.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1296240056}) 2024-08-19T09:57:12.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2938031871}) 2024-08-19T09:57:12.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1592754709}) 2024-08-19T09:57:12.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21385990}) 2024-08-19T09:57:12.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.415+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2764093326}) 2024-08-19T09:57:12.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.416+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.416+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.416+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2322211268}) 2024-08-19T09:57:12.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.416+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.416+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538107928}) 2024-08-19T09:57:12.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3924436856}) 2024-08-19T09:57:12.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3367504586}) 2024-08-19T09:57:12.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2978835979}) 2024-08-19T09:57:12.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2452182814}) 2024-08-19T09:57:12.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=12782131}) 2024-08-19T09:57:12.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577438802}) 2024-08-19T09:57:12.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.417+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1755495510}) 2024-08-19T09:57:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.434+0000 7f2a054b1640 10 monclient: get_monmap_and_config 2024-08-19T09:57:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.434+0000 7f2a054b1640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.434+0000 7f2a054b1640 10 monclient: monmap: 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:12.435316+0000 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:12.435316+0000 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.434+0000 7f2a054b1640 1 Processor -- start 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 1 -- start start 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient: init 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient: _add_conns ranks=[5,3,7,4,2,8,1,6,0] 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient(hunting): picked mon.noname-h con 0x7f2a000a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient(hunting): picked mon.noname-g con 0x7f2a000a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient(hunting): picked mon.noname-i con 0x7f2a0013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a000a3b40 con 0x7f2a000a6ba0 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a000a8130 con 0x7f2a000a67c0 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0006ede0 con 0x7f2a0013f5c0 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f2a054b1640 10 monclient(hunting): authenticate will time out at 1088.511516s 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f29fe7fc640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f2a000a6ba0 0x7f2a000a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:37236/0 (socket says 172.21.15.1:37236) 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f29ff7fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f2a0013f5c0 0x7f2a001419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:35886/0 (socket says 172.21.15.1:35886) 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.435+0000 7f29fe7fc640 1 -- 172.21.15.1:0/922586310 learned_addr learned my addr 172.21.15.1:0/922586310 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29feffd640 1 -- 172.21.15.1:0/922586310 >> v1:172.21.15.82:6791/0 conn(0x7f2a000a67c0 legacy=0x7f2a000a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fe7fc640 1 -- 172.21.15.1:0/922586310 >> v1:172.21.15.82:6790/0 conn(0x7f2a000a6ba0 legacy=0x7f2a000a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29ff7fe640 1 -- 172.21.15.1:0/922586310 >> v1:172.21.15.82:6792/0 conn(0x7f2a0013f5c0 legacy=0x7f2a001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626275837 0 0) 0x7f2a000a8130 con 0x7f2a000a67c0 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): my global_id is 26978 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e8003520 con 0x7f2a000a67c0 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572446240 0 0) 0x7f2a000a3b40 con 0x7f2a000a6ba0 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): my global_id is 134958 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a000a8130 con 0x7f2a000a6ba0 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808434005 0 0) 0x7f2a0006ede0 con 0x7f2a0013f5c0 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient(hunting): my global_id is 144484 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a000a3b40 con 0x7f2a0013f5c0 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1697311608 0 0) 0x7f29e8003520 con 0x7f2a000a67c0 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 >> v1:172.21.15.82:6792/0 conn(0x7f2a0013f5c0 legacy=0x7f2a001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 >> v1:172.21.15.82:6790/0 conn(0x7f2a000a6ba0 legacy=0x7f2a000a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 monclient: found mon.noname-h 2024-08-19T09:57:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a000a92a0 con 0x7f2a000a67c0 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f0001eb0 con 0x7f2a000a67c0 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: dump: 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.483 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f80019b0 con 0x7f2a0013f5c0 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f2a054b1640 5 monclient: authenticate success, global_id 26978 2024-08-19T09:57:12.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f2a054b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:12.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:57:12.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29f0003990 con 0x7f2a000a67c0 2024-08-19T09:57:12.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f2a054b1640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f2a054b1640 4 set_mon_vals no callback set 2024-08-19T09:57:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 1 -- 172.21.15.1:0/922586310 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f0003f40 con 0x7f2a000a67c0 2024-08-19T09:57:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.438+0000 7f29fdffb640 10 monclient: dump: 2024-08-19T09:57:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.490 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.490 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 10 monclient: shutdown 2024-08-19T09:57:12.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 -- 172.21.15.1:0/922586310 >> v1:172.21.15.82:6791/0 conn(0x7f2a000a67c0 legacy=0x7f2a000a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 -- 172.21.15.1:0/922586310 shutdown_connections 2024-08-19T09:57:12.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 -- 172.21.15.1:0/922586310 >> 172.21.15.1:0/922586310 conn(0x7f2a000ac4b0 msgr2=0x7f2a000a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 -- 172.21.15.1:0/922586310 shutdown_connections 2024-08-19T09:57:12.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 -- 172.21.15.1:0/922586310 wait complete. 2024-08-19T09:57:12.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 10 monclient: monmap: 2024-08-19T09:57:12.492 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.492 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:12.492 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:12.492 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:12.492 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.492 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.493 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.493 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.493 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.493 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.493 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.494 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 librados: starting msgr at 2024-08-19T09:57:12.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.439+0000 7f2a054b1640 1 librados: starting objecter 2024-08-19T09:57:12.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 Processor -- start 2024-08-19T09:57:12.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 -- start start 2024-08-19T09:57:12.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 librados: setting wanted keys 2024-08-19T09:57:12.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 librados: calling monclient init 2024-08-19T09:57:12.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient: init 2024-08-19T09:57:12.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient: _add_conns ranks=[4,2,1,8,0,7,3,5,6] 2024-08-19T09:57:12.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient(hunting): picked mon.noname-c con 0x7f2a000a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:12.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient(hunting): picked mon.noname-b con 0x7f2a000a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:12.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient(hunting): picked mon.noname-f con 0x7f2a0013f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:12.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a000a9cc0 con 0x7f2a0013f5c0 2024-08-19T09:57:12.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0013ab70 con 0x7f2a000a6ba0 2024-08-19T09:57:12.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0013d250 con 0x7f2a000a67c0 2024-08-19T09:57:12.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f2a054b1640 10 monclient(hunting): authenticate will time out at 1088.516812s 2024-08-19T09:57:12.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f29feffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f2a000a67c0 0x7f2a00139d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37464/0 (socket says 172.21.15.1:37464) 2024-08-19T09:57:12.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.440+0000 7f29feffd640 1 -- 172.21.15.1:0/1785942560 learned_addr learned my addr 172.21.15.1:0/1785942560 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29feffd640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6791/0 conn(0x7f2a000a67c0 legacy=0x7f2a00139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 client.?.objecter ms_handle_connect 0x7f2a000a67c0 2024-08-19T09:57:12.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29fe7fc640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6790/0 conn(0x7f2a000a6ba0 legacy=0x7f2a0013a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 client.?.objecter ms_handle_connect 0x7f2a000a6ba0 2024-08-19T09:57:12.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2133986119 0 0) 0x7f2a0013d250 con 0x7f2a000a67c0 2024-08-19T09:57:12.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): my global_id is 27247 2024-08-19T09:57:12.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29ff7fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.82:6789/0 conn(0x7f2a0013f5c0 legacy=0x7f2a001d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29ec004eb0 con 0x7f2a000a67c0 2024-08-19T09:57:12.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 client.?.objecter ms_handle_connect 0x7f2a0013f5c0 2024-08-19T09:57:12.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3768066251 0 0) 0x7f2a0013ab70 con 0x7f2a000a6ba0 2024-08-19T09:57:12.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): my global_id is 164216 2024-08-19T09:57:12.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a0013d250 con 0x7f2a000a6ba0 2024-08-19T09:57:12.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1003190946 0 0) 0x7f29ec004eb0 con 0x7f2a000a67c0 2024-08-19T09:57:12.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a0013ab70 con 0x7f2a000a67c0 2024-08-19T09:57:12.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f0003e80 con 0x7f2a000a67c0 2024-08-19T09:57:12.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:12.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): dump: 2024-08-19T09:57:12.503 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.504 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.505 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.505 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.505 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:12.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3437530604 0 0) 0x7f2a0013d250 con 0x7f2a000a6ba0 2024-08-19T09:57:12.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29ec004eb0 con 0x7f2a000a6ba0 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f4001d30 con 0x7f2a000a6ba0 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): dump: 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.506 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.507 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302849218 0 0) 0x7f2a000a9cc0 con 0x7f2a0013f5c0 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient(hunting): my global_id is 164170 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a0013d250 con 0x7f2a0013f5c0 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1305095412 0 0) 0x7f2a0013ab70 con 0x7f2a000a67c0 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6790/0 conn(0x7f2a000a6ba0 legacy=0x7f2a0013a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.82:6789/0 conn(0x7f2a0013f5c0 legacy=0x7f2a001d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 monclient: found mon.c 2024-08-19T09:57:12.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0013c590 con 0x7f2a000a67c0 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 5 monclient: authenticate success, global_id 27247 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 10 monclient: _renew_subs 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0013f250 con 0x7f2a000a67c0 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 10 client.27247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:12.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 10 monclient: _renew_subs 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a0013e0f0 con 0x7f2a000a67c0 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f2a054b1640 1 librados: init done 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29f0003990 con 0x7f2a000a67c0 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f00047f0 con 0x7f2a000a67c0 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29fd7fa640 4 set_mon_vals no callback set 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.443+0000 7f29e77fe640 10 monclient: dump: 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.510 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.511 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.444+0000 7f2a054b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.444+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f29e00065c0 con 0x7f29e0002860 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.444+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f29f0011e00 con 0x7f2a000a67c0 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.445+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(297..297 src has 1..297) ==== 5854+0+0 (unknown 1085451124 0 0) 0x7f29f004e8c0 con 0x7f2a000a67c0 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.445+0000 7f29e77fe640 10 client.27247.objecter ms_dispatch 0x7f2a00000b90 osd_map(297..297 src has 1..297) 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.445+0000 7f29e77fe640 3 client.27247.objecter handle_osd_map got epochs [297,297] > 0 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.445+0000 7f29e77fe640 3 client.27247.objecter handle_osd_map decoding full epoch 297 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.445+0000 7f29e77fe640 20 client.27247.objecter dump_active .. 0 homeless 2024-08-19T09:57:12.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.445+0000 7f29ff7fe640 10 monclient: get_auth_request con 0x7f29ec041690 auth_method 0 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.446+0000 7f29fe7fc640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6789/0 conn(0x7f29e0002860 legacy=0x7f29e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.446+0000 7f29e77fe640 10 client.27247.objecter ms_handle_connect 0x7f29e0002860 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.446+0000 7f29e77fe640 10 client.27247.objecter resend_mon_ops 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.446+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592745580 0 0) 0x7f29e00065c0 con 0x7f29e0002860 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient(hunting): my global_id is 164178 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a0013ab70 con 0x7f29e0002860 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2518554124 0 0) 0x7f2a0013ab70 con 0x7f29e0002860 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29e00065c0 con 0x7f29e0002860 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f4004da0 con 0x7f29e0002860 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:12.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient: dump: 2024-08-19T09:57:12.514 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.514 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.514 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.514 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.514 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.515 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3512889798 0 0) 0x7f29e00065c0 con 0x7f29e0002860 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29e000a160 con 0x7f29e0002860 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 client.27247.objecter ms_handle_connect 0x7f29ec041690 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f29f40070e0 con 0x7f29e0002860 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6789/0 conn(0x7f29e0002860 legacy=0x7f29e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.447+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6789/0 conn(0x7f29e0002860 legacy=0x7f29e0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:12.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.453+0000 7f2a054b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.453+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f29e0009e70 con 0x7f29e0009130 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 client.27247.objecter ms_handle_connect 0x7f29e0009130 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 client.27247.objecter resend_mon_ops 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29fe7fc640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6789/0 conn(0x7f29e0009130 legacy=0x7f29e00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934145139 0 0) 0x7f29e0009e70 con 0x7f29e0009130 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient(hunting): my global_id is 164187 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e00065c0 con 0x7f29e0009130 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1383168413 0 0) 0x7f29e00065c0 con 0x7f29e0009130 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29e0009e70 con 0x7f29e0009130 2024-08-19T09:57:12.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29f4005780 con 0x7f29e0009130 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient: dump: 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.518 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2575819189 0 0) 0x7f29e0009e70 con 0x7f29e0009130 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29e000c410 con 0x7f29e0009130 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1101608897) 0x7f29f4006d30 con 0x7f29e0009130 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6789/0 conn(0x7f29e0009130 legacy=0x7f29e00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.456+0000 7f29e77fe640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6789/0 conn(0x7f29e0009130 legacy=0x7f29e00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19274, 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:12.520 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:12.521 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.522 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:12.524 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.526 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.526 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.526 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.526 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.526 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:12.530 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.533 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.534 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.537 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.538 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.543 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:12.543 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 10 librados: watch_flush enter 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 10 librados: watch_flush exit 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 20 client.27247.objecter shutdown clearing up homeless session... 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 10 client.27247.objecter successfully canceled tick 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.82:6812/2549104415 conn(0x7f29ec041690 legacy=0x7f29ec043b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 10 monclient: shutdown 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 >> v1:172.21.15.1:6791/0 conn(0x7f2a000a67c0 legacy=0x7f2a00139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f29ff7fe640 1 -- 172.21.15.1:0/1785942560 reap_dead start 2024-08-19T09:57:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f29ff7fe640 1 -- 172.21.15.1:0/1785942560 reap_dead start 2024-08-19T09:57:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 shutdown_connections 2024-08-19T09:57:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 >> 172.21.15.1:0/1785942560 conn(0x7f2a000ac4b0 msgr2=0x7f2a000ab050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 shutdown_connections 2024-08-19T09:57:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 -- 172.21.15.1:0/1785942560 wait complete. 2024-08-19T09:57:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.457+0000 7f2a054b1640 1 librados: shutdown 2024-08-19T09:57:12.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:12.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:57:12.547 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578931259}) 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618396465}) 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1830766838}) 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.549 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=152168269}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=110846114}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3329867554}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2294382598}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860876830}) 2024-08-19T09:57:12.550 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.514+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1340929997}) 2024-08-19T09:57:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.648+0000 7f608b4ec640 10 monclient: get_monmap_and_config 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.648+0000 7f608b4ec640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.648+0000 7f608b4ec640 10 monclient: monmap: 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:12.649826+0000 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:12.649826+0000 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.651 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.649+0000 7f608b4ec640 1 Processor -- start 2024-08-19T09:57:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.649+0000 7f608b4ec640 1 -- start start 2024-08-19T09:57:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.649+0000 7f608b4ec640 10 monclient: init 2024-08-19T09:57:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient: _add_conns ranks=[1,6,2,7,3,0,5,4,8] 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient(hunting): picked mon.noname-f con 0x7f60840a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient(hunting): picked mon.noname-d con 0x7f60840a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient(hunting): picked mon.noname-b con 0x7f608413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60840a3b40 con 0x7f60840a67c0 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60840a8130 con 0x7f608413f5c0 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608406ede0 con 0x7f60840a6ba0 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f608b4ec640 10 monclient(hunting): authenticate will time out at 1088.726288s 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f6088a87640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f60840a6ba0 0x7f60840a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36922/0 (socket says 172.21.15.1:36922) 2024-08-19T09:57:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.650+0000 7f6088a87640 1 -- 172.21.15.1:0/18904362 learned_addr learned my addr 172.21.15.1:0/18904362 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6088a87640 1 -- 172.21.15.1:0/18904362 >> v1:172.21.15.1:6792/0 conn(0x7f60840a6ba0 legacy=0x7f60840a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6089a89640 1 -- 172.21.15.1:0/18904362 >> v1:172.21.15.1:6790/0 conn(0x7f608413f5c0 legacy=0x7f60841419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1796706035 0 0) 0x7f608406ede0 con 0x7f60840a6ba0 2024-08-19T09:57:12.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6089288640 1 -- 172.21.15.1:0/18904362 >> v1:172.21.15.82:6789/0 conn(0x7f60840a67c0 legacy=0x7f60840a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): my global_id is 164229 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f606c003520 con 0x7f60840a6ba0 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198954254 0 0) 0x7f60840a8130 con 0x7f608413f5c0 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): my global_id is 164225 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f608406ede0 con 0x7f608413f5c0 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266121018 0 0) 0x7f60840a3b40 con 0x7f60840a67c0 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient(hunting): my global_id is 164179 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60840a8130 con 0x7f60840a67c0 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3599348992 0 0) 0x7f608406ede0 con 0x7f608413f5c0 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 >> v1:172.21.15.1:6792/0 conn(0x7f60840a6ba0 legacy=0x7f60840a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 >> v1:172.21.15.82:6789/0 conn(0x7f60840a67c0 legacy=0x7f60840a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 monclient: found mon.noname-b 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60840a92a0 con 0x7f608413f5c0 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6074001e10 con 0x7f608413f5c0 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.652+0000 7f6083fff640 10 monclient: dump: 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.664 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 5 monclient: authenticate success, global_id 164225 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6074003990 con 0x7f608413f5c0 2024-08-19T09:57:12.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f6083fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f6083fff640 1 -- 172.21.15.1:0/18904362 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6074003e80 con 0x7f608413f5c0 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 4 set_mon_vals no callback set 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f6083fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f6083fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f6083fff640 10 monclient: dump: 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.666 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 10 monclient: shutdown 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 1 -- 172.21.15.1:0/18904362 >> v1:172.21.15.1:6790/0 conn(0x7f608413f5c0 legacy=0x7f60841419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 1 -- 172.21.15.1:0/18904362 shutdown_connections 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 1 -- 172.21.15.1:0/18904362 >> 172.21.15.1:0/18904362 conn(0x7f60840ac4b0 msgr2=0x7f60840a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 1 -- 172.21.15.1:0/18904362 shutdown_connections 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.653+0000 7f608b4ec640 1 -- 172.21.15.1:0/18904362 wait complete. 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 10 monclient: monmap: 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.668 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 1 librados: starting msgr at 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 1 librados: starting objecter 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 1 Processor -- start 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 1 -- start start 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 1 librados: setting wanted keys 2024-08-19T09:57:12.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 1 librados: calling monclient init 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.654+0000 7f608b4ec640 10 monclient: init 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient: _add_conns ranks=[5,2,6,3,7,1,0,4,8] 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient(hunting): picked mon.noname-h con 0x7f60840a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient(hunting): picked mon.noname-b con 0x7f60840a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient(hunting): picked mon.noname-d con 0x7f608413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60840a2890 con 0x7f60840a6ba0 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6084142bf0 con 0x7f60840a67c0 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60840a15c0 con 0x7f608413f5c0 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f608b4ec640 10 monclient(hunting): authenticate will time out at 1088.731096s 2024-08-19T09:57:12.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f6089a89640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f608413f5c0 0x7f60841d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36928/0 (socket says 172.21.15.1:36928) 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f6089a89640 1 -- 172.21.15.1:0/1700389380 learned_addr learned my addr 172.21.15.1:0/1700389380 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.655+0000 7f6088a87640 1 --1- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f60840a6ba0 0x7f60840a05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48070/0 (socket says 172.21.15.1:48070) 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6089a89640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6792/0 conn(0x7f608413f5c0 legacy=0x7f60841d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6088a87640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f60840a6ba0 legacy=0x7f60840a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 client.?.objecter ms_handle_connect 0x7f608413f5c0 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 client.?.objecter ms_handle_connect 0x7f60840a6ba0 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 client.?.objecter ms_handle_connect 0x7f60840a67c0 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6089288640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.82:6791/0 conn(0x7f60840a67c0 legacy=0x7f608409fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2857462199 0 0) 0x7f60840a15c0 con 0x7f608413f5c0 2024-08-19T09:57:12.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): my global_id is 164238 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6070004ee0 con 0x7f608413f5c0 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 600483856 0 0) 0x7f60840a2890 con 0x7f60840a6ba0 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): my global_id is 164234 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60840a15c0 con 0x7f60840a6ba0 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212845164 0 0) 0x7f6084142bf0 con 0x7f60840a67c0 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): my global_id is 26987 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60840a2890 con 0x7f60840a67c0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3527989488 0 0) 0x7f60840a15c0 con 0x7f60840a6ba0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6084142bf0 con 0x7f60840a6ba0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 664856663 0 0) 0x7f6070004ee0 con 0x7f608413f5c0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60840a15c0 con 0x7f608413f5c0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60780019d0 con 0x7f60840a6ba0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): dump: 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.673 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.674 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6074003e80 con 0x7f608413f5c0 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): dump: 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.675 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 375617965 0 0) 0x7f60840a2890 con 0x7f60840a67c0 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6070004ee0 con 0x7f60840a67c0 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f607c003610 con 0x7f60840a67c0 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 10 monclient(hunting): dump: 2024-08-19T09:57:12.676 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.677 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.657+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2627663080 0 0) 0x7f6084142bf0 con 0x7f60840a6ba0 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6792/0 conn(0x7f608413f5c0 legacy=0x7f60841d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.82:6791/0 conn(0x7f60840a67c0 legacy=0x7f608409fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 monclient: found mon.b 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608413a960 con 0x7f60840a6ba0 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 5 monclient: authenticate success, global_id 164234 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 10 monclient: _renew_subs 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6084139ce0 con 0x7f60840a6ba0 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6078004dd0 con 0x7f60840a6ba0 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60780052c0 con 0x7f60840a6ba0 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 10 monclient: dump: 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.679 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 10 client.164234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 10 monclient: _renew_subs 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f60840a0cb0 con 0x7f60840a6ba0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f60837fe640 4 set_mon_vals no callback set 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 1 librados: init done 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6078011720 con 0x7f60840a6ba0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.658+0000 7f608b4ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f60840a0f90 con 0x7f608413feb0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(297..297 src has 1..297) ==== 5854+0+0 (unknown 1085451124 0 0) 0x7f607804e250 con 0x7f60840a6ba0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f6081ffb640 10 client.164234.objecter ms_dispatch 0x7f6084000b90 osd_map(297..297 src has 1..297) 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f6081ffb640 3 client.164234.objecter handle_osd_map got epochs [297,297] > 0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f6081ffb640 3 client.164234.objecter handle_osd_map decoding full epoch 297 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f6089a89640 10 monclient: get_auth_request con 0x7f60700418c0 auth_method 0 2024-08-19T09:57:12.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.659+0000 7f6081ffb640 20 client.164234.objecter dump_active .. 0 homeless 2024-08-19T09:57:12.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 client.164234.objecter ms_handle_connect 0x7f608413feb0 2024-08-19T09:57:12.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6089288640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f608413feb0 legacy=0x7f6084140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 client.164234.objecter resend_mon_ops 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1402495457 0 0) 0x7f60840a0f90 con 0x7f608413feb0 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient(hunting): my global_id is 164243 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6084142bf0 con 0x7f608413feb0 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2942857031 0 0) 0x7f6084142bf0 con 0x7f608413feb0 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60840a0f90 con 0x7f608413feb0 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f607c003e90 con 0x7f608413feb0 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 monclient: dump: 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.686 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 10 client.164234.objecter ms_handle_connect 0x7f60700418c0 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3761356903 0 0) 0x7f60840a0f90 con 0x7f608413feb0 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.661+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f608413a300 con 0x7f608413feb0 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.662+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f607c004140 con 0x7f608413feb0 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.662+0000 7f6081ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.662+0000 7f6081ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.662+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f608413feb0 legacy=0x7f6084140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.662+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f608413feb0 legacy=0x7f6084140290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.668+0000 7f608b4ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.668+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6084068f60 con 0x7f60841409a0 2024-08-19T09:57:12.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6089288640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f60841409a0 legacy=0x7f60841db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 client.164234.objecter ms_handle_connect 0x7f60841409a0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 client.164234.objecter resend_mon_ops 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296292855 0 0) 0x7f6084068f60 con 0x7f60841409a0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient(hunting): my global_id is 164252 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60840a0f90 con 0x7f60841409a0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2190412352 0 0) 0x7f60840a0f90 con 0x7f60841409a0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6084068f60 con 0x7f60841409a0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f607c005b00 con 0x7f60841409a0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.670+0000 7f6081ffb640 10 monclient: dump: 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.689 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 154338653 0 0) 0x7f6084068f60 con 0x7f60841409a0 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f608413a5e0 con 0x7f60841409a0 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 731985760) 0x7f607c007d70 con 0x7f60841409a0 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f60841409a0 legacy=0x7f60841db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6081ffb640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f60841409a0 legacy=0x7f60841db990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:12.690 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19486, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:12.712 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:12.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 10 librados: watch_flush enter 2024-08-19T09:57:12.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 10 librados: watch_flush exit 2024-08-19T09:57:12.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 20 client.164234.objecter shutdown clearing up homeless session... 2024-08-19T09:57:12.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 10 client.164234.objecter successfully canceled tick 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.82:6812/2549104415 conn(0x7f60700418c0 legacy=0x7f6070043d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 10 monclient: shutdown 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 >> v1:172.21.15.1:6790/0 conn(0x7f60840a6ba0 legacy=0x7f60840a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6089a89640 1 -- 172.21.15.1:0/1700389380 reap_dead start 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f6089a89640 1 -- 172.21.15.1:0/1700389380 reap_dead start 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 shutdown_connections 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 >> 172.21.15.1:0/1700389380 conn(0x7f60840ac4b0 msgr2=0x7f60840a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 shutdown_connections 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 -- 172.21.15.1:0/1700389380 wait complete. 2024-08-19T09:57:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.671+0000 7f608b4ec640 1 librados: shutdown 2024-08-19T09:57:12.716 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:12.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:57:12.716 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:57:12.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.814+0000 7f3d3a060640 10 monclient: get_monmap_and_config 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.814+0000 7f3d3a060640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.814+0000 7f3d3a060640 10 monclient: monmap: 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:12.815561+0000 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:12.815561+0000 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.816 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.817 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.817 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.817 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.815+0000 7f3d3a060640 1 Processor -- start 2024-08-19T09:57:12.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.815+0000 7f3d3a060640 1 -- start start 2024-08-19T09:57:12.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.815+0000 7f3d3a060640 10 monclient: init 2024-08-19T09:57:12.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.815+0000 7f3d3a060640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.815+0000 7f3d3a060640 10 monclient: _add_conns ranks=[3,8,7,5,1,0,2,4,6] 2024-08-19T09:57:12.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 10 monclient(hunting): picked mon.noname-g con 0x7f3d340a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 10 monclient(hunting): picked mon.noname-e con 0x7f3d340a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 10 monclient(hunting): picked mon.noname-i con 0x7f3d3413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d340a3b40 con 0x7f3d340a67c0 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d340a8130 con 0x7f3d3413f5c0 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3406ede0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d3a060640 10 monclient(hunting): authenticate will time out at 1088.892085s 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d2bfff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f3d340a6ba0 0x7f3d340a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52932/0 (socket says 172.21.15.1:52932) 2024-08-19T09:57:12.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.816+0000 7f3d2bfff640 1 -- 172.21.15.1:0/1453465555 learned_addr learned my addr 172.21.15.1:0/1453465555 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d2bfff640 1 -- 172.21.15.1:0/1453465555 >> v1:172.21.15.1:6793/0 conn(0x7f3d340a6ba0 legacy=0x7f3d340a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d337fe640 1 -- 172.21.15.1:0/1453465555 >> v1:172.21.15.82:6790/0 conn(0x7f3d340a67c0 legacy=0x7f3d340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510433525 0 0) 0x7f3d3406ede0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient(hunting): my global_id is 27782 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d20003520 con 0x7f3d340a6ba0 2024-08-19T09:57:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d33fff640 1 -- 172.21.15.1:0/1453465555 >> v1:172.21.15.82:6792/0 conn(0x7f3d3413f5c0 legacy=0x7f3d341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1924871601 0 0) 0x7f3d20003520 con 0x7f3d340a6ba0 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 >> v1:172.21.15.82:6792/0 conn(0x7f3d3413f5c0 legacy=0x7f3d341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 >> v1:172.21.15.82:6790/0 conn(0x7f3d340a67c0 legacy=0x7f3d340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 monclient: found mon.noname-e 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d340a92a0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d3a060640 5 monclient: authenticate success, global_id 27782 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d1c001ad0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d3a060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: dump: 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.825 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.826 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d3a060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d1c0048a0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 1 -- 172.21.15.1:0/1453465555 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d1c004e10 con 0x7f3d340a6ba0 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d3a060640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d3a060640 4 set_mon_vals no callback set 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.818+0000 7f3d32ffd640 10 monclient: dump: 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.827 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 10 monclient: shutdown 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 1 -- 172.21.15.1:0/1453465555 >> v1:172.21.15.1:6793/0 conn(0x7f3d340a6ba0 legacy=0x7f3d340a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 1 -- 172.21.15.1:0/1453465555 shutdown_connections 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 1 -- 172.21.15.1:0/1453465555 >> 172.21.15.1:0/1453465555 conn(0x7f3d340ac4b0 msgr2=0x7f3d340a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 1 -- 172.21.15.1:0/1453465555 shutdown_connections 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.819+0000 7f3d3a060640 1 -- 172.21.15.1:0/1453465555 wait complete. 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient: monmap: 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:12.829 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 1 librados: starting msgr at 2024-08-19T09:57:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 1 librados: starting objecter 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 1 Processor -- start 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 1 -- start start 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 1 librados: setting wanted keys 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 1 librados: calling monclient init 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient: init 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient: _add_conns ranks=[5,2,0,8,6,7,4,3,1] 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient(hunting): picked mon.noname-h con 0x7f3d340a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.820+0000 7f3d3a060640 10 monclient(hunting): picked mon.noname-b con 0x7f3d340a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d3a060640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3413f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d3a060640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3413a560 con 0x7f3d3413f5c0 2024-08-19T09:57:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d3a060640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3413a840 con 0x7f3d340a6ba0 2024-08-19T09:57:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d3a060640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d34139b50 con 0x7f3d340a67c0 2024-08-19T09:57:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d3a060640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d3a060640 10 monclient(hunting): authenticate will time out at 1088.897068s 2024-08-19T09:57:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d2bfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f3d340a6ba0 0x7f3d340a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48102/0 (socket says 172.21.15.1:48102) 2024-08-19T09:57:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.821+0000 7f3d2bfff640 1 -- 172.21.15.1:0/969644404 learned_addr learned my addr 172.21.15.1:0/969644404 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d33fff640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6789/0 conn(0x7f3d3413f5c0 legacy=0x7f3d340a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 client.?.objecter ms_handle_connect 0x7f3d3413f5c0 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 client.?.objecter ms_handle_connect 0x7f3d340a6ba0 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d2bfff640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6790/0 conn(0x7f3d340a6ba0 legacy=0x7f3d340a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444873920 0 0) 0x7f3d3413a560 con 0x7f3d3413f5c0 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): my global_id is 164196 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d14004eb0 con 0x7f3d3413f5c0 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327958803 0 0) 0x7f3d3413a840 con 0x7f3d340a6ba0 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): my global_id is 164261 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d3413a560 con 0x7f3d340a6ba0 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 client.?.objecter ms_handle_connect 0x7f3d340a67c0 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d337fe640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.82:6791/0 conn(0x7f3d340a67c0 legacy=0x7f3d3409fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3352943337 0 0) 0x7f3d3413a560 con 0x7f3d340a6ba0 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d3413a840 con 0x7f3d340a6ba0 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d1c004d90 con 0x7f3d340a6ba0 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.840 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2206621524 0 0) 0x7f3d14004eb0 con 0x7f3d3413f5c0 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d3413a560 con 0x7f3d3413f5c0 2024-08-19T09:57:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d2c002030 con 0x7f3d3413f5c0 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.842 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1299180171 0 0) 0x7f3d34139b50 con 0x7f3d340a67c0 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient(hunting): my global_id is 26996 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d14004eb0 con 0x7f3d340a67c0 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2909504654 0 0) 0x7f3d3413a840 con 0x7f3d340a6ba0 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.82:6791/0 conn(0x7f3d340a67c0 legacy=0x7f3d3409fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6789/0 conn(0x7f3d3413f5c0 legacy=0x7f3d340a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 monclient: found mon.b 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d34142c60 con 0x7f3d340a6ba0 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient: _finish_auth 0 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d3a060640 5 monclient: authenticate success, global_id 164261 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d3a060640 10 monclient: _renew_subs 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d3a060640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.823+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d340a2850 con 0x7f3d340a6ba0 2024-08-19T09:57:12.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 10 client.164261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 10 monclient: _renew_subs 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d34139fa0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 1 librados: init done 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d1c001d20 con 0x7f3d340a6ba0 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d30ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d1c005740 con 0x7f3d340a6ba0 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d327fc640 4 set_mon_vals no callback set 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d30ff9640 10 monclient: dump: 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.845 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.846 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.824+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d180065c0 con 0x7f3d18002860 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3d1c011ba0 con 0x7f3d340a6ba0 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(297..297 src has 1..297) ==== 5854+0+0 (unknown 1085451124 0 0) 0x7f3d1c04e480 con 0x7f3d340a6ba0 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d30ff9640 10 client.164261.objecter ms_dispatch 0x7f3d34000b90 osd_map(297..297 src has 1..297) 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d30ff9640 3 client.164261.objecter handle_osd_map got epochs [297,297] > 0 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d30ff9640 3 client.164261.objecter handle_osd_map decoding full epoch 297 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d30ff9640 20 client.164261.objecter dump_active .. 0 homeless 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.825+0000 7f3d33fff640 10 monclient: get_auth_request con 0x7f3d140416b0 auth_method 0 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 client.164261.objecter ms_handle_connect 0x7f3d18002860 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 client.164261.objecter resend_mon_ops 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d337fe640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6791/0 conn(0x7f3d18002860 legacy=0x7f3d18004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 881786020 0 0) 0x7f3d180065c0 con 0x7f3d18002860 2024-08-19T09:57:12.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient(hunting): my global_id is 27256 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d3413a840 con 0x7f3d18002860 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3181831133 0 0) 0x7f3d3413a840 con 0x7f3d18002860 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d180065c0 con 0x7f3d18002860 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d24001be0 con 0x7f3d18002860 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient: dump: 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3097566398 0 0) 0x7f3d180065c0 con 0x7f3d18002860 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d1800a160 con 0x7f3d18002860 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 client.164261.objecter ms_handle_connect 0x7f3d140416b0 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3d240057c0 con 0x7f3d18002860 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6791/0 conn(0x7f3d18002860 legacy=0x7f3d18004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.827+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6791/0 conn(0x7f3d18002860 legacy=0x7f3d18004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.833+0000 7f3d3a060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.833+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d18009e50 con 0x7f3d18009110 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d337fe640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6791/0 conn(0x7f3d18009110 legacy=0x7f3d180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 client.164261.objecter ms_handle_connect 0x7f3d18009110 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 client.164261.objecter resend_mon_ops 2024-08-19T09:57:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090993118 0 0) 0x7f3d18009e50 con 0x7f3d18009110 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient(hunting): my global_id is 27265 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d180065c0 con 0x7f3d18009110 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1571113730 0 0) 0x7f3d180065c0 con 0x7f3d18009110 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d18009e50 con 0x7f3d18009110 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d24003d40 con 0x7f3d18009110 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 10 monclient: dump: 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:12.851 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1868175728 0 0) 0x7f3d18009e50 con 0x7f3d18009110 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.836+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d1800c3f0 con 0x7f3d18009110 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3824118383) 0x7f3d24005a30 con 0x7f3d18009110 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d30ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6791/0 conn(0x7f3d18009110 legacy=0x7f3d180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d30ff9640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6791/0 conn(0x7f3d18009110 legacy=0x7f3d180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:12.853 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 439108, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:12.854 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.855 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.856 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:12.857 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:12.858 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.863 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:12.865 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:12.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.867 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:12.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.871 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.873 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:12.874 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:12.875 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 10 librados: watch_flush enter 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 10 librados: watch_flush exit 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 20 client.164261.objecter shutdown clearing up homeless session... 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 10 client.164261.objecter successfully canceled tick 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3d140416b0 legacy=0x7f3d14043b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 10 monclient: shutdown 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 >> v1:172.21.15.1:6790/0 conn(0x7f3d340a6ba0 legacy=0x7f3d340a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d33fff640 1 -- 172.21.15.1:0/969644404 reap_dead start 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d33fff640 1 -- 172.21.15.1:0/969644404 reap_dead start 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 shutdown_connections 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 >> 172.21.15.1:0/969644404 conn(0x7f3d340ac4b0 msgr2=0x7f3d340aa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 shutdown_connections 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 -- 172.21.15.1:0/969644404 wait complete. 2024-08-19T09:57:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.837+0000 7f3d3a060640 1 librados: shutdown 2024-08-19T09:57:12.879 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:12.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:57:12.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526392262}) 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1066731743}) 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2099875945}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1380429278}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2668856945}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3258196506}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2190221987}) 2024-08-19T09:57:12.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.883 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560940696}) 2024-08-19T09:57:12.883 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.871+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3702446014}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983954043}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723512986}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2774297725}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.891 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2910907047}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3665678596}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3379713195}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=541098476}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084406130}) 2024-08-19T09:57:12.892 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.889+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2598879182}) 2024-08-19T09:57:12.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192799588}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1791573441}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3816141186}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2886269349}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2590766180}) 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1406799604}) 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2132609386}) 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846644719}) 2024-08-19T09:57:12.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.906+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2429879071}) 2024-08-19T09:57:12.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224597062}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=109533299}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3505100125}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=55799836}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2078247895}) 2024-08-19T09:57:12.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2529155203}) 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=645940184}) 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.910+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476020322}) 2024-08-19T09:57:12.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.911+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2965004413}) 2024-08-19T09:57:12.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2989697573}) 2024-08-19T09:57:12.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590779617}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1115945819}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209569749}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2852182887}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3923060706}) 2024-08-19T09:57:12.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3856839066}) 2024-08-19T09:57:12.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187130231}) 2024-08-19T09:57:12.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.916+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2736322683}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093007842}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056193121}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1975782107}) 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3015274975}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3692317960}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3571642715}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2006442011}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859706609}) 2024-08-19T09:57:12.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.927+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=763599745}) 2024-08-19T09:57:12.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679855633}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716025250}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2660534116}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4233530109}) 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3656455080}) 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.931+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3272797637}) 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1217728893}) 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:12.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133573575}) 2024-08-19T09:57:12.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:12.932+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=715798049}) 2024-08-19T09:57:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.994+0000 7f59b9e32640 10 monclient: get_monmap_and_config 2024-08-19T09:57:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.994+0000 7f59b9e32640 10 monclient: build_initial_monmap 2024-08-19T09:57:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.994+0000 7f59b9e32640 10 monclient: monmap: 2024-08-19T09:57:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:12.995329+0000 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:12.995329+0000 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:12.997 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:12.997 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:12.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 1 Processor -- start 2024-08-19T09:57:12.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 1 -- start start 2024-08-19T09:57:12.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient: init 2024-08-19T09:57:12.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:12.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient: _add_conns ranks=[6,1,3,4,5,0,7,8,2] 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient(hunting): picked mon.noname-d con 0x7f59b40a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient(hunting): picked mon.noname-f con 0x7f59b40a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient(hunting): picked mon.noname-g con 0x7f59b413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b40a3b40 con 0x7f59b40a6ba0 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b40a8130 con 0x7f59b413f5c0 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b406ede0 con 0x7f59b40a67c0 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.995+0000 7f59b9e32640 10 monclient(hunting): authenticate will time out at 1089.071992s 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.996+0000 7f59b37fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f59b40a67c0 0x7f59b40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36940/0 (socket says 172.21.15.1:36940) 2024-08-19T09:57:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.996+0000 7f59b37fe640 1 -- 172.21.15.1:0/4137067788 learned_addr learned my addr 172.21.15.1:0/4137067788 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b37fe640 1 -- 172.21.15.1:0/4137067788 >> v1:172.21.15.1:6792/0 conn(0x7f59b40a67c0 legacy=0x7f59b40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b3fff640 1 -- 172.21.15.1:0/4137067788 >> v1:172.21.15.82:6790/0 conn(0x7f59b413f5c0 legacy=0x7f59b41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59abfff640 1 -- 172.21.15.1:0/4137067788 >> v1:172.21.15.82:6789/0 conn(0x7f59b40a6ba0 legacy=0x7f59b40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3171851118 0 0) 0x7f59b406ede0 con 0x7f59b40a67c0 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): my global_id is 164247 2024-08-19T09:57:13.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a0003520 con 0x7f59b40a67c0 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727644895 0 0) 0x7f59b40a3b40 con 0x7f59b40a6ba0 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): my global_id is 164188 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b406ede0 con 0x7f59b40a6ba0 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161951954 0 0) 0x7f59b40a8130 con 0x7f59b413f5c0 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient(hunting): my global_id is 134976 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b40a3b40 con 0x7f59b413f5c0 2024-08-19T09:57:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 917009758 0 0) 0x7f59a0003520 con 0x7f59b40a67c0 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 >> v1:172.21.15.82:6790/0 conn(0x7f59b413f5c0 legacy=0x7f59b41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 >> v1:172.21.15.82:6789/0 conn(0x7f59b40a6ba0 legacy=0x7f59b40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 monclient: found mon.noname-d 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b40a92a0 con 0x7f59b40a67c0 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59a4001ad0 con 0x7f59b40a67c0 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b9e32640 5 monclient: authenticate success, global_id 164247 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b9e32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: dump: 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.014 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b9e32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59a40048a0 con 0x7f59b40a67c0 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 1 -- 172.21.15.1:0/4137067788 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59a4004e10 con 0x7f59b40a67c0 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b9e32640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b9e32640 4 set_mon_vals no callback set 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.998+0000 7f59b2ffd640 10 monclient: dump: 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.016 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 10 monclient: shutdown 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 1 -- 172.21.15.1:0/4137067788 >> v1:172.21.15.1:6792/0 conn(0x7f59b40a67c0 legacy=0x7f59b40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 1 -- 172.21.15.1:0/4137067788 shutdown_connections 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 1 -- 172.21.15.1:0/4137067788 >> 172.21.15.1:0/4137067788 conn(0x7f59b40ac4b0 msgr2=0x7f59b40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 1 -- 172.21.15.1:0/4137067788 shutdown_connections 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 1 -- 172.21.15.1:0/4137067788 wait complete. 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:12.999+0000 7f59b9e32640 10 monclient: monmap: 2024-08-19T09:57:13.017 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.018 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 librados: starting msgr at 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 librados: starting objecter 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 Processor -- start 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 -- start start 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 librados: setting wanted keys 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 librados: calling monclient init 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient: init 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient: _add_conns ranks=[3,4,8,7,1,5,6,2,0] 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient(hunting): picked mon.noname-g con 0x7f59b40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient(hunting): picked mon.noname-c con 0x7f59b40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient(hunting): picked mon.noname-e con 0x7f59b413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b4139a70 con 0x7f59b40a67c0 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b40a9cc0 con 0x7f59b40a6ba0 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b413bdb0 con 0x7f59b413f5c0 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b9e32640 10 monclient(hunting): authenticate will time out at 1089.076821s 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b37fe640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f59b40a67c0 0x7f59b41ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:37256/0 (socket says 172.21.15.1:37256) 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.000+0000 7f59b37fe640 1 -- 172.21.15.1:0/1203350167 learned_addr learned my addr 172.21.15.1:0/1203350167 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59abfff640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6791/0 conn(0x7f59b40a6ba0 legacy=0x7f59b41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 client.?.objecter ms_handle_connect 0x7f59b40a6ba0 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 client.?.objecter ms_handle_connect 0x7f59b413f5c0 2024-08-19T09:57:13.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b3fff640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6793/0 conn(0x7f59b413f5c0 legacy=0x7f59b41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b37fe640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.82:6790/0 conn(0x7f59b40a67c0 legacy=0x7f59b41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 client.?.objecter ms_handle_connect 0x7f59b40a67c0 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223468276 0 0) 0x7f59b413bdb0 con 0x7f59b413f5c0 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): my global_id is 27791 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5994004eb0 con 0x7f59b413f5c0 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403764033 0 0) 0x7f59b406ede0 con 0x7f59b40a6ba0 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): my global_id is 27274 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b413bdb0 con 0x7f59b40a6ba0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279702680 0 0) 0x7f59b4139a70 con 0x7f59b40a67c0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): my global_id is 134985 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b406ede0 con 0x7f59b40a67c0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3565916227 0 0) 0x7f5994004eb0 con 0x7f59b413f5c0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59b4139a70 con 0x7f59b413f5c0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59ac001ff0 con 0x7f59b413f5c0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.023 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 45985445 0 0) 0x7f59b413bdb0 con 0x7f59b40a6ba0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5994004eb0 con 0x7f59b40a6ba0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3653205444 0 0) 0x7f59b406ede0 con 0x7f59b40a67c0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59b413bdb0 con 0x7f59b40a67c0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f599c001d00 con 0x7f59b40a6ba0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59a4004d90 con 0x7f59b40a67c0 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient(hunting): dump: 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3364106955 0 0) 0x7f59b4139a70 con 0x7f59b413f5c0 2024-08-19T09:57:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6791/0 conn(0x7f59b40a6ba0 legacy=0x7f59b41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.82:6790/0 conn(0x7f59b40a67c0 legacy=0x7f59b41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 monclient: found mon.e 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b41cf020 con 0x7f59b413f5c0 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 5 monclient: authenticate success, global_id 27791 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 10 monclient: _renew_subs 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59b413af40 con 0x7f59b413f5c0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 10 client.27791.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 10 monclient: _renew_subs 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f59b413a880 con 0x7f59b413f5c0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b9e32640 1 librados: init done 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59ac003c40 con 0x7f59b413f5c0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59ac004170 con 0x7f59b413f5c0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.003+0000 7f59b27fc640 4 set_mon_vals no callback set 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.004+0000 7f59b0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.004+0000 7f59b0ff9640 10 monclient: dump: 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.004+0000 7f59b9e32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.004+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59980065c0 con 0x7f5998002860 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.004+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f59ac004310 con 0x7f59b413f5c0 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.005+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(298..298 src has 1..298) ==== 5540+0+0 (unknown 2885429892 0 0) 0x7f59ac04e4b0 con 0x7f59b413f5c0 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.005+0000 7f59b0ff9640 10 client.27791.objecter ms_dispatch 0x7f59b4000b90 osd_map(298..298 src has 1..298) 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.005+0000 7f59b0ff9640 3 client.27791.objecter handle_osd_map got epochs [298,298] > 0 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.005+0000 7f59b0ff9640 3 client.27791.objecter handle_osd_map decoding full epoch 298 2024-08-19T09:57:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.005+0000 7f59b0ff9640 20 client.27791.objecter dump_active .. 0 homeless 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.005+0000 7f59abfff640 10 monclient: get_auth_request con 0x7f59940418a0 auth_method 0 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b37fe640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6792/0 conn(0x7f5998002860 legacy=0x7f5998004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 client.27791.objecter ms_handle_connect 0x7f5998002860 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 client.27791.objecter resend_mon_ops 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844639815 0 0) 0x7f59980065c0 con 0x7f5998002860 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient(hunting): my global_id is 164256 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b4139a70 con 0x7f5998002860 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3566471021 0 0) 0x7f59b4139a70 con 0x7f5998002860 2024-08-19T09:57:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59980065c0 con 0x7f5998002860 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59a40048a0 con 0x7f5998002860 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient: dump: 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2783030245 0 0) 0x7f59980065c0 con 0x7f5998002860 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f599800a160 con 0x7f5998002860 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 client.27791.objecter ms_handle_connect 0x7f59940418a0 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f59a4005740 con 0x7f5998002860 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:13.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6792/0 conn(0x7f5998002860 legacy=0x7f5998004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.007+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6792/0 conn(0x7f5998002860 legacy=0x7f5998004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.014+0000 7f59b9e32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.014+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5998009e70 con 0x7f5998009130 2024-08-19T09:57:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b37fe640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6792/0 conn(0x7f5998009130 legacy=0x7f59980095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 client.27791.objecter ms_handle_connect 0x7f5998009130 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 client.27791.objecter resend_mon_ops 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824691065 0 0) 0x7f5998009e70 con 0x7f5998009130 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient(hunting): my global_id is 164265 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59980065c0 con 0x7f5998009130 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4089599115 0 0) 0x7f59980065c0 con 0x7f5998009130 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5998009e70 con 0x7f5998009130 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59a4005f50 con 0x7f5998009130 2024-08-19T09:57:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient: dump: 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1537919604 0 0) 0x7f5998009e70 con 0x7f5998009130 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f599800c410 con 0x7f5998009130 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 376290142) 0x7f59a4008250 con 0x7f5998009130 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6792/0 conn(0x7f5998009130 legacy=0x7f59980095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.016+0000 7f59b0ff9640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6792/0 conn(0x7f5998009130 legacy=0x7f59980095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19834, 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:13.040 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.041 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:13.042 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:13.044 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.045 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.048 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.049 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.050 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.052 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.053 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.054 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.055 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.057 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.058 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:13.060 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 10 librados: watch_flush enter 2024-08-19T09:57:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 10 librados: watch_flush exit 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 20 client.27791.objecter shutdown clearing up homeless session... 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 10 client.27791.objecter successfully canceled tick 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.82:6812/2549104415 conn(0x7f59940418a0 legacy=0x7f5994043d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 10 monclient: shutdown 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 >> v1:172.21.15.1:6793/0 conn(0x7f59b413f5c0 legacy=0x7f59b41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b3fff640 1 -- 172.21.15.1:0/1203350167 reap_dead start 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b3fff640 1 -- 172.21.15.1:0/1203350167 reap_dead start 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 shutdown_connections 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 >> 172.21.15.1:0/1203350167 conn(0x7f59b40ac4b0 msgr2=0x7f59b41421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 shutdown_connections 2024-08-19T09:57:13.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 -- 172.21.15.1:0/1203350167 wait complete. 2024-08-19T09:57:13.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.017+0000 7f59b9e32640 1 librados: shutdown 2024-08-19T09:57:13.064 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:13.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:57:13.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235388373}) 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308983244}) 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.085 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2701188474}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2568721250}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=36133803}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2114606856}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.086 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3654527077}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582960161}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.058+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1525166676}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461636631}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2992587391}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2204428887}) 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.087 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1766631021}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1044605871}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2405785311}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980035477}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789654489}) 2024-08-19T09:57:13.088 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.059+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=942846636}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45939310}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413722229}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=939162021}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.089 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2850984127}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3803634662}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4250171824}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356832050}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381439097}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.060+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989754299}) 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.090 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495579712}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2800869963}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1675701901}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1414525918}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.091 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=918250459}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3058961473}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2594111409}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897902083}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.061+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2575554331}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13} crc {logm=3933329255,mdsmap=61829903,mgr=2021673060,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3299166291,mon_config_key=2370035867}) 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.092 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=65,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13} crc {logm=743783660,mdsmap=3690439819,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2480333628,mon_config_key=1721741179}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13} crc {logm=2573942284,mdsmap=61829903,mgr=865555092,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1721741179}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15} crc {logm=3847717465,mdsmap=61829903,mgr=2689613648,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3472800138,mon_config_key=3720761797}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13} crc {logm=3485871485,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3299166291,mon_config_key=2909152007}) 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.093 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=62,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=2} crc {logm=2592366020,mdsmap=3690439819,mgr=865555092,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1083757145,mon_config_key=561037852,monmap=3580681835}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=64,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15} crc {logm=2140453927,mdsmap=3690439819,mgr=2689613648,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=2909786187,mon_config_key=778439930}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=2956388245,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3521215106,mon_config_key=1857860018}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.063+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=14} crc {logm=2800855803,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgrstat=3521215106,mon_config_key=2705821000}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.064+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.064+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.064+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=60} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=1075655958,osdmap=389643093}) 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.064+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.094 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=58} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=2239274195,osdmap=1607213398}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=60} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=2185554599,osdmap=486238236}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=59} crc {mon_config_key=1018616092,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=3465090670,osdmap=627535345}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=59} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=193370827,osdmap=1764887465}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.095 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=61} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=1609305515,osd_pg_creating=2410592366,osd_snap=2342168089,osdmap=922644590}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=59} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=2149214569,osdmap=1050563256}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=62} crc {mon_config_key=1759712728,monmap=3580681835,osd_metadata=2559605274,osd_pg_creating=2410592366,osd_snap=2562738570,osdmap=2419313051}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.065+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=58} crc {mon_config_key=1759712728,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=2410592366,osd_snap=2239274195,osdmap=1481535656}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=886583811}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3347106016}) 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.096 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2371186980}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3341998758}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4125314827}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=460984423}) 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.097 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4065862840}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=51182694}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.066+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1841684145}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3392727200}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2290476083}) 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.098 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2062319227}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2961129094}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2999663995}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2942878492}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.099 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1043764412}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=787435394}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.068+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1643075588}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.070+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.070+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.070+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=212226959}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.070+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.070+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2952473651}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=117381548}) 2024-08-19T09:57:13.100 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4115057753}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3455831307}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3396729999}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=414448385}) 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2323570674}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.071+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3375578891}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=82} crc {osdmap=3730714785}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=1617941056}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=80} crc {osdmap=1823660716}) 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=81} crc {osdmap=2312028436}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=77} crc {osdmap=3616619100}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=79} crc {osdmap=1428487696}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=79} crc {osdmap=2237399600}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=1875337541}) 2024-08-19T09:57:13.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:57:13.073+0000 7f45ee974640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=81} crc {osdmap=642220000}) 2024-08-19T09:57:13.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.174+0000 7f26f8bb8640 10 monclient: get_monmap_and_config 2024-08-19T09:57:13.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.174+0000 7f26f8bb8640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.174+0000 7f26f8bb8640 10 monclient: monmap: 2024-08-19T09:57:13.175 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:13.175450+0000 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:13.175450+0000 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.177 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 1 Processor -- start 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 1 -- start start 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient: init 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient: _add_conns ranks=[6,4,5,3,2,8,1,0,7] 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient(hunting): picked mon.noname-d con 0x7f26f40a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient(hunting): picked mon.noname-c con 0x7f26f40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient(hunting): picked mon.noname-h con 0x7f26f413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f40a3b40 con 0x7f26f40a6ba0 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f40a8130 con 0x7f26f413f5c0 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f406ede0 con 0x7f26f40a67c0 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f8bb8640 10 monclient(hunting): authenticate will time out at 1089.251814s 2024-08-19T09:57:13.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f259c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f26f40a67c0 0x7f26f40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36960/0 (socket says 172.21.15.1:36960) 2024-08-19T09:57:13.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26f259c640 1 -- 172.21.15.1:0/1056806356 learned_addr learned my addr 172.21.15.1:0/1056806356 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.175+0000 7f26ebfff640 1 --1- 172.21.15.1:0/1056806356 >> v1:172.21.15.1:6791/0 conn(0x7f26f40a6ba0 0x7f26f40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37498/0 (socket says 172.21.15.1:37498) 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26ebfff640 1 -- 172.21.15.1:0/1056806356 >> v1:172.21.15.1:6791/0 conn(0x7f26f40a6ba0 legacy=0x7f26f40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f259c640 1 -- 172.21.15.1:0/1056806356 >> v1:172.21.15.1:6792/0 conn(0x7f26f40a67c0 legacy=0x7f26f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f2d9d640 1 -- 172.21.15.1:0/1056806356 >> v1:172.21.15.82:6791/0 conn(0x7f26f413f5c0 legacy=0x7f26f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192502131 0 0) 0x7f26f40a3b40 con 0x7f26f40a6ba0 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): my global_id is 27283 2024-08-19T09:57:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e0003520 con 0x7f26f40a6ba0 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663837729 0 0) 0x7f26f406ede0 con 0x7f26f40a67c0 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): my global_id is 164274 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f40a3b40 con 0x7f26f40a67c0 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767150504 0 0) 0x7f26f40a8130 con 0x7f26f413f5c0 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient(hunting): my global_id is 27005 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f406ede0 con 0x7f26f413f5c0 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1694289008 0 0) 0x7f26f40a3b40 con 0x7f26f40a67c0 2024-08-19T09:57:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 >> v1:172.21.15.82:6791/0 conn(0x7f26f413f5c0 legacy=0x7f26f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 >> v1:172.21.15.1:6791/0 conn(0x7f26f40a6ba0 legacy=0x7f26f40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 monclient: found mon.noname-d 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f40a92a0 con 0x7f26f40a67c0 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26e4001a40 con 0x7f26f40a67c0 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f8bb8640 5 monclient: authenticate success, global_id 164274 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f8bb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: dump: 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f8bb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26e4001c50 con 0x7f26f40a67c0 2024-08-19T09:57:13.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 1 -- 172.21.15.1:0/1056806356 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26e4004be0 con 0x7f26f40a67c0 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f8bb8640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f8bb8640 4 set_mon_vals no callback set 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.178+0000 7f26f1d9b640 10 monclient: dump: 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.188 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 10 monclient: shutdown 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1056806356 >> v1:172.21.15.1:6792/0 conn(0x7f26f40a67c0 legacy=0x7f26f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1056806356 shutdown_connections 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1056806356 >> 172.21.15.1:0/1056806356 conn(0x7f26f40ac4b0 msgr2=0x7f26f40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1056806356 shutdown_connections 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1056806356 wait complete. 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.179+0000 7f26f8bb8640 10 monclient: monmap: 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:13.190 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 librados: starting msgr at 2024-08-19T09:57:13.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 librados: starting objecter 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 Processor -- start 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 -- start start 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 librados: setting wanted keys 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 librados: calling monclient init 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient: init 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient: _add_conns ranks=[3,1,8,6,7,0,4,2,5] 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient(hunting): picked mon.noname-g con 0x7f26f40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient(hunting): picked mon.noname-f con 0x7f26f40a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient(hunting): picked mon.noname-e con 0x7f26f413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f413b8a0 con 0x7f26f40a6ba0 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f413bb80 con 0x7f26f40a67c0 2024-08-19T09:57:13.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f41cf020 con 0x7f26f413f5c0 2024-08-19T09:57:13.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f8bb8640 10 monclient(hunting): authenticate will time out at 1089.256763s 2024-08-19T09:57:13.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f2d9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f26f413f5c0 0x7f26f41d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52950/0 (socket says 172.21.15.1:52950) 2024-08-19T09:57:13.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.180+0000 7f26f2d9d640 1 -- 172.21.15.1:0/1314106109 learned_addr learned my addr 172.21.15.1:0/1314106109 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.182+0000 7f26f2d9d640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26f413f5c0 legacy=0x7f26f41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 client.?.objecter ms_handle_connect 0x7f26f413f5c0 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471578506 0 0) 0x7f26f41cf020 con 0x7f26f413f5c0 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): my global_id is 27800 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d4004eb0 con 0x7f26f413f5c0 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f259c640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.82:6790/0 conn(0x7f26f40a67c0 legacy=0x7f26f41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 client.?.objecter ms_handle_connect 0x7f26f40a67c0 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26ebfff640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.82:6789/0 conn(0x7f26f40a6ba0 legacy=0x7f26f41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 client.?.objecter ms_handle_connect 0x7f26f40a6ba0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2625174848 0 0) 0x7f26f413bb80 con 0x7f26f40a67c0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): my global_id is 134994 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f41cf020 con 0x7f26f40a67c0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1943497214 0 0) 0x7f26d4004eb0 con 0x7f26f413f5c0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26f413bb80 con 0x7f26f413f5c0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26ec0020b0 con 0x7f26f413f5c0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:57:13.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): dump: 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.200 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728612045 0 0) 0x7f26f413b8a0 con 0x7f26f40a6ba0 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient(hunting): my global_id is 164197 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d4004eb0 con 0x7f26f40a6ba0 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2456357337 0 0) 0x7f26f413bb80 con 0x7f26f413f5c0 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.82:6790/0 conn(0x7f26f40a67c0 legacy=0x7f26f41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.82:6789/0 conn(0x7f26f40a6ba0 legacy=0x7f26f41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 monclient: found mon.e 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f413a380 con 0x7f26f413f5c0 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 5 monclient: authenticate success, global_id 27800 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 10 monclient: _renew_subs 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26f4139820 con 0x7f26f413f5c0 2024-08-19T09:57:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 10 client.27800.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 10 monclient: _renew_subs 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f26f413b2e0 con 0x7f26f413f5c0 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f8bb8640 1 librados: init done 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26ec003ef0 con 0x7f26f413f5c0 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26ec0043e0 con 0x7f26f413f5c0 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26f159a640 4 set_mon_vals no callback set 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.183+0000 7f26eaffd640 10 monclient: dump: 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.203 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.204 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26f8bb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26d80065c0 con 0x7f26d8002860 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f26ec004580 con 0x7f26f413f5c0 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(298..298 src has 1..298) ==== 5540+0+0 (unknown 2885429892 0 0) 0x7f26ec04e500 con 0x7f26f413f5c0 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26eaffd640 10 client.27800.objecter ms_dispatch 0x7f26f4000b90 osd_map(298..298 src has 1..298) 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26eaffd640 3 client.27800.objecter handle_osd_map got epochs [298,298] > 0 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26eaffd640 3 client.27800.objecter handle_osd_map decoding full epoch 298 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.184+0000 7f26ebfff640 10 monclient: get_auth_request con 0x7f26d4041060 auth_method 0 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.185+0000 7f26eaffd640 20 client.27800.objecter dump_active .. 0 homeless 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.186+0000 7f26f259c640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26d8002860 legacy=0x7f26d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.186+0000 7f26eaffd640 10 client.27800.objecter ms_handle_connect 0x7f26d8002860 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.186+0000 7f26eaffd640 10 client.27800.objecter resend_mon_ops 2024-08-19T09:57:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569828435 0 0) 0x7f26d80065c0 con 0x7f26d8002860 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient(hunting): my global_id is 27809 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f413bb80 con 0x7f26d8002860 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4241770359 0 0) 0x7f26f413bb80 con 0x7f26d8002860 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26d80065c0 con 0x7f26d8002860 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26e4001c50 con 0x7f26d8002860 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient: dump: 2024-08-19T09:57:13.206 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 client.27800.objecter ms_handle_connect 0x7f26d4041060 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3470261864 0 0) 0x7f26d80065c0 con 0x7f26d8002860 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26d800a160 con 0x7f26d8002860 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f26e40053c0 con 0x7f26d8002860 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26d8002860 legacy=0x7f26d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.187+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26d8002860 legacy=0x7f26d8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.195+0000 7f26f8bb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.195+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26d8009e70 con 0x7f26d8009130 2024-08-19T09:57:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 client.27800.objecter ms_handle_connect 0x7f26d8009130 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 client.27800.objecter resend_mon_ops 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26f259c640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26d8009130 legacy=0x7f26d80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3082456613 0 0) 0x7f26d8009e70 con 0x7f26d8009130 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient(hunting): my global_id is 27818 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d80065c0 con 0x7f26d8009130 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 728898431 0 0) 0x7f26d80065c0 con 0x7f26d8009130 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26d8009e70 con 0x7f26d8009130 2024-08-19T09:57:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f26e4005bf0 con 0x7f26d8009130 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.198+0000 7f26eaffd640 10 monclient: dump: 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1661219620 0 0) 0x7f26d8009e70 con 0x7f26d8009130 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26d800c410 con 0x7f26d8009130 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2104309410) 0x7f26e4007fe0 con 0x7f26d8009130 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26d8009130 legacy=0x7f26d80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.199+0000 7f26eaffd640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26d8009130 legacy=0x7f26d80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.214 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:13.214 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.214 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.214 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:13.214 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:13.214 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 439472, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:57:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.216 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.217 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:13.218 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.219 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.220 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.222 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.223 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.227 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.228 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.230 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.232 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:13.235 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.235 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.235 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:13.235 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 10 librados: watch_flush enter 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 10 librados: watch_flush exit 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 20 client.27800.objecter shutdown clearing up homeless session... 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 10 client.27800.objecter successfully canceled tick 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.82:6812/2549104415 conn(0x7f26d4041060 legacy=0x7f26d4043520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 10 monclient: shutdown 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 >> v1:172.21.15.1:6793/0 conn(0x7f26f413f5c0 legacy=0x7f26f41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f2d9d640 1 -- 172.21.15.1:0/1314106109 reap_dead start 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f2d9d640 1 -- 172.21.15.1:0/1314106109 reap_dead start 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 shutdown_connections 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 >> 172.21.15.1:0/1314106109 conn(0x7f26f40ac4b0 msgr2=0x7f26f4141bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 shutdown_connections 2024-08-19T09:57:13.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 -- 172.21.15.1:0/1314106109 wait complete. 2024-08-19T09:57:13.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.200+0000 7f26f8bb8640 1 librados: shutdown 2024-08-19T09:57:13.238 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:13.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:57:13.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:57:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.337+0000 7f5f4598a640 10 monclient: get_monmap_and_config 2024-08-19T09:57:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.337+0000 7f5f4598a640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.337+0000 7f5f4598a640 10 monclient: monmap: 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:13.338477+0000 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:13.338477+0000 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.340 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 1 Processor -- start 2024-08-19T09:57:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 1 -- start start 2024-08-19T09:57:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient: init 2024-08-19T09:57:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient: _add_conns ranks=[4,2,8,1,7,0,6,5,3] 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient(hunting): picked mon.noname-c con 0x7f5f400a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient(hunting): picked mon.noname-b con 0x7f5f400a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient(hunting): picked mon.noname-e con 0x7f5f4013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f400a3b40 con 0x7f5f400a6ba0 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f400a8130 con 0x7f5f400a67c0 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4006ede0 con 0x7f5f4013f5c0 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f4598a640 10 monclient(hunting): authenticate will time out at 1089.414705s 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f3effd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f5f400a67c0 0x7f5f400a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37510/0 (socket says 172.21.15.1:37510) 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f3effd640 1 -- 172.21.15.1:0/2517510793 learned_addr learned my addr 172.21.15.1:0/2517510793 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.338+0000 7f5f367fc640 1 --1- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6790/0 conn(0x7f5f400a6ba0 0x7f5f400a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48118/0 (socket says 172.21.15.1:48118) 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.340+0000 7f5f3effd640 1 -- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6791/0 conn(0x7f5f400a67c0 legacy=0x7f5f400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f367fc640 1 -- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6790/0 conn(0x7f5f400a6ba0 legacy=0x7f5f400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3f7fe640 1 -- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6793/0 conn(0x7f5f4013f5c0 legacy=0x7f5f401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651970851 0 0) 0x7f5f400a8130 con 0x7f5f400a67c0 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): my global_id is 27292 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f2c003520 con 0x7f5f400a67c0 2024-08-19T09:57:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2785110223 0 0) 0x7f5f400a3b40 con 0x7f5f400a6ba0 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): my global_id is 164279 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f400a8130 con 0x7f5f400a6ba0 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 967457862 0 0) 0x7f5f4006ede0 con 0x7f5f4013f5c0 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient(hunting): my global_id is 27836 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f400a3b40 con 0x7f5f4013f5c0 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 955702854 0 0) 0x7f5f400a8130 con 0x7f5f400a6ba0 2024-08-19T09:57:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6793/0 conn(0x7f5f4013f5c0 legacy=0x7f5f401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6791/0 conn(0x7f5f400a67c0 legacy=0x7f5f400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 monclient: found mon.noname-b 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f400a92a0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f280019f0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: dump: 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.352 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f4598a640 5 monclient: authenticate success, global_id 164279 2024-08-19T09:57:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f4598a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f28001cc0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 1 -- 172.21.15.1:0/2517510793 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f28004b20 con 0x7f5f400a6ba0 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f4598a640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f4598a640 4 set_mon_vals no callback set 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.341+0000 7f5f3e7fc640 10 monclient: dump: 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 10 monclient: shutdown 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 1 -- 172.21.15.1:0/2517510793 >> v1:172.21.15.1:6790/0 conn(0x7f5f400a6ba0 legacy=0x7f5f400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 1 -- 172.21.15.1:0/2517510793 shutdown_connections 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 1 -- 172.21.15.1:0/2517510793 >> 172.21.15.1:0/2517510793 conn(0x7f5f400ac4b0 msgr2=0x7f5f400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 1 -- 172.21.15.1:0/2517510793 shutdown_connections 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 1 -- 172.21.15.1:0/2517510793 wait complete. 2024-08-19T09:57:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.342+0000 7f5f4598a640 10 monclient: monmap: 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 librados: starting msgr at 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 librados: starting objecter 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 Processor -- start 2024-08-19T09:57:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 -- start start 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 librados: setting wanted keys 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 librados: calling monclient init 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient: init 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient: _add_conns ranks=[3,4,5,7,0,1,6,2,8] 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient(hunting): picked mon.noname-g con 0x7f5f400a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient(hunting): picked mon.noname-c con 0x7f5f400a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient(hunting): picked mon.noname-h con 0x7f5f4013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f40139ad0 con 0x7f5f400a67c0 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4013a990 con 0x7f5f400a6ba0 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f40142e70 con 0x7f5f4013f5c0 2024-08-19T09:57:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f4598a640 10 monclient(hunting): authenticate will time out at 1089.419835s 2024-08-19T09:57:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f367fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f5f400a6ba0 0x7f5f400a0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37526/0 (socket says 172.21.15.1:37526) 2024-08-19T09:57:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.343+0000 7f5f367fc640 1 -- 172.21.15.1:0/2459568823 learned_addr learned my addr 172.21.15.1:0/2459568823 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f367fc640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.1:6791/0 conn(0x7f5f400a6ba0 legacy=0x7f5f400a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 client.?.objecter ms_handle_connect 0x7f5f400a6ba0 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214311280 0 0) 0x7f5f4013a990 con 0x7f5f400a6ba0 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 27301 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f20004eb0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 client.?.objecter ms_handle_connect 0x7f5f4013f5c0 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3f7fe640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6791/0 conn(0x7f5f4013f5c0 legacy=0x7f5f401d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 client.?.objecter ms_handle_connect 0x7f5f400a67c0 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3effd640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6790/0 conn(0x7f5f400a67c0 legacy=0x7f5f400a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 629732234 0 0) 0x7f5f20004eb0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f4013a990 con 0x7f5f400a6ba0 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f28004b20 con 0x7f5f400a6ba0 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): dump: 2024-08-19T09:57:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261804841 0 0) 0x7f5f40142e70 con 0x7f5f4013f5c0 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 27023 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f20004eb0 con 0x7f5f4013f5c0 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1708391324 0 0) 0x7f5f4013a990 con 0x7f5f400a6ba0 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6791/0 conn(0x7f5f4013f5c0 legacy=0x7f5f401d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6790/0 conn(0x7f5f400a67c0 legacy=0x7f5f400a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 monclient: found mon.c 2024-08-19T09:57:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f400a17d0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 5 monclient: authenticate success, global_id 27301 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 10 monclient: _renew_subs 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4013a6b0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 10 client.27301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 10 monclient: _renew_subs 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f400a0e30 con 0x7f5f400a6ba0 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f4598a640 1 librados: init done 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f28001cc0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f28005450 con 0x7f5f400a6ba0 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3dffb640 4 set_mon_vals no callback set 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.346+0000 7f5f3cff9640 10 monclient: dump: 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f4598a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f240065c0 con 0x7f5f24002860 2024-08-19T09:57:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5f280118b0 con 0x7f5f400a6ba0 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7f5f2804e350 con 0x7f5f400a6ba0 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f3cff9640 10 client.27301.objecter ms_dispatch 0x7f5f40000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f3cff9640 3 client.27301.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f3cff9640 3 client.27301.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.347+0000 7f5f3cff9640 20 client.27301.objecter dump_active .. 0 homeless 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.348+0000 7f5f3f7fe640 10 monclient: get_auth_request con 0x7f5f20040ec0 auth_method 0 2024-08-19T09:57:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 10 client.27301.objecter ms_handle_connect 0x7f5f24002860 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3effd640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6789/0 conn(0x7f5f24002860 legacy=0x7f5f24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 10 client.27301.objecter resend_mon_ops 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270531075 0 0) 0x7f5f240065c0 con 0x7f5f24002860 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 164215 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.349+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f4013a990 con 0x7f5f24002860 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 187257623 0 0) 0x7f5f4013a990 con 0x7f5f24002860 2024-08-19T09:57:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f240065c0 con 0x7f5f24002860 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f300047e0 con 0x7f5f24002860 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 10 monclient: dump: 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 10 client.27301.objecter ms_handle_connect 0x7f5f20040ec0 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 739948126 0 0) 0x7f5f240065c0 con 0x7f5f24002860 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f2400a160 con 0x7f5f24002860 2024-08-19T09:57:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5f30004980 con 0x7f5f24002860 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6789/0 conn(0x7f5f24002860 legacy=0x7f5f24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.350+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6789/0 conn(0x7f5f24002860 legacy=0x7f5f24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.356+0000 7f5f4598a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.356+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f24009e50 con 0x7f5f24009110 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3effd640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6789/0 conn(0x7f5f24009110 legacy=0x7f5f240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 client.27301.objecter ms_handle_connect 0x7f5f24009110 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 client.27301.objecter resend_mon_ops 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429195566 0 0) 0x7f5f24009e50 con 0x7f5f24009110 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient(hunting): my global_id is 164224 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f240065c0 con 0x7f5f24009110 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 915870204 0 0) 0x7f5f240065c0 con 0x7f5f24009110 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f24009e50 con 0x7f5f24009110 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f300051f0 con 0x7f5f24009110 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:57:13.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.359+0000 7f5f3cff9640 10 monclient: dump: 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2700557582 0 0) 0x7f5f24009e50 con 0x7f5f24009110 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f2400c3f0 con 0x7f5f24009110 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 2514581049) 0x7f5f30007510 con 0x7f5f24009110 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6789/0 conn(0x7f5f24009110 legacy=0x7f5f240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3cff9640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6789/0 conn(0x7f5f24009110 legacy=0x7f5f240095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 20172, 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:13.382 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:13.383 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.384 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.385 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:13.386 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:13.387 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.388 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:13.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.390 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.395 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.397 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.398 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.399 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.401 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.402 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.403 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:13.404 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:13.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 10 librados: watch_flush enter 2024-08-19T09:57:13.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 10 librados: watch_flush exit 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 20 client.27301.objecter shutdown clearing up homeless session... 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 10 client.27301.objecter successfully canceled tick 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5f20040ec0 legacy=0x7f5f20043380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 10 monclient: shutdown 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 >> v1:172.21.15.1:6791/0 conn(0x7f5f400a6ba0 legacy=0x7f5f400a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3f7fe640 1 -- 172.21.15.1:0/2459568823 reap_dead start 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f3f7fe640 1 -- 172.21.15.1:0/2459568823 reap_dead start 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 shutdown_connections 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 >> 172.21.15.1:0/2459568823 conn(0x7f5f400ac4b0 msgr2=0x7f5f400aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 shutdown_connections 2024-08-19T09:57:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.360+0000 7f5f4598a640 1 -- 172.21.15.1:0/2459568823 wait complete. 2024-08-19T09:57:13.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.361+0000 7f5f4598a640 1 librados: shutdown 2024-08-19T09:57:13.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:13.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:57:13.408 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:57:13.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.506+0000 7f3d54baa640 10 monclient: get_monmap_and_config 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.506+0000 7f3d54baa640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.506+0000 7f3d54baa640 10 monclient: monmap: 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:13.507859+0000 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:13.507859+0000 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.508 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.509 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.507+0000 7f3d54baa640 1 Processor -- start 2024-08-19T09:57:13.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.507+0000 7f3d54baa640 1 -- start start 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.507+0000 7f3d54baa640 10 monclient: init 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient: _add_conns ranks=[7,4,6,8,2,5,0,3,1] 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient(hunting): picked mon.noname-i con 0x7f3d500a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient(hunting): picked mon.noname-c con 0x7f3d500a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient(hunting): picked mon.noname-d con 0x7f3d5013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d500a3b40 con 0x7f3d500a6ba0 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d500a8130 con 0x7f3d5013f5c0 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5006ede0 con 0x7f3d500a67c0 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d54baa640 10 monclient(hunting): authenticate will time out at 1089.584233s 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d4dd9b640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f3d500a6ba0 0x7f3d500a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37538/0 (socket says 172.21.15.1:37538) 2024-08-19T09:57:13.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d4dd9b640 1 -- 172.21.15.1:0/3751700626 learned_addr learned my addr 172.21.15.1:0/3751700626 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.508+0000 7f3d4ed9d640 1 --1- 172.21.15.1:0/3751700626 >> v1:172.21.15.1:6792/0 conn(0x7f3d5013f5c0 0x7f3d501419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36968/0 (socket says 172.21.15.1:36968) 2024-08-19T09:57:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4dd9b640 1 -- 172.21.15.1:0/3751700626 >> v1:172.21.15.1:6791/0 conn(0x7f3d500a6ba0 legacy=0x7f3d500a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4ed9d640 1 -- 172.21.15.1:0/3751700626 >> v1:172.21.15.1:6792/0 conn(0x7f3d5013f5c0 legacy=0x7f3d501419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4e59c640 1 -- 172.21.15.1:0/3751700626 >> v1:172.21.15.82:6792/0 conn(0x7f3d500a67c0 legacy=0x7f3d500a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765572477 0 0) 0x7f3d500a3b40 con 0x7f3d500a6ba0 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): my global_id is 27310 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d3c003520 con 0x7f3d500a6ba0 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 424628451 0 0) 0x7f3d5006ede0 con 0x7f3d500a67c0 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): my global_id is 144502 2024-08-19T09:57:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d500a3b40 con 0x7f3d500a67c0 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 984167288 0 0) 0x7f3d500a8130 con 0x7f3d5013f5c0 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient(hunting): my global_id is 164292 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d5006ede0 con 0x7f3d5013f5c0 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4ed9d640 1 --1- 172.21.15.1:0/3751700626 >> v1:172.21.15.1:6792/0 conn(0x7f3d5013f5c0 0x7f3d501419b0 :-1 s=OPENED pgs=3350 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f3d5006ede0 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1501615611 0 0) 0x7f3d500a3b40 con 0x7f3d500a67c0 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 >> v1:172.21.15.1:6792/0 conn(0x7f3d5013f5c0 legacy=0x7f3d501419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 >> v1:172.21.15.1:6791/0 conn(0x7f3d500a6ba0 legacy=0x7f3d500a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 1 monclient: found mon.noname-i 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.510+0000 7f3d4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d500a92a0 con 0x7f3d500a67c0 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d40001da0 con 0x7f3d500a67c0 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: dump: 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 5 monclient: authenticate success, global_id 144502 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d40002070 con 0x7f3d500a67c0 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 1 -- 172.21.15.1:0/3751700626 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d40003cd0 con 0x7f3d500a67c0 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d4d59a640 10 monclient: dump: 2024-08-19T09:57:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.521 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 4 set_mon_vals no callback set 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 10 monclient: shutdown 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 1 -- 172.21.15.1:0/3751700626 >> v1:172.21.15.82:6792/0 conn(0x7f3d500a67c0 legacy=0x7f3d500a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 1 -- 172.21.15.1:0/3751700626 shutdown_connections 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 1 -- 172.21.15.1:0/3751700626 >> 172.21.15.1:0/3751700626 conn(0x7f3d500ac4b0 msgr2=0x7f3d500a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 1 -- 172.21.15.1:0/3751700626 shutdown_connections 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.511+0000 7f3d54baa640 1 -- 172.21.15.1:0/3751700626 wait complete. 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 10 monclient: monmap: 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 1 librados: starting msgr at 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 1 librados: starting objecter 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 1 Processor -- start 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 1 -- start start 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 1 librados: setting wanted keys 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 1 librados: calling monclient init 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.512+0000 7f3d54baa640 10 monclient: init 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient: _add_conns ranks=[3,0,7,6,8,4,5,2,1] 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient(hunting): picked mon.noname-g con 0x7f3d500a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient(hunting): picked mon.noname-a con 0x7f3d500a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient(hunting): picked mon.noname-i con 0x7f3d5013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5013b380 con 0x7f3d500a6ba0 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d50142e40 con 0x7f3d500a67c0 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5013d6e0 con 0x7f3d5013f5c0 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d54baa640 10 monclient(hunting): authenticate will time out at 1089.589353s 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d4dd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3d500a6ba0 0x7f3d501d2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38230/0 (socket says 172.21.15.1:38230) 2024-08-19T09:57:13.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.513+0000 7f3d4dd9b640 1 -- 172.21.15.1:0/820672098 learned_addr learned my addr 172.21.15.1:0/820672098 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d4dd9b640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.1:6789/0 conn(0x7f3d500a6ba0 legacy=0x7f3d501d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 client.?.objecter ms_handle_connect 0x7f3d500a6ba0 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d4e59c640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d500a67c0 legacy=0x7f3d5013ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 client.?.objecter ms_handle_connect 0x7f3d500a67c0 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 client.?.objecter ms_handle_connect 0x7f3d5013f5c0 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d4ed9d640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6792/0 conn(0x7f3d5013f5c0 legacy=0x7f3d501d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 505946296 0 0) 0x7f3d5013b380 con 0x7f3d500a6ba0 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): my global_id is 164214 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d30004eb0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978192781 0 0) 0x7f3d5013d6e0 con 0x7f3d5013f5c0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 10 monclient(hunting): my global_id is 144511 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d5013b380 con 0x7f3d5013f5c0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.515+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 144555583 0 0) 0x7f3d30004eb0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d5013d6e0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d380047e0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557493539 0 0) 0x7f3d500a92a0 con 0x7f3d500a67c0 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient(hunting): my global_id is 135012 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d30004eb0 con 0x7f3d500a67c0 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3574130869 0 0) 0x7f3d5013d6e0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6792/0 conn(0x7f3d5013f5c0 legacy=0x7f3d501d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d500a67c0 legacy=0x7f3d5013ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 monclient: found mon.a 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d501429d0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 5 monclient: authenticate success, global_id 164214 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 10 monclient: _renew_subs 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5013d400 con 0x7f3d500a6ba0 2024-08-19T09:57:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 10 client.164214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 10 monclient: _renew_subs 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d5013c120 con 0x7f3d500a6ba0 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d54baa640 1 librados: init done 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d38004ca0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d380051d0 con 0x7f3d500a6ba0 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d4cd99640 4 set_mon_vals no callback set 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.516+0000 7f3d46ffd640 10 monclient: dump: 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d54baa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d340065c0 con 0x7f3d34002860 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3d38005470 con 0x7f3d500a6ba0 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7f3d3804e340 con 0x7f3d500a6ba0 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d46ffd640 10 client.164214.objecter ms_dispatch 0x7f3d50000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d46ffd640 3 client.164214.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d46ffd640 3 client.164214.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d46ffd640 20 client.164214.objecter dump_active .. 0 homeless 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.517+0000 7f3d4ed9d640 10 monclient: get_auth_request con 0x7f3d30041110 auth_method 0 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d4e59c640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 legacy=0x7f3d34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 client.164214.objecter ms_handle_connect 0x7f3d34002860 2024-08-19T09:57:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 client.164214.objecter resend_mon_ops 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133808673 0 0) 0x7f3d340065c0 con 0x7f3d34002860 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient(hunting): my global_id is 135021 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d5013d6e0 con 0x7f3d34002860 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 client.164214.objecter ms_handle_connect 0x7f3d30041110 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 402468734 0 0) 0x7f3d5013d6e0 con 0x7f3d34002860 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d340065c0 con 0x7f3d34002860 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d400020c0 con 0x7f3d34002860 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.519+0000 7f3d46ffd640 10 monclient: dump: 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.541 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d4e59c640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 legacy=0x7f3d34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d4e59c640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 legacy=0x7f3d34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d4e59c640 1 --1- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 0x7f3d34004d00 :-1 s=OPENED pgs=369 cs=1 l=1).handle_message read tag failed 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d4e59c640 1 --1- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 0x7f3d34004d00 :-1 s=OPENED pgs=369 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 legacy=0x7f3d34004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d34002860 legacy=0x7f3d34004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.520+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d5013d6e0 con 0x7f3d3004c320 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d4e59c640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d3004c320 legacy=0x7f3d3004ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 client.164214.objecter ms_handle_connect 0x7f3d3004c320 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 client.164214.objecter resend_mon_ops 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502721416 0 0) 0x7f3d5013d6e0 con 0x7f3d3004c320 2024-08-19T09:57:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient(hunting): my global_id is 135030 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d5013b380 con 0x7f3d3004c320 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2408441823 0 0) 0x7f3d5013b380 con 0x7f3d3004c320 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d5013d6e0 con 0x7f3d3004c320 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d40004010 con 0x7f3d3004c320 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.522+0000 7f3d46ffd640 10 monclient: dump: 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3094479454 0 0) 0x7f3d5013d6e0 con 0x7f3d3004c320 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d48001320 con 0x7f3d3004c320 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3d40005b40 con 0x7f3d3004c320 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d3004c320 legacy=0x7f3d3004ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.523+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d3004c320 legacy=0x7f3d3004ab80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.529+0000 7f3d54baa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:13.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.529+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d3400c4c0 con 0x7f3d340090e0 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d4e59c640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d340090e0 legacy=0x7f3d34009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 client.164214.objecter ms_handle_connect 0x7f3d340090e0 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 client.164214.objecter resend_mon_ops 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827801096 0 0) 0x7f3d3400c4c0 con 0x7f3d340090e0 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient(hunting): my global_id is 135039 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d5013d6e0 con 0x7f3d340090e0 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 465512299 0 0) 0x7f3d5013d6e0 con 0x7f3d340090e0 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d3400c4c0 con 0x7f3d340090e0 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d40005cf0 con 0x7f3d340090e0 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 10 monclient: dump: 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1365560005 0 0) 0x7f3d3400c4c0 con 0x7f3d340090e0 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.532+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d34009c60 con 0x7f3d340090e0 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3222030371) 0x7f3d40007e10 con 0x7f3d340090e0 2024-08-19T09:57:13.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d46ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d46ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d340090e0 legacy=0x7f3d34009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d46ffd640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6790/0 conn(0x7f3d340090e0 legacy=0x7f3d34009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d4ed9d640 1 -- 172.21.15.1:0/820672098 reap_dead start 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.550 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 290592, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.552 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.553 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.554 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:13.556 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.557 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.560 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.561 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.564 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.565 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.569 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:13.570 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 10 librados: watch_flush enter 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 10 librados: watch_flush exit 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 20 client.164214.objecter shutdown clearing up homeless session... 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 10 client.164214.objecter successfully canceled tick 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3d30041110 legacy=0x7f3d300435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 10 monclient: shutdown 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 >> v1:172.21.15.1:6789/0 conn(0x7f3d500a6ba0 legacy=0x7f3d501d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 shutdown_connections 2024-08-19T09:57:13.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 >> 172.21.15.1:0/820672098 conn(0x7f3d500ac4b0 msgr2=0x7f3d501419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 shutdown_connections 2024-08-19T09:57:13.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 -- 172.21.15.1:0/820672098 wait complete. 2024-08-19T09:57:13.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.533+0000 7f3d54baa640 1 librados: shutdown 2024-08-19T09:57:13.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:13.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:57:13.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:57:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.674+0000 7fa5dd914640 10 monclient: get_monmap_and_config 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.674+0000 7fa5dd914640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.674+0000 7fa5dd914640 10 monclient: monmap: 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:13.675067+0000 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:13.675067+0000 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.674+0000 7fa5dd914640 1 Processor -- start 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.674+0000 7fa5dd914640 1 -- start start 2024-08-19T09:57:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.674+0000 7fa5dd914640 10 monclient: init 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient: _add_conns ranks=[1,2,0,6,3,8,5,7,4] 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient(hunting): picked mon.noname-f con 0x7fa5d80a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient(hunting): picked mon.noname-b con 0x7fa5d80694b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient(hunting): picked mon.noname-a con 0x7fa5d80a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d80a3c70 con 0x7fa5d80a46a0 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d80a33d0 con 0x7fa5d80a6830 2024-08-19T09:57:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d80a36b0 con 0x7fa5d80694b0 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5dd914640 10 monclient(hunting): authenticate will time out at 1089.751324s 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5d67fc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa5d80694b0 0x7fa5d80a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48122/0 (socket says 172.21.15.1:48122) 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5d77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa5d80a46a0 0x7fa5d8141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38236/0 (socket says 172.21.15.1:38236) 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.675+0000 7fa5d67fc640 1 -- 172.21.15.1:0/2947825902 learned_addr learned my addr 172.21.15.1:0/2947825902 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d67fc640 1 -- 172.21.15.1:0/2947825902 >> v1:172.21.15.1:6790/0 conn(0x7fa5d80694b0 legacy=0x7fa5d80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d77fe640 1 -- 172.21.15.1:0/2947825902 >> v1:172.21.15.1:6789/0 conn(0x7fa5d80a46a0 legacy=0x7fa5d8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d6ffd640 1 -- 172.21.15.1:0/2947825902 >> v1:172.21.15.82:6789/0 conn(0x7fa5d80a6830 legacy=0x7fa5d80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4250760961 0 0) 0x7fa5d80a3c70 con 0x7fa5d80a46a0 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): my global_id is 164223 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5c4003520 con 0x7fa5d80a46a0 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631595982 0 0) 0x7fa5d80a33d0 con 0x7fa5d80a6830 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient(hunting): my global_id is 164233 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d80a3c70 con 0x7fa5d80a6830 2024-08-19T09:57:13.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 588165272 0 0) 0x7fa5c4003520 con 0x7fa5d80a46a0 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 >> v1:172.21.15.1:6790/0 conn(0x7fa5d80694b0 legacy=0x7fa5d80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 >> v1:172.21.15.82:6789/0 conn(0x7fa5d80a6830 legacy=0x7fa5d80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 monclient: found mon.noname-a 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.677+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d806dbb0 con 0x7fa5d80a46a0 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5c8001e60 con 0x7fa5d80a46a0 2024-08-19T09:57:13.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 5 monclient: authenticate success, global_id 164223 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: dump: 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.686 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 471622111 0 0) 0x7fa5d80a3c70 con 0x7fa5d80a6830 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5c8004b60 con 0x7fa5d80a46a0 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 1 -- 172.21.15.1:0/2947825902 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5c80050d0 con 0x7fa5d80a46a0 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 4 set_mon_vals no callback set 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5d5ffb640 10 monclient: dump: 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.688 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 10 monclient: shutdown 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 1 -- 172.21.15.1:0/2947825902 >> v1:172.21.15.1:6789/0 conn(0x7fa5d80a46a0 legacy=0x7fa5d8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 1 -- 172.21.15.1:0/2947825902 shutdown_connections 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 1 -- 172.21.15.1:0/2947825902 >> 172.21.15.1:0/2947825902 conn(0x7fa5d80a9020 msgr2=0x7fa5d8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 1 -- 172.21.15.1:0/2947825902 shutdown_connections 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.678+0000 7fa5dd914640 1 -- 172.21.15.1:0/2947825902 wait complete. 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 10 monclient: monmap: 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.690 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 1 librados: starting msgr at 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 1 librados: starting objecter 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 1 Processor -- start 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 1 -- start start 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 1 librados: setting wanted keys 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 1 librados: calling monclient init 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.679+0000 7fa5dd914640 10 monclient: init 2024-08-19T09:57:13.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient: _add_conns ranks=[3,2,0,4,1,5,8,7,6] 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient(hunting): picked mon.noname-g con 0x7fa5d80694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient(hunting): picked mon.noname-b con 0x7fa5d80a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient(hunting): picked mon.noname-a con 0x7fa5d80a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d80a3120 con 0x7fa5d80a6830 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d8136950 con 0x7fa5d80a46a0 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d8138cb0 con 0x7fa5d80694b0 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5dd914640 10 monclient(hunting): authenticate will time out at 1089.756207s 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5d67fc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa5d80a46a0 0x7fa5d81356d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48124/0 (socket says 172.21.15.1:48124) 2024-08-19T09:57:13.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.680+0000 7fa5d67fc640 1 -- 172.21.15.1:0/687487401 learned_addr learned my addr 172.21.15.1:0/687487401 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5d67fc640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.1:6790/0 conn(0x7fa5d80a46a0 legacy=0x7fa5d81356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 client.?.objecter ms_handle_connect 0x7fa5d80a46a0 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5d77fe640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.1:6789/0 conn(0x7fa5d80a6830 legacy=0x7fa5d8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 client.?.objecter ms_handle_connect 0x7fa5d80a6830 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 client.?.objecter ms_handle_connect 0x7fa5d80694b0 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5d6ffd640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6790/0 conn(0x7fa5d80694b0 legacy=0x7fa5d8134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468835815 0 0) 0x7fa5d80a3120 con 0x7fa5d80a6830 2024-08-19T09:57:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): my global_id is 164232 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5b8004ee0 con 0x7fa5d80a6830 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473784553 0 0) 0x7fa5d8136950 con 0x7fa5d80a46a0 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): my global_id is 164288 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d80a3120 con 0x7fa5d80a46a0 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3731158175 0 0) 0x7fa5b8004ee0 con 0x7fa5d80a6830 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5d8136950 con 0x7fa5d80a6830 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5c8005050 con 0x7fa5d80a6830 2024-08-19T09:57:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): dump: 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686485712 0 0) 0x7fa5d8138cb0 con 0x7fa5d80694b0 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): my global_id is 135048 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5b8004ee0 con 0x7fa5d80694b0 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4253103433 0 0) 0x7fa5d80a3120 con 0x7fa5d80a46a0 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5d8138cb0 con 0x7fa5d80a46a0 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5c0001d00 con 0x7fa5d80a46a0 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.682+0000 7fa5cffff640 10 monclient(hunting): dump: 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 334813018 0 0) 0x7fa5d8136950 con 0x7fa5d80a6830 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6790/0 conn(0x7fa5d80694b0 legacy=0x7fa5d8134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.1:6790/0 conn(0x7fa5d80a46a0 legacy=0x7fa5d81356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 monclient: found mon.a 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d8137720 con 0x7fa5d80a6830 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 5 monclient: authenticate success, global_id 164232 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 10 monclient: _renew_subs 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d81d7410 con 0x7fa5d80a6830 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 10 client.164232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 10 monclient: _renew_subs 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5d81d77c0 con 0x7fa5d80a6830 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 1 librados: init done 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5c80020b0 con 0x7fa5d80a6830 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5c8001900 con 0x7fa5d80a6830 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5d57fa640 4 set_mon_vals no callback set 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5cffff640 10 monclient: dump: 2024-08-19T09:57:13.706 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.707 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.707 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.707 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.707 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.707 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.707 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.683+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5bc0065c0 con 0x7fa5bc002860 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa5c8011cc0 con 0x7fa5d80a6830 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7fa5c804e650 con 0x7fa5d80a6830 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5cffff640 10 client.164232.objecter ms_dispatch 0x7fa5d8000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5cffff640 3 client.164232.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5cffff640 3 client.164232.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5cffff640 20 client.164232.objecter dump_active .. 0 homeless 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.684+0000 7fa5d67fc640 10 monclient: get_auth_request con 0x7fa5b80416e0 auth_method 0 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5d6ffd640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6791/0 conn(0x7fa5bc002860 legacy=0x7fa5bc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 client.164232.objecter ms_handle_connect 0x7fa5bc002860 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 client.164232.objecter resend_mon_ops 2024-08-19T09:57:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124864574 0 0) 0x7fa5bc0065c0 con 0x7fa5bc002860 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient(hunting): my global_id is 27032 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d8136950 con 0x7fa5bc002860 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3872470223 0 0) 0x7fa5d8136950 con 0x7fa5bc002860 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5bc0065c0 con 0x7fa5bc002860 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5d0001bc0 con 0x7fa5bc002860 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 monclient: dump: 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.710 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.711 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 890112243 0 0) 0x7fa5bc0065c0 con 0x7fa5bc002860 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5bc00a160 con 0x7fa5bc002860 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.686+0000 7fa5cffff640 10 client.164232.objecter ms_handle_connect 0x7fa5b80416e0 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.687+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa5d00057c0 con 0x7fa5bc002860 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.687+0000 7fa5cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.687+0000 7fa5cffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.687+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6791/0 conn(0x7fa5bc002860 legacy=0x7fa5bc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.687+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6791/0 conn(0x7fa5bc002860 legacy=0x7fa5bc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.693+0000 7fa5dd914640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.693+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5bc009e50 con 0x7fa5bc009110 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5d6ffd640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6791/0 conn(0x7fa5bc009110 legacy=0x7fa5bc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 10 client.164232.objecter ms_handle_connect 0x7fa5bc009110 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 10 client.164232.objecter resend_mon_ops 2024-08-19T09:57:13.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108917372 0 0) 0x7fa5bc009e50 con 0x7fa5bc009110 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 10 monclient(hunting): my global_id is 27041 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.695+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5bc0065c0 con 0x7fa5bc009110 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 569501402 0 0) 0x7fa5bc0065c0 con 0x7fa5bc009110 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5bc009e50 con 0x7fa5bc009110 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa5d0003d80 con 0x7fa5bc009110 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 10 monclient: dump: 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.713 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.714 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 328262533 0 0) 0x7fa5bc009e50 con 0x7fa5bc009110 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5bc00c3a0 con 0x7fa5bc009110 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2953746677) 0x7fa5d0005a30 con 0x7fa5bc009110 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6791/0 conn(0x7fa5bc009110 legacy=0x7fa5bc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.696+0000 7fa5cffff640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6791/0 conn(0x7fa5bc009110 legacy=0x7fa5bc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.715 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 439961, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:13.716 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:13.720 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.721 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.722 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.723 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:13.725 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.726 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.732 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.733 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.735 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.736 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:13.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.737 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:13.737 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 10 librados: watch_flush enter 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 10 librados: watch_flush exit 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 20 client.164232.objecter shutdown clearing up homeless session... 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 10 client.164232.objecter successfully canceled tick 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa5b80416e0 legacy=0x7fa5b8043ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 10 monclient: shutdown 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 >> v1:172.21.15.1:6789/0 conn(0x7fa5d80a6830 legacy=0x7fa5d8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5d77fe640 1 -- 172.21.15.1:0/687487401 reap_dead start 2024-08-19T09:57:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5d77fe640 1 -- 172.21.15.1:0/687487401 reap_dead start 2024-08-19T09:57:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 shutdown_connections 2024-08-19T09:57:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 >> 172.21.15.1:0/687487401 conn(0x7fa5d80a9020 msgr2=0x7fa5d80a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 shutdown_connections 2024-08-19T09:57:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 -- 172.21.15.1:0/687487401 wait complete. 2024-08-19T09:57:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.697+0000 7fa5dd914640 1 librados: shutdown 2024-08-19T09:57:13.740 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:13.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:57:13.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:57:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.839+0000 7fedc2056640 10 monclient: get_monmap_and_config 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.839+0000 7fedc2056640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.839+0000 7fedc2056640 10 monclient: monmap: 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:13.840803+0000 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:13.840803+0000 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.842 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.840+0000 7fedc2056640 1 Processor -- start 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.840+0000 7fedc2056640 1 -- start start 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.840+0000 7fedc2056640 10 monclient: init 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient: _add_conns ranks=[6,1,8,4,5,0,3,2,7] 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient(hunting): picked mon.noname-d con 0x7fedbc0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient(hunting): picked mon.noname-f con 0x7fedbc0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient(hunting): picked mon.noname-e con 0x7fedbc13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc0a3b40 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc0a8130 con 0x7fedbc0a67c0 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc06ede0 con 0x7fedbc13f5c0 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedc2056640 10 monclient(hunting): authenticate will time out at 1089.917235s 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedbb7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fedbc0a67c0 0x7fedbc0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36982/0 (socket says 172.21.15.1:36982) 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.841+0000 7fedbb7fe640 1 -- 172.21.15.1:0/3342082268 learned_addr learned my addr 172.21.15.1:0/3342082268 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedbbfff640 1 -- 172.21.15.1:0/3342082268 >> v1:172.21.15.1:6793/0 conn(0x7fedbc13f5c0 legacy=0x7fedbc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedbb7fe640 1 -- 172.21.15.1:0/3342082268 >> v1:172.21.15.1:6792/0 conn(0x7fedbc0a67c0 legacy=0x7fedbc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990902241 0 0) 0x7fedbc0a8130 con 0x7fedbc0a67c0 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): my global_id is 164301 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda4003520 con 0x7fedbc0a67c0 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409883502 0 0) 0x7fedbc06ede0 con 0x7fedbc13f5c0 2024-08-19T09:57:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient(hunting): my global_id is 27845 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedbc0a8130 con 0x7fedbc13f5c0 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedbaffd640 1 -- 172.21.15.1:0/3342082268 >> v1:172.21.15.82:6789/0 conn(0x7fedbc0a6ba0 legacy=0x7fedbc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2729985365 0 0) 0x7feda4003520 con 0x7fedbc0a67c0 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 >> v1:172.21.15.1:6793/0 conn(0x7fedbc13f5c0 legacy=0x7fedbc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 >> v1:172.21.15.82:6789/0 conn(0x7fedbc0a6ba0 legacy=0x7fedbc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 monclient: found mon.noname-d 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc0a92a0 con 0x7fedbc0a67c0 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedac001ae0 con 0x7fedbc0a67c0 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedc2056640 5 monclient: authenticate success, global_id 164301 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedc2056640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedba7fc640 10 monclient: dump: 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.843+0000 7fedc2056640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedac0048a0 con 0x7fedbc0a67c0 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedba7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedba7fc640 1 -- 172.21.15.1:0/3342082268 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedac004e10 con 0x7fedbc0a67c0 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedba7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedba7fc640 10 monclient: dump: 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 4 set_mon_vals no callback set 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 10 monclient: shutdown 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 1 -- 172.21.15.1:0/3342082268 >> v1:172.21.15.1:6792/0 conn(0x7fedbc0a67c0 legacy=0x7fedbc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 1 -- 172.21.15.1:0/3342082268 shutdown_connections 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 1 -- 172.21.15.1:0/3342082268 >> 172.21.15.1:0/3342082268 conn(0x7fedbc0ac4b0 msgr2=0x7fedbc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 1 -- 172.21.15.1:0/3342082268 shutdown_connections 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.844+0000 7fedc2056640 1 -- 172.21.15.1:0/3342082268 wait complete. 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 10 monclient: build_initial_monmap 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 10 monclient: monmap: 2024-08-19T09:57:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 1 librados: starting msgr at 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 1 librados: starting objecter 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 1 Processor -- start 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 1 -- start start 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 1 librados: setting wanted keys 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 1 librados: calling monclient init 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.845+0000 7fedc2056640 10 monclient: init 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient: _add_conns ranks=[0,4,2,3,5,7,6,8,1] 2024-08-19T09:57:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient(hunting): picked mon.noname-a con 0x7fedbc0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient(hunting): picked mon.noname-c con 0x7fedbc0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient(hunting): picked mon.noname-b con 0x7fedbc13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc142c60 con 0x7fedbc0a67c0 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc0a1810 con 0x7fedbc13f5c0 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc13a9e0 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedc2056640 10 monclient(hunting): authenticate will time out at 1089.922383s 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedbb7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fedbc0a67c0 0x7fedbc1ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38260/0 (socket says 172.21.15.1:38260) 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedbaffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fedbc0a6ba0 0x7fedbc1d3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37554/0 (socket says 172.21.15.1:37554) 2024-08-19T09:57:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.846+0000 7fedbb7fe640 1 -- 172.21.15.1:0/817794973 learned_addr learned my addr 172.21.15.1:0/817794973 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:13.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7fedbaffd640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.1:6791/0 conn(0x7fedbc0a6ba0 legacy=0x7fedbc1d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7fedbbfff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.1:6790/0 conn(0x7fedbc13f5c0 legacy=0x7fedbc1d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 client.?.objecter ms_handle_connect 0x7fedbc0a6ba0 2024-08-19T09:57:13.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 client.?.objecter ms_handle_connect 0x7fedbc13f5c0 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7fedbb7fe640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.1:6789/0 conn(0x7fedbc0a67c0 legacy=0x7fedbc1ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 client.?.objecter ms_handle_connect 0x7fedbc0a67c0 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263929162 0 0) 0x7fedbc13a9e0 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): my global_id is 27319 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda8004eb0 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430800377 0 0) 0x7fedbc0a1810 con 0x7fedbc13f5c0 2024-08-19T09:57:13.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): my global_id is 164297 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedbc13a9e0 con 0x7fedbc13f5c0 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995585000 0 0) 0x7fedbc142c60 con 0x7fedbc0a67c0 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 10 monclient(hunting): my global_id is 164241 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.848+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedbc0a1810 con 0x7fedbc0a67c0 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2792471887 0 0) 0x7fedbc13a9e0 con 0x7fedbc13f5c0 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedbc142c60 con 0x7fedbc13f5c0 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedb4002030 con 0x7fedbc13f5c0 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): dump: 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.868 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 465161587 0 0) 0x7fedbc0a1810 con 0x7fedbc0a67c0 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedbc13a9e0 con 0x7fedbc0a67c0 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3861542765 0 0) 0x7feda8004eb0 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedbc0a1810 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedac004d90 con 0x7fedbc0a67c0 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): dump: 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.870 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedb0003610 con 0x7fedbc0a6ba0 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:13.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient(hunting): dump: 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3394936149 0 0) 0x7fedbc142c60 con 0x7fedbc13f5c0 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.1:6791/0 conn(0x7fedbc0a6ba0 legacy=0x7fedbc1d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.1:6789/0 conn(0x7fedbc0a67c0 legacy=0x7fedbc1ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 monclient: found mon.b 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc13acc0 con 0x7fedbc13f5c0 2024-08-19T09:57:13.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: _finish_auth 0 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 5 monclient: authenticate success, global_id 164297 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 10 monclient: _renew_subs 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc0a2370 con 0x7fedbc13f5c0 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 10 client.164297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 10 monclient: _renew_subs 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fedbc0a0f60 con 0x7fedbc13f5c0 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedc2056640 1 librados: init done 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedb4003c40 con 0x7fedbc13f5c0 2024-08-19T09:57:13.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedb40041b0 con 0x7fedbc13f5c0 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7fedb9ffb640 4 set_mon_vals no callback set 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.849+0000 7feda3fff640 10 monclient: dump: 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.875 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7fedc2056640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fed9c0065c0 con 0x7fed9c002860 2024-08-19T09:57:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fedb4004350 con 0x7fedbc13f5c0 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7fedb404e310 con 0x7fedbc13f5c0 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7feda3fff640 10 client.164297.objecter ms_dispatch 0x7fedbc000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7feda3fff640 3 client.164297.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7feda3fff640 3 client.164297.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7feda3fff640 20 client.164297.objecter dump_active .. 0 homeless 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.850+0000 7fedbaffd640 10 monclient: get_auth_request con 0x7feda8041910 auth_method 0 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7fedbb7fe640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6792/0 conn(0x7fed9c002860 legacy=0x7fed9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 10 client.164297.objecter ms_handle_connect 0x7fed9c002860 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 10 client.164297.objecter resend_mon_ops 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3544828375 0 0) 0x7fed9c0065c0 con 0x7fed9c002860 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 10 monclient(hunting): my global_id is 144520 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.852+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedbc142c60 con 0x7fed9c002860 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 10 client.164297.objecter ms_handle_connect 0x7feda8041910 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 460351134 0 0) 0x7fedbc142c60 con 0x7fed9c002860 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed9c0065c0 con 0x7fed9c002860 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedac001320 con 0x7fed9c002860 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 10 monclient: dump: 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2846139615 0 0) 0x7fed9c0065c0 con 0x7fed9c002860 2024-08-19T09:57:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed9c00a160 con 0x7fed9c002860 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fedac005490 con 0x7fed9c002860 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6792/0 conn(0x7fed9c002860 legacy=0x7fed9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.853+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6792/0 conn(0x7fed9c002860 legacy=0x7fed9c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.859+0000 7fedc2056640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.859+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fed9c009e70 con 0x7fed9c009130 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7fedbb7fe640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6792/0 conn(0x7fed9c009130 legacy=0x7fed9c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 client.164297.objecter ms_handle_connect 0x7fed9c009130 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 client.164297.objecter resend_mon_ops 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388931445 0 0) 0x7fed9c009e70 con 0x7fed9c009130 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:13.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient(hunting): my global_id is 144529 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed9c0065c0 con 0x7fed9c009130 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1454383057 0 0) 0x7fed9c0065c0 con 0x7fed9c009130 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed9c009e70 con 0x7fed9c009130 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fedac005c70 con 0x7fed9c009130 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 10 monclient: dump: 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:13.881 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 95848399 0 0) 0x7fed9c009e70 con 0x7fed9c009130 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.862+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed9c00c410 con 0x7fed9c009130 2024-08-19T09:57:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7851 (unknown 0 0 350975525) 0x7fedac005f80 con 0x7fed9c009130 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7feda3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7feda3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6792/0 conn(0x7fed9c009130 legacy=0x7fed9c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7feda3fff640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6792/0 conn(0x7fed9c009130 legacy=0x7fed9c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.883 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 158242, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:57:13.884 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.885 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.886 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:13.888 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.889 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.890 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.892 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:13.896 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.897 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.899 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.900 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.901 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.902 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.903 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:13.904 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 10 librados: watch_flush enter 2024-08-19T09:57:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 10 librados: watch_flush exit 2024-08-19T09:57:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 20 client.164297.objecter shutdown clearing up homeless session... 2024-08-19T09:57:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 10 client.164297.objecter successfully canceled tick 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.82:6812/2549104415 conn(0x7feda8041910 legacy=0x7feda8043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 10 monclient: shutdown 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 >> v1:172.21.15.1:6790/0 conn(0x7fedbc13f5c0 legacy=0x7fedbc1d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedbbfff640 1 -- 172.21.15.1:0/817794973 reap_dead start 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedbbfff640 1 -- 172.21.15.1:0/817794973 reap_dead start 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 shutdown_connections 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 >> 172.21.15.1:0/817794973 conn(0x7fedbc0ac4b0 msgr2=0x7fedbc0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 shutdown_connections 2024-08-19T09:57:13.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 -- 172.21.15.1:0/817794973 wait complete. 2024-08-19T09:57:13.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:13.863+0000 7fedc2056640 1 librados: shutdown 2024-08-19T09:57:13.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:57:13.908 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['f', 'e', 'd', 'a'] 2024-08-19T09:57:13.909 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['g'] 2024-08-19T09:57:13.909 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-08-19T09:57:13.909 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-08-19T09:57:13.909 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T09:57:13.909 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:57:13.921 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T09:57:13.922 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-08-19T09:57:13.922 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-08-19T09:57:13.922 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T09:57:13.922 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:57:13.935 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T09:57:13.935 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T09:57:13.935 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T09:57:13.935 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T09:57:13.935 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:57:13.985 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T09:57:13.985 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T09:57:13.985 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T09:57:13.985 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T09:57:13.985 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:57:14.036 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T09:57:14.037 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-08-19T09:57:14.037 INFO:tasks.ceph.mon.g:Sent signal 19 2024-08-19T09:57:14.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:57:29.038 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-08-19T09:57:29.039 INFO:tasks.ceph.mon.g:Sent signal 18 2024-08-19T09:57:29.039 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-08-19T09:57:29.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.149+0000 7f5e0d3c5640 10 monclient: get_monmap_and_config 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.149+0000 7f5e0d3c5640 10 monclient: build_initial_monmap 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.149+0000 7f5e0d3c5640 10 monclient: monmap: 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:29.150072+0000 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:29.150072+0000 2024-08-19T09:57:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:29.151 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.149+0000 7f5e0d3c5640 1 Processor -- start 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.149+0000 7f5e0d3c5640 1 -- start start 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.149+0000 7f5e0d3c5640 10 monclient: init 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient: _add_conns ranks=[8,1,6,3,2,0,7,5,4] 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient(hunting): picked mon.noname-e con 0x7f5e080a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient(hunting): picked mon.noname-f con 0x7f5e080a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:29.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient(hunting): picked mon.noname-d con 0x7f5e0813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080a3b40 con 0x7f5e080a6ba0 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080a8130 con 0x7f5e0813f5c0 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0806ede0 con 0x7f5e080a67c0 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e0d3c5640 10 monclient(hunting): authenticate will time out at 1105.226323s 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e077fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f5e0813f5c0 legacy=0x7f5e081419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e06ffd640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f5e080a67c0 legacy=0x7f5e080a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.150+0000 7f5e067fc640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f5e080a6ba0 legacy=0x7f5e080a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.351+0000 7f5e077fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f5e0813f5c0 legacy=0x7f5e081419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.351+0000 7f5e06ffd640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f5e080a67c0 legacy=0x7f5e080a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.351+0000 7f5e067fc640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f5e080a6ba0 legacy=0x7f5e080a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:29.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.752+0000 7f5e06ffd640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f5e080a67c0 legacy=0x7f5e080a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:29.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.752+0000 7f5e077fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f5e0813f5c0 legacy=0x7f5e081419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:29.753+0000 7f5e067fc640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f5e080a6ba0 legacy=0x7f5e080a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:30.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:30.553+0000 7f5e06ffd640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f5e080a67c0 legacy=0x7f5e080a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:30.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:30.553+0000 7f5e077fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f5e0813f5c0 legacy=0x7f5e081419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:30.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:30.554+0000 7f5e067fc640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f5e080a6ba0 legacy=0x7f5e080a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:30.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:30.554+0000 7f5e05ffb640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): tick 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 monclient(hunting): continuing hunt 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f5e080a67c0 legacy=0x7f5e080a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f5e0813f5c0 legacy=0x7f5e081419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f5e080a6ba0 legacy=0x7f5e080a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient: _add_conns ranks=[0,5,3,4,7,2] 2024-08-19T09:57:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f5df4003770 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-h con 0x7f5df4007400 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-g con 0x7f5df400b190 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080a92a0 con 0x7f5df4003770 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080a3b40 con 0x7f5df400b190 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080a8130 con 0x7f5df4007400 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e04ff9640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e06ffd640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f5df4003770 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:32.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e067fc640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 0x7f5df40098c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:47102/0 (socket says 172.21.15.1:47102) 2024-08-19T09:57:32.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.150+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 learned_addr learned my addr 172.21.15.1:0/490602603 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:32.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.153+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 legacy=0x7f5df40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:32.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.153+0000 7f5e077fe640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6790/0 conn(0x7f5df400b190 legacy=0x7f5df400d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.351+0000 7f5e06ffd640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6789/0 conn(0x7f5df4003770 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:32.752+0000 7f5e06ffd640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6789/0 conn(0x7f5df4003770 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:33.553+0000 7f5e06ffd640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6789/0 conn(0x7f5df4003770 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:33.780+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 legacy=0x7f5df40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T09:57:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:33.780+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 legacy=0x7f5df40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:57:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:33.780+0000 7f5e067fc640 1 --1- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 0x7f5df40098c0 :-1 s=OPENED pgs=3436 cs=1 l=1).handle_message read tag failed 2024-08-19T09:57:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:33.780+0000 7f5e067fc640 1 --1- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 0x7f5df40098c0 :-1 s=OPENED pgs=3436 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:57:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:33.780+0000 7f5e05ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6791/0 2024-08-19T09:57:35.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): tick 2024-08-19T09:57:35.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:57:35.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 monclient(hunting): continuing hunt 2024-08-19T09:57:35.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:57:35.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4007400 legacy=0x7f5df40098c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:35.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6790/0 conn(0x7f5df400b190 legacy=0x7f5df400d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6789/0 conn(0x7f5df4003770 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient: _add_conns ranks=[4,7,2] 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e077fe640 1 -- 172.21.15.1:0/490602603 reap_dead start 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f5df400ffe0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-i con 0x7f5df400a090 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f5df400a4d0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0806ede0 con 0x7f5df400a4d0 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e077fe640 1 -- 172.21.15.1:0/490602603 reap_dead start 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080a92a0 con 0x7f5df400ffe0 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5df400a950 con 0x7f5df400a090 2024-08-19T09:57:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.151+0000 7f5e04ff9640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:35.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.153+0000 7f5e06ffd640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6791/0 conn(0x7f5df400ffe0 legacy=0x7f5df40103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:35.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.153+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6792/0 conn(0x7f5df400a090 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:35.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.153+0000 7f5e077fe640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6790/0 conn(0x7f5df400a4d0 legacy=0x7f5df40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.648+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6792/0 conn(0x7f5df400a090 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T09:57:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.648+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6792/0 conn(0x7f5df400a090 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:57:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.648+0000 7f5e067fc640 1 --1- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6792/0 conn(0x7f5df400a090 0x7f5df4005bc0 :-1 s=OPENED pgs=296 cs=1 l=1).handle_message read tag failed 2024-08-19T09:57:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.648+0000 7f5e067fc640 1 --1- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6792/0 conn(0x7f5df400a090 0x7f5df4005bc0 :-1 s=OPENED pgs=296 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:57:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:35.648+0000 7f5e05ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6792/0 2024-08-19T09:57:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): tick 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 monclient(hunting): continuing hunt 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6792/0 conn(0x7f5df400a090 legacy=0x7f5df4005bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6791/0 conn(0x7f5df400ffe0 legacy=0x7f5df40103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6790/0 conn(0x7f5df400a4d0 legacy=0x7f5df40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient: _add_conns ranks=[8,1,5,3,2,0,4,7,6] 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-e con 0x7f5df400de20 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-f con 0x7f5df400e200 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): picked mon.noname-h con 0x7f5df4018160 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5df400abc0 con 0x7f5df400e200 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5df401ce40 con 0x7f5df4018160 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5df401e0c0 con 0x7f5df400de20 2024-08-19T09:57:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e04ff9640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e06ffd640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6793/0 conn(0x7f5df400de20 legacy=0x7f5df400d580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.151+0000 7f5e067fc640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6789/0 conn(0x7f5df400e200 legacy=0x7f5df400e680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.154+0000 7f5e077fe640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4018160 legacy=0x7f5df401a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.232+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1753143609 0 0) 0x7f5df401ce40 con 0x7f5df4018160 2024-08-19T09:57:38.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.232+0000 7f5e05ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.232+0000 7f5e05ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.232+0000 7f5e05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.232+0000 7f5e05ffb640 10 monclient(hunting): my global_id is 27050 2024-08-19T09:57:38.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.232+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5df0003520 con 0x7f5df4018160 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1354937 0 0) 0x7f5df0003520 con 0x7f5df4018160 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.1:6793/0 conn(0x7f5df400de20 legacy=0x7f5df400d580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6789/0 conn(0x7f5df400e200 legacy=0x7f5df400e680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e077fe640 1 -- 172.21.15.1:0/490602603 reap_dead start 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 1 monclient: found mon.noname-h 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df401f340 con 0x7f5df4018160 2024-08-19T09:57:38.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5e000046d0 con 0x7f5df4018160 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e0d3c5640 5 monclient: authenticate success, global_id 27050 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e05ffb640 10 monclient: dump: 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.236 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.233+0000 7f5e0d3c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e000048e0 con 0x7f5df4018160 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e05ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e05ffb640 1 -- 172.21.15.1:0/490602603 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5e00004dd0 con 0x7f5df4018160 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e05ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e05ffb640 10 monclient: dump: 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.241 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e0d3c5640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.234+0000 7f5e0d3c5640 4 set_mon_vals no callback set 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 10 monclient: shutdown 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/490602603 >> v1:172.21.15.82:6791/0 conn(0x7f5df4018160 legacy=0x7f5df401a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/490602603 shutdown_connections 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/490602603 >> 172.21.15.1:0/490602603 conn(0x7f5e080ac4b0 msgr2=0x7f5e080a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/490602603 shutdown_connections 2024-08-19T09:57:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.235+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/490602603 wait complete. 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.236+0000 7f5e0d3c5640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.236+0000 7f5e0d3c5640 10 monclient: monmap: 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.236+0000 7f5e0d3c5640 1 librados: starting msgr at 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.236+0000 7f5e0d3c5640 1 librados: starting objecter 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 1 Processor -- start 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 1 -- start start 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 1 librados: setting wanted keys 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 1 librados: calling monclient init 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 10 monclient: init 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 10 monclient: _add_conns ranks=[1,0,7,3,8,6,4,2,5] 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 10 monclient(hunting): picked mon.noname-f con 0x7f5df4018160 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.237+0000 7f5e0d3c5640 10 monclient(hunting): picked mon.noname-a con 0x7f5e08123d60 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e0d3c5640 10 monclient(hunting): picked mon.noname-i con 0x7f5e081d4350 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e0d3c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e08125560 con 0x7f5e08123d60 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e06ffd640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f5df4018160 legacy=0x7f5e08122410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e067fc640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f5e08123d60 legacy=0x7f5e08122b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e0d3c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e08123230 con 0x7f5df4018160 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e0d3c5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080aa5d0 con 0x7f5e081d4350 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e0d3c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e0d3c5640 10 monclient(hunting): authenticate will time out at 1114.314091s 2024-08-19T09:57:38.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e077fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f5e081d4350 0x7f5e081d6740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:59030/0 (socket says 172.21.15.1:59030) 2024-08-19T09:57:38.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.238+0000 7f5e077fe640 1 -- 172.21.15.1:0/855399294 learned_addr learned my addr 172.21.15.1:0/855399294 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5e077fe640 1 -- 172.21.15.1:0/855399294 >> v1:172.21.15.82:6792/0 conn(0x7f5e081d4350 legacy=0x7f5e081d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 10 client.?.objecter ms_handle_connect 0x7f5e081d4350 2024-08-19T09:57:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 619634890 0 0) 0x7f5e080aa5d0 con 0x7f5e081d4350 2024-08-19T09:57:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 10 monclient(hunting): my global_id is 144538 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.240+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5df400abc0 con 0x7f5e081d4350 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4103707182 0 0) 0x7f5df400abc0 con 0x7f5e081d4350 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e080aa5d0 con 0x7f5e081d4350 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5e00004dd0 con 0x7f5e081d4350 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient(hunting): dump: 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.251 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1621365797 0 0) 0x7f5e080aa5d0 con 0x7f5e081d4350 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 >> v1:172.21.15.82:6789/0 conn(0x7f5df4018160 legacy=0x7f5e08122410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 >> v1:172.21.15.1:6789/0 conn(0x7f5e08123d60 legacy=0x7f5e08122b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 monclient: found mon.i 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e080a1f00 con 0x7f5e081d4350 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 5 monclient: authenticate success, global_id 144538 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 10 monclient: _renew_subs 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5e080a1670 con 0x7f5e081d4350 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 10 client.144538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 10 monclient: _renew_subs 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f5e081d7e60 con 0x7f5e081d4350 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e0d3c5640 1 librados: init done 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e000047d0 con 0x7f5e081d4350 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5e00005560 con 0x7f5e081d4350 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5e057fa640 4 set_mon_vals no callback set 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.241+0000 7f5def7fe640 10 monclient: dump: 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.242+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5e000119c0 con 0x7f5e081d4350 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.242+0000 7f5e0d3c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.242+0000 7f5e0d3c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.242+0000 7f5e0d3c5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.242+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de8005080 con 0x7f5e081d4350 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.243+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7f5e0004e380 con 0x7f5e081d4350 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.243+0000 7f5def7fe640 10 client.144538.objecter ms_dispatch 0x7f5e08000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.243+0000 7f5def7fe640 3 client.144538.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.243+0000 7f5def7fe640 3 client.144538.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:38.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.243+0000 7f5def7fe640 20 client.144538.objecter dump_active .. 0 homeless 2024-08-19T09:57:38.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.243+0000 7f5e06ffd640 10 monclient: get_auth_request con 0x7f5df4042bd0 auth_method 0 2024-08-19T09:57:38.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.245+0000 7f5def7fe640 10 client.144538.objecter ms_handle_connect 0x7f5df4042bd0 2024-08-19T09:57:38.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.248+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f5e00017800 con 0x7f5e081d4350 2024-08-19T09:57:38.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.248+0000 7f5def7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.248+0000 7f5def7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:38.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.504+0000 7f5e0d3c5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:57:38.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.504+0000 7f5e0d3c5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:57:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.504+0000 7f5e0d3c5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:57:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.504+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5de8005370 con 0x7f5e081d4350 2024-08-19T09:57:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.505+0000 7f5def7fe640 1 -- 172.21.15.1:0/855399294 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2678 (unknown 254746657 0 4252877966) 0x7f5e00017100 con 0x7f5e081d4350 2024-08-19T09:57:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.505+0000 7f5def7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:57:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.505+0000 7f5def7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:38.507 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":76,"quorum":[2,3,4,5,7],"quorum_names":["b","g","c","h","i"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:57:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 10 librados: watch_flush enter 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 10 librados: watch_flush exit 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 20 client.144538.objecter shutdown clearing up homeless session... 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 10 client.144538.objecter successfully canceled tick 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5df4042bd0 legacy=0x7f5df4045090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 10 monclient: shutdown 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 >> v1:172.21.15.82:6792/0 conn(0x7f5e081d4350 legacy=0x7f5e081d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 shutdown_connections 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 >> 172.21.15.1:0/855399294 conn(0x7f5e080ac4b0 msgr2=0x7f5e0813c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 shutdown_connections 2024-08-19T09:57:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 -- 172.21.15.1:0/855399294 wait complete. 2024-08-19T09:57:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.507+0000 7f5e0d3c5640 1 librados: shutdown 2024-08-19T09:57:38.519 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-08-19T09:57:38.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-08-19T09:57:38.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:57:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.619+0000 7faed4141640 10 monclient: get_monmap_and_config 2024-08-19T09:57:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.619+0000 7faed4141640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.619+0000 7faed4141640 10 monclient: monmap: 2024-08-19T09:57:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:38.620138+0000 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:38.620138+0000 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.622 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.619+0000 7faed4141640 1 Processor -- start 2024-08-19T09:57:38.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.619+0000 7faed4141640 1 -- start start 2024-08-19T09:57:38.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.619+0000 7faed4141640 10 monclient: init 2024-08-19T09:57:38.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient: _add_conns ranks=[0,2,4,8,1,3,5,6,7] 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient(hunting): picked mon.noname-a con 0x7faecc0a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient(hunting): picked mon.noname-b con 0x7faecc0694b0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient(hunting): picked mon.noname-c con 0x7faecc0a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc0a3c70 con 0x7faecc0a6830 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc0a33d0 con 0x7faecc0694b0 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc0a36b0 con 0x7faecc0a46a0 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed4141640 10 monclient(hunting): authenticate will time out at 1114.696497s 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed1edd640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7faecc0a6830 legacy=0x7faecc0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed26de640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7faecc0a46a0 0x7faecc141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54228/0 (socket says 172.21.15.1:54228) 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed16dc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7faecc0694b0 0x7faecc0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33118/0 (socket says 172.21.15.1:33118) 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.620+0000 7faed26de640 1 -- 172.21.15.1:0/3661364522 learned_addr learned my addr 172.21.15.1:0/3661364522 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed26de640 1 -- 172.21.15.1:0/3661364522 >> v1:172.21.15.1:6791/0 conn(0x7faecc0a46a0 legacy=0x7faecc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed16dc640 1 -- 172.21.15.1:0/3661364522 >> v1:172.21.15.1:6790/0 conn(0x7faecc0694b0 legacy=0x7faecc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737326091 0 0) 0x7faecc0a33d0 con 0x7faecc0694b0 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed0edb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed0edb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed0edb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed0edb640 10 monclient(hunting): my global_id is 164306 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.622+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeb4003520 con 0x7faecc0694b0 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039332355 0 0) 0x7faecc0a36b0 con 0x7faecc0a46a0 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient(hunting): my global_id is 27328 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faecc0a33d0 con 0x7faecc0a46a0 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2344346456 0 0) 0x7faecc0a33d0 con 0x7faecc0a46a0 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 >> v1:172.21.15.1:6790/0 conn(0x7faecc0694b0 legacy=0x7faecc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 >> v1:172.21.15.1:6789/0 conn(0x7faecc0a6830 legacy=0x7faecc0a6c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 monclient: found mon.noname-c 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc06dbb0 con 0x7faecc0a46a0 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 5 monclient: authenticate success, global_id 27328 2024-08-19T09:57:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faebc001e40 con 0x7faecc0a46a0 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: dump: 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faebc004c20 con 0x7faecc0a46a0 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 1 -- 172.21.15.1:0/3661364522 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faebc005190 con 0x7faecc0a46a0 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 4 set_mon_vals no callback set 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed0edb640 10 monclient: dump: 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 10 monclient: shutdown 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 1 -- 172.21.15.1:0/3661364522 >> v1:172.21.15.1:6791/0 conn(0x7faecc0a46a0 legacy=0x7faecc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 1 -- 172.21.15.1:0/3661364522 shutdown_connections 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 1 -- 172.21.15.1:0/3661364522 >> 172.21.15.1:0/3661364522 conn(0x7faecc0a9020 msgr2=0x7faecc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.623+0000 7faed4141640 1 -- 172.21.15.1:0/3661364522 shutdown_connections 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 -- 172.21.15.1:0/3661364522 wait complete. 2024-08-19T09:57:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 10 monclient: monmap: 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 librados: starting msgr at 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 librados: starting objecter 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 Processor -- start 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 -- start start 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 librados: setting wanted keys 2024-08-19T09:57:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 1 librados: calling monclient init 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.624+0000 7faed4141640 10 monclient: init 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient: _add_conns ranks=[4,3,6,8,7,1,2,5,0] 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient(hunting): picked mon.noname-c con 0x7faecc0694b0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient(hunting): picked mon.noname-g con 0x7faecc0a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient(hunting): picked mon.noname-d con 0x7faecc0a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc13baf0 con 0x7faecc0a46a0 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc0a9f50 con 0x7faecc0694b0 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc13b810 con 0x7faecc0a6830 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed4141640 10 monclient(hunting): authenticate will time out at 1114.701443s 2024-08-19T09:57:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed26de640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7faecc0a6830 legacy=0x7faecc1d6450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed1edd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7faecc0694b0 0x7faecc141f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54236/0 (socket says 172.21.15.1:54236) 2024-08-19T09:57:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.625+0000 7faed1edd640 1 -- 172.21.15.1:0/2211986691 learned_addr learned my addr 172.21.15.1:0/2211986691 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faed1edd640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6791/0 conn(0x7faecc0694b0 legacy=0x7faecc141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 client.?.objecter ms_handle_connect 0x7faecc0694b0 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508599027 0 0) 0x7faecc0a9f50 con 0x7faecc0694b0 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): my global_id is 27337 2024-08-19T09:57:38.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeb8004eb0 con 0x7faecc0694b0 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faed16dc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.82:6790/0 conn(0x7faecc0a46a0 legacy=0x7faecc1d2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 client.?.objecter ms_handle_connect 0x7faecc0a46a0 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 858531634 0 0) 0x7faeb8004eb0 con 0x7faecc0694b0 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faecc0a9f50 con 0x7faecc0694b0 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faec4004760 con 0x7faecc0694b0 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.627+0000 7faeca7fc640 10 monclient(hunting): dump: 2024-08-19T09:57:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2521638842 0 0) 0x7faecc13baf0 con 0x7faecc0a46a0 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient(hunting): my global_id is 135057 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeb8004eb0 con 0x7faecc0a46a0 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3743703871 0 0) 0x7faecc0a9f50 con 0x7faecc0694b0 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6792/0 conn(0x7faecc0a6830 legacy=0x7faecc1d6450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.82:6790/0 conn(0x7faecc0a46a0 legacy=0x7faecc1d2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 monclient: found mon.c 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc13a5c0 con 0x7faecc0694b0 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 5 monclient: authenticate success, global_id 27337 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 10 monclient: _renew_subs 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faecc13a2f0 con 0x7faecc0694b0 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 10 client.27337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 10 monclient: _renew_subs 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7faecc142680 con 0x7faecc0694b0 2024-08-19T09:57:38.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faed4141640 1 librados: init done 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faec4004980 con 0x7faecc0694b0 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faec4004ef0 con 0x7faecc0694b0 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faecbfff640 4 set_mon_vals no callback set 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.628+0000 7faeca7fc640 10 monclient: dump: 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faed4141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faeac0065c0 con 0x7faeac002860 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7faec40051c0 con 0x7faecc0694b0 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7faec404e3d0 con 0x7faecc0694b0 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faeca7fc640 10 client.27337.objecter ms_dispatch 0x7faecc000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faeca7fc640 3 client.27337.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faeca7fc640 3 client.27337.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faeca7fc640 20 client.27337.objecter dump_active .. 0 homeless 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.629+0000 7faed26de640 10 monclient: get_auth_request con 0x7faeb8040ed0 auth_method 0 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faed16dc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6790/0 conn(0x7faeac002860 legacy=0x7faeac004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 client.27337.objecter ms_handle_connect 0x7faeac002860 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 client.27337.objecter resend_mon_ops 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1683950139 0 0) 0x7faeac0065c0 con 0x7faeac002860 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient(hunting): my global_id is 164315 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faecc0a9f50 con 0x7faeac002860 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1428821706 0 0) 0x7faecc0a9f50 con 0x7faeac002860 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faeac0065c0 con 0x7faeac002860 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faec0001b30 con 0x7faeac002860 2024-08-19T09:57:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 monclient: dump: 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3640919246 0 0) 0x7faeac0065c0 con 0x7faeac002860 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faeac00a160 con 0x7faeac002860 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.631+0000 7faeca7fc640 10 client.27337.objecter ms_handle_connect 0x7faeb8040ed0 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.632+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7faec0006840 con 0x7faeac002860 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.632+0000 7faeca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.632+0000 7faeca7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.632+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6790/0 conn(0x7faeac002860 legacy=0x7faeac004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.632+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6790/0 conn(0x7faeac002860 legacy=0x7faeac004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.638+0000 7faed4141640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.638+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faeac009e70 con 0x7faeac009130 2024-08-19T09:57:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 client.27337.objecter ms_handle_connect 0x7faeac009130 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 client.27337.objecter resend_mon_ops 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faed16dc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6790/0 conn(0x7faeac009130 legacy=0x7faeac0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3324605462 0 0) 0x7faeac009e70 con 0x7faeac009130 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient(hunting): my global_id is 164324 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeac0065c0 con 0x7faeac009130 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1288098175 0 0) 0x7faeac0065c0 con 0x7faeac009130 2024-08-19T09:57:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faeac009e70 con 0x7faeac009130 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faec0004f10 con 0x7faeac009130 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 10 monclient: dump: 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 417629682 0 0) 0x7faeac009e70 con 0x7faeac009130 2024-08-19T09:57:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.640+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faeac00c3f0 con 0x7faeac009130 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7474 (unknown 0 0 4022137758) 0x7faec0006390 con 0x7faeac009130 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faeca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faeca7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6790/0 conn(0x7faeac009130 legacy=0x7faeac0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faeca7fc640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6790/0 conn(0x7faeac009130 legacy=0x7faeac0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:38.663 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 45455, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:57:38.664 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.665 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:38.667 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:38.668 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.669 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:38.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:38.671 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:38.675 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:38.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.678 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.680 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:38.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:38.683 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.686 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:38.687 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 10 librados: watch_flush enter 2024-08-19T09:57:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 10 librados: watch_flush exit 2024-08-19T09:57:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 20 client.27337.objecter shutdown clearing up homeless session... 2024-08-19T09:57:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 10 client.27337.objecter successfully canceled tick 2024-08-19T09:57:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.82:6812/2549104415 conn(0x7faeb8040ed0 legacy=0x7faeb8043390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 10 monclient: shutdown 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 >> v1:172.21.15.1:6791/0 conn(0x7faecc0694b0 legacy=0x7faecc141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed26de640 1 -- 172.21.15.1:0/2211986691 reap_dead start 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed26de640 1 -- 172.21.15.1:0/2211986691 reap_dead start 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 shutdown_connections 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 >> 172.21.15.1:0/2211986691 conn(0x7faecc0a9020 msgr2=0x7faecc0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 shutdown_connections 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 -- 172.21.15.1:0/2211986691 wait complete. 2024-08-19T09:57:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.641+0000 7faed4141640 1 librados: shutdown 2024-08-19T09:57:38.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:57:38.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-08-19T09:57:38.691 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:57:38.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.790+0000 7fe69913a640 10 monclient: get_monmap_and_config 2024-08-19T09:57:38.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.790+0000 7fe69913a640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.790+0000 7fe69913a640 10 monclient: monmap: 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:38.791484+0000 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:38.791484+0000 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.792 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.793 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.793 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 1 Processor -- start 2024-08-19T09:57:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 1 -- start start 2024-08-19T09:57:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient: init 2024-08-19T09:57:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient: _add_conns ranks=[2,0,1,8,5,3,7,6,4] 2024-08-19T09:57:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient(hunting): picked mon.noname-b con 0x7fe6940a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient(hunting): picked mon.noname-a con 0x7fe6940a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient(hunting): picked mon.noname-f con 0x7fe69413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6940a3b40 con 0x7fe6940a6ba0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6940a8130 con 0x7fe69413f5c0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69406ede0 con 0x7fe6940a67c0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69913a640 10 monclient(hunting): authenticate will time out at 1114.867692s 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69259c640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fe6940a6ba0 legacy=0x7fe6940a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe69359e640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7fe69413f5c0 legacy=0x7fe6941419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe692d9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fe6940a67c0 0x7fe6940a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33134/0 (socket says 172.21.15.1:33134) 2024-08-19T09:57:38.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.791+0000 7fe692d9d640 1 -- 172.21.15.1:0/3987992178 learned_addr learned my addr 172.21.15.1:0/3987992178 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.793+0000 7fe692d9d640 1 -- 172.21.15.1:0/3987992178 >> v1:172.21.15.1:6790/0 conn(0x7fe6940a67c0 legacy=0x7fe6940a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 569838925 0 0) 0x7fe69406ede0 con 0x7fe6940a67c0 2024-08-19T09:57:38.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient(hunting): my global_id is 164333 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe67c003520 con 0x7fe6940a67c0 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1679798782 0 0) 0x7fe67c003520 con 0x7fe6940a67c0 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 >> v1:172.21.15.82:6789/0 conn(0x7fe69413f5c0 legacy=0x7fe6941419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 >> v1:172.21.15.1:6789/0 conn(0x7fe6940a6ba0 legacy=0x7fe6940a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 monclient: found mon.noname-b 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6940a92a0 con 0x7fe6940a67c0 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 5 monclient: authenticate success, global_id 164333 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe684001ad0 con 0x7fe6940a67c0 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: dump: 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6840048a0 con 0x7fe6940a67c0 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 1 -- 172.21.15.1:0/3987992178 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe684004e10 con 0x7fe6940a67c0 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe691d9b640 10 monclient: dump: 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 4 set_mon_vals no callback set 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 10 monclient: shutdown 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.794+0000 7fe69913a640 1 -- 172.21.15.1:0/3987992178 >> v1:172.21.15.1:6790/0 conn(0x7fe6940a67c0 legacy=0x7fe6940a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 1 -- 172.21.15.1:0/3987992178 shutdown_connections 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 1 -- 172.21.15.1:0/3987992178 >> 172.21.15.1:0/3987992178 conn(0x7fe6940ac4b0 msgr2=0x7fe6940a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 1 -- 172.21.15.1:0/3987992178 shutdown_connections 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 1 -- 172.21.15.1:0/3987992178 wait complete. 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 10 monclient: monmap: 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.805 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 1 librados: starting msgr at 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.795+0000 7fe69913a640 1 librados: starting objecter 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 Processor -- start 2024-08-19T09:57:38.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 -- start start 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 librados: setting wanted keys 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 librados: calling monclient init 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient: init 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient: _add_conns ranks=[6,1,5,0,2,4,7,3,8] 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient(hunting): picked mon.noname-d con 0x7fe6940a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient(hunting): picked mon.noname-f con 0x7fe6940a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient(hunting): picked mon.noname-h con 0x7fe69413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69413a430 con 0x7fe6940a6ba0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69413a710 con 0x7fe69413f5c0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe692d9d640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7fe6940a67c0 legacy=0x7fe69413ec50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6940a2150 con 0x7fe6940a67c0 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69913a640 10 monclient(hunting): authenticate will time out at 1114.872782s 2024-08-19T09:57:38.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.796+0000 7fe69259c640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7fe6940a6ba0 legacy=0x7fe6941d2ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:38.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.797+0000 7fe69359e640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fe69413f5c0 0x7fe6941d62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:57282/0 (socket says 172.21.15.1:57282) 2024-08-19T09:57:38.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.797+0000 7fe69359e640 1 -- 172.21.15.1:0/2364793891 learned_addr learned my addr 172.21.15.1:0/2364793891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe69359e640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.82:6791/0 conn(0x7fe69413f5c0 legacy=0x7fe6941d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 client.?.objecter ms_handle_connect 0x7fe69413f5c0 2024-08-19T09:57:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380506267 0 0) 0x7fe69413a710 con 0x7fe69413f5c0 2024-08-19T09:57:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): my global_id is 27059 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe680004eb0 con 0x7fe69413f5c0 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3806330211 0 0) 0x7fe680004eb0 con 0x7fe69413f5c0 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe69413a710 con 0x7fe69413f5c0 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe68c0049d0 con 0x7fe69413f5c0 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): dump: 2024-08-19T09:57:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 995903190 0 0) 0x7fe69413a710 con 0x7fe69413f5c0 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6792/0 conn(0x7fe6940a67c0 legacy=0x7fe69413ec50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.82:6789/0 conn(0x7fe6940a6ba0 legacy=0x7fe6941d2ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 monclient: found mon.h 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6940a15f0 con 0x7fe69413f5c0 2024-08-19T09:57:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe67b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe69913a640 5 monclient: authenticate success, global_id 27059 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe69913a640 10 monclient: _renew_subs 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe69913a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.799+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe694139a40 con 0x7fe69413f5c0 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 10 client.27059.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 10 monclient: _renew_subs 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6941ce2a0 con 0x7fe69413f5c0 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 1 librados: init done 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe68c004bf0 con 0x7fe69413f5c0 2024-08-19T09:57:38.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe67b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe68c005220 con 0x7fe69413f5c0 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe67b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe67b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69159a640 4 set_mon_vals no callback set 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe67b7fe640 10 monclient: dump: 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.800+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe6740065c0 con 0x7fe674002860 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe68c005440 con 0x7fe69413f5c0 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7fe68c04e6f0 con 0x7fe69413f5c0 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe67b7fe640 10 client.27059.objecter ms_dispatch 0x7fe694000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe67b7fe640 3 client.27059.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe67b7fe640 3 client.27059.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe67b7fe640 20 client.27059.objecter dump_active .. 0 homeless 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.801+0000 7fe69259c640 10 monclient: get_auth_request con 0x7fe680040c70 auth_method 0 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe692d9d640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6791/0 conn(0x7fe674002860 legacy=0x7fe674004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 client.27059.objecter ms_handle_connect 0x7fe674002860 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 client.27059.objecter resend_mon_ops 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1746423646 0 0) 0x7fe6740065c0 con 0x7fe674002860 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient(hunting): my global_id is 27346 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe69413a710 con 0x7fe674002860 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1809404143 0 0) 0x7fe69413a710 con 0x7fe674002860 2024-08-19T09:57:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6740065c0 con 0x7fe674002860 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe684001320 con 0x7fe674002860 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient: dump: 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2026025506 0 0) 0x7fe6740065c0 con 0x7fe674002860 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe67400a160 con 0x7fe674002860 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe684004950 con 0x7fe674002860 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6791/0 conn(0x7fe674002860 legacy=0x7fe674004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.803+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6791/0 conn(0x7fe674002860 legacy=0x7fe674004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.804+0000 7fe67b7fe640 10 client.27059.objecter ms_handle_connect 0x7fe680040c70 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.810+0000 7fe69913a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.810+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe674009e50 con 0x7fe674009110 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe692d9d640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6791/0 conn(0x7fe674009110 legacy=0x7fe6740095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 client.27059.objecter ms_handle_connect 0x7fe674009110 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 client.27059.objecter resend_mon_ops 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3404796492 0 0) 0x7fe674009e50 con 0x7fe674009110 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient(hunting): my global_id is 27355 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6740065c0 con 0x7fe674009110 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3616212972 0 0) 0x7fe6740065c0 con 0x7fe674009110 2024-08-19T09:57:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe674009e50 con 0x7fe674009110 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe684005490 con 0x7fe674009110 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 10 monclient: dump: 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2335332560 0 0) 0x7fe674009e50 con 0x7fe674009110 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.812+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe67400c3f0 con 0x7fe674009110 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7473 (unknown 0 0 569402076) 0x7fe684007d40 con 0x7fe674009110 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe67b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe67b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6791/0 conn(0x7fe674009110 legacy=0x7fe6740095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe67b7fe640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.1:6791/0 conn(0x7fe674009110 legacy=0x7fe6740095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 465084, 2024-08-19T09:57:38.827 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.828 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.829 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:38.830 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:38.831 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.832 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.835 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.838 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:38.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.844 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:38.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.847 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.848 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:38.848 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 10 librados: watch_flush enter 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 10 librados: watch_flush exit 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 20 client.27059.objecter shutdown clearing up homeless session... 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 10 client.27059.objecter successfully canceled tick 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe680040c70 legacy=0x7fe680043130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 10 monclient: shutdown 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 >> v1:172.21.15.82:6791/0 conn(0x7fe69413f5c0 legacy=0x7fe6941d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69359e640 1 -- 172.21.15.1:0/2364793891 reap_dead start 2024-08-19T09:57:38.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69359e640 1 -- 172.21.15.1:0/2364793891 reap_dead start 2024-08-19T09:57:38.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 shutdown_connections 2024-08-19T09:57:38.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 >> 172.21.15.1:0/2364793891 conn(0x7fe6940ac4b0 msgr2=0x7fe6940a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 shutdown_connections 2024-08-19T09:57:38.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 -- 172.21.15.1:0/2364793891 wait complete. 2024-08-19T09:57:38.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.813+0000 7fe69913a640 1 librados: shutdown 2024-08-19T09:57:38.851 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:57:38.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-08-19T09:57:38.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:57:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.950+0000 7f0e0f87e640 10 monclient: get_monmap_and_config 2024-08-19T09:57:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.950+0000 7f0e0f87e640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.950+0000 7f0e0f87e640 10 monclient: monmap: 2024-08-19T09:57:38.951 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:38.951637+0000 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:38.951637+0000 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.953 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.953 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 1 Processor -- start 2024-08-19T09:57:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 1 -- start start 2024-08-19T09:57:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient: init 2024-08-19T09:57:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient: _add_conns ranks=[1,4,3,5,0,6,2,8,7] 2024-08-19T09:57:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient(hunting): picked mon.noname-f con 0x7f0e080a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e080a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient(hunting): picked mon.noname-g con 0x7f0e0813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e080a3b40 con 0x7f0e080a67c0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e080a8130 con 0x7f0e0813f5c0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e0806ede0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.951+0000 7f0e0f87e640 10 monclient(hunting): authenticate will time out at 1115.027974s 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.952+0000 7f0e0d61a640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f0e080a67c0 legacy=0x7f0e080a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.952+0000 7f0e0ce19640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f0e080a6ba0 0x7f0e080a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54278/0 (socket says 172.21.15.1:54278) 2024-08-19T09:57:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.952+0000 7f0e0ce19640 1 -- 172.21.15.1:0/2751071361 learned_addr learned my addr 172.21.15.1:0/2751071361 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0e0ce19640 1 -- 172.21.15.1:0/2751071361 >> v1:172.21.15.1:6791/0 conn(0x7f0e080a6ba0 legacy=0x7f0e080a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2469497331 0 0) 0x7f0e0806ede0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient(hunting): my global_id is 27364 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0df4003520 con 0x7f0e080a6ba0 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0e0de1b640 1 -- 172.21.15.1:0/2751071361 >> v1:172.21.15.82:6790/0 conn(0x7f0e0813f5c0 legacy=0x7f0e081419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2031451860 0 0) 0x7f0df4003520 con 0x7f0e080a6ba0 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 >> v1:172.21.15.82:6790/0 conn(0x7f0e0813f5c0 legacy=0x7f0e081419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 >> v1:172.21.15.82:6789/0 conn(0x7f0e080a67c0 legacy=0x7f0e080a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 monclient: found mon.noname-c 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e080a92a0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df8001ad0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: dump: 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.964 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0e0f87e640 5 monclient: authenticate success, global_id 27364 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0e0f87e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0df80048a0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 1 -- 172.21.15.1:0/2751071361 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df8004e10 con 0x7f0e080a6ba0 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0e0f87e640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0e0f87e640 4 set_mon_vals no callback set 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.954+0000 7f0dfffff640 10 monclient: dump: 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.966 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 10 monclient: shutdown 2024-08-19T09:57:38.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 1 -- 172.21.15.1:0/2751071361 >> v1:172.21.15.1:6791/0 conn(0x7f0e080a6ba0 legacy=0x7f0e080a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 1 -- 172.21.15.1:0/2751071361 shutdown_connections 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 1 -- 172.21.15.1:0/2751071361 >> 172.21.15.1:0/2751071361 conn(0x7f0e080ac4b0 msgr2=0x7f0e080a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 1 -- 172.21.15.1:0/2751071361 shutdown_connections 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 1 -- 172.21.15.1:0/2751071361 wait complete. 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 10 monclient: build_initial_monmap 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.955+0000 7f0e0f87e640 10 monclient: monmap: 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:38.968 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 librados: starting msgr at 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 librados: starting objecter 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 Processor -- start 2024-08-19T09:57:38.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 -- start start 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 librados: setting wanted keys 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 librados: calling monclient init 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient: init 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient: _add_conns ranks=[7,4,6,8,2,3,5,1,0] 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient(hunting): picked mon.noname-i con 0x7f0e080a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e080a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient(hunting): picked mon.noname-d con 0x7f0e0813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e0813c4d0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e080a2a30 con 0x7f0e0813f5c0 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e080aa240 con 0x7f0e080a67c0 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:38.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0f87e640 10 monclient(hunting): authenticate will time out at 1115.032682s 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0de1b640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f0e0813f5c0 legacy=0x7f0e081d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0ce19640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f0e080a6ba0 0x7f0e0813a030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54292/0 (socket says 172.21.15.1:54292) 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.956+0000 7f0e0ce19640 1 -- 172.21.15.1:0/3385081998 learned_addr learned my addr 172.21.15.1:0/3385081998 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.958+0000 7f0e0ce19640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.1:6791/0 conn(0x7f0e080a6ba0 legacy=0x7f0e0813a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.958+0000 7f0dfdffb640 10 client.?.objecter ms_handle_connect 0x7f0e080a6ba0 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.958+0000 7f0dfdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1241218172 0 0) 0x7f0e0813c4d0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): my global_id is 27373 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec004eb0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0d61a640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6792/0 conn(0x7f0e080a67c0 legacy=0x7f0e08139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 client.?.objecter ms_handle_connect 0x7f0e080a67c0 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3098693356 0 0) 0x7f0dec004eb0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e0813c4d0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df8004d90 con 0x7f0e080a6ba0 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): dump: 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569762496 0 0) 0x7f0e080aa240 con 0x7f0e080a67c0 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient(hunting): my global_id is 144547 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec004eb0 con 0x7f0e080a67c0 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2267191920 0 0) 0x7f0e0813c4d0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6792/0 conn(0x7f0e080a67c0 legacy=0x7f0e08139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.1:6792/0 conn(0x7f0e0813f5c0 legacy=0x7f0e081d6250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: _finish_hunting 0 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 monclient: found mon.c 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0813c1f0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: _finish_auth 0 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: _check_auth_tickets 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 5 monclient: authenticate success, global_id 27373 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 10 monclient: _renew_subs 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0813a740 con 0x7f0e080a6ba0 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0df8001d20 con 0x7f0e080a6ba0 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0df8005740 con 0x7f0e080a6ba0 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dfdffb640 10 monclient: dump: 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 10 client.27373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 10 monclient: _renew_subs 2024-08-19T09:57:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0dff7fe640 4 set_mon_vals no callback set 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e0813f1e0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.959+0000 7f0e0f87e640 1 librados: init done 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0e0f87e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0e080a27d0 con 0x7f0e0809da50 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0df8011ba0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7f0df804e2d0 con 0x7f0e080a6ba0 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0dfdffb640 10 client.27373.objecter ms_dispatch 0x7f0e08000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0dfdffb640 3 client.27373.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0dfdffb640 3 client.27373.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0dfdffb640 20 client.27373.objecter dump_active .. 0 homeless 2024-08-19T09:57:38.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.960+0000 7f0e0de1b640 10 monclient: get_auth_request con 0x7f0dec040ef0 auth_method 0 2024-08-19T09:57:38.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0e0d61a640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6790/0 conn(0x7f0e0809da50 legacy=0x7f0e0809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 10 client.27373.objecter ms_handle_connect 0x7f0e0809da50 2024-08-19T09:57:38.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 10 client.27373.objecter resend_mon_ops 2024-08-19T09:57:38.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665873691 0 0) 0x7f0dec004eb0 con 0x7f0e0809da50 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 10 monclient(hunting): my global_id is 135075 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.962+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e0813c4d0 con 0x7f0e0809da50 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 492172544 0 0) 0x7f0e0813c4d0 con 0x7f0e0809da50 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dec004eb0 con 0x7f0e0809da50 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 10 client.27373.objecter ms_handle_connect 0x7f0dec040ef0 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0e04004530 con 0x7f0e0809da50 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 10 monclient: dump: 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3280970103 0 0) 0x7f0dec004eb0 con 0x7f0e0809da50 2024-08-19T09:57:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e08142bd0 con 0x7f0e0809da50 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0e04006850 con 0x7f0e0809da50 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6790/0 conn(0x7f0e0809da50 legacy=0x7f0e0809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.963+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6790/0 conn(0x7f0e0809da50 legacy=0x7f0e0809de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.970+0000 7f0e0f87e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.970+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0e08068f60 con 0x7f0e0809e540 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0e0d61a640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6790/0 conn(0x7f0e0809e540 legacy=0x7f0e081db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 client.27373.objecter ms_handle_connect 0x7f0e0809e540 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 client.27373.objecter resend_mon_ops 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085884319 0 0) 0x7f0e08068f60 con 0x7f0e0809e540 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient(hunting): _negotiate 2024-08-19T09:57:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient(hunting): my global_id is 135084 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec004eb0 con 0x7f0e0809e540 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3523545574 0 0) 0x7f0dec004eb0 con 0x7f0e0809e540 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e08068f60 con 0x7f0e0809e540 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0e04004f10 con 0x7f0e0809e540 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.972+0000 7f0dfdffb640 10 monclient: dump: 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 849194681 0 0) 0x7f0e08068f60 con 0x7f0e0809e540 2024-08-19T09:57:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0e0809e920 con 0x7f0e0809e540 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7640 (unknown 0 0 3358386730) 0x7f0e040064e0 con 0x7f0e0809e540 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6790/0 conn(0x7f0e0809e540 legacy=0x7f0e081db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0dfdffb640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6790/0 conn(0x7f0e0809e540 legacy=0x7f0e081db990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:38.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 316034, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:57:38.991 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.992 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.993 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:38.994 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:38.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:38.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:38.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:38.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.001 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.002 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:39.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.010 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:39.011 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 10 librados: watch_flush enter 2024-08-19T09:57:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 10 librados: watch_flush exit 2024-08-19T09:57:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 20 client.27373.objecter shutdown clearing up homeless session... 2024-08-19T09:57:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 10 client.27373.objecter successfully canceled tick 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0dec040ef0 legacy=0x7f0dec0433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 10 monclient: shutdown 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 >> v1:172.21.15.1:6791/0 conn(0x7f0e080a6ba0 legacy=0x7f0e0813a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0de1b640 1 -- 172.21.15.1:0/3385081998 reap_dead start 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0de1b640 1 -- 172.21.15.1:0/3385081998 reap_dead start 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 shutdown_connections 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 >> 172.21.15.1:0/3385081998 conn(0x7f0e080ac4b0 msgr2=0x7f0e08141bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 shutdown_connections 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.973+0000 7f0e0f87e640 1 -- 172.21.15.1:0/3385081998 wait complete. 2024-08-19T09:57:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:38.974+0000 7f0e0f87e640 1 librados: shutdown 2024-08-19T09:57:39.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:57:39.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-08-19T09:57:39.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:57:39.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.113+0000 7f173cbc3640 10 monclient: get_monmap_and_config 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.113+0000 7f173cbc3640 10 monclient: build_initial_monmap 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.113+0000 7f173cbc3640 10 monclient: monmap: 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:39.114821+0000 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:39.114821+0000 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:39.116 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.114+0000 7f173cbc3640 1 Processor -- start 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.114+0000 7f173cbc3640 1 -- start start 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.114+0000 7f173cbc3640 10 monclient: init 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient: _add_conns ranks=[0,1,5,3,7,8,2,6,4] 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient(hunting): picked mon.noname-a con 0x7f17380a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient(hunting): picked mon.noname-f con 0x7f17380a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient(hunting): picked mon.noname-h con 0x7f173813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17380a3b40 con 0x7f17380a67c0 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17380a8130 con 0x7f17380a6ba0 2024-08-19T09:57:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173806ede0 con 0x7f173813f5c0 2024-08-19T09:57:39.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:39.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173cbc3640 10 monclient(hunting): authenticate will time out at 1115.191251s 2024-08-19T09:57:39.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f173659c640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f17380a67c0 legacy=0x7f17380a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T09:57:39.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f1735d9b640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f17380a6ba0 legacy=0x7f17380a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:39.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f1736d9d640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f173813f5c0 0x7f17381419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:57292/0 (socket says 172.21.15.1:57292) 2024-08-19T09:57:39.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.115+0000 7f1736d9d640 1 -- 172.21.15.1:0/2560705284 learned_addr learned my addr 172.21.15.1:0/2560705284 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.117+0000 7f1736d9d640 1 -- 172.21.15.1:0/2560705284 >> v1:172.21.15.82:6791/0 conn(0x7f173813f5c0 legacy=0x7f17381419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2066089962 0 0) 0x7f173806ede0 con 0x7f173813f5c0 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient(hunting): my global_id is 27068 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1724003520 con 0x7f173813f5c0 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 58470283 0 0) 0x7f1724003520 con 0x7f173813f5c0 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 >> v1:172.21.15.82:6789/0 conn(0x7f17380a6ba0 legacy=0x7f17380a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:39.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 >> v1:172.21.15.1:6789/0 conn(0x7f17380a67c0 legacy=0x7f17380a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: _finish_hunting 0 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 monclient: found mon.noname-h 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17380a92a0 con 0x7f173813f5c0 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: _finish_auth 0 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: _check_auth_tickets 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1730001ad0 con 0x7f173813f5c0 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 5 monclient: authenticate success, global_id 27068 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: dump: 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17300048a0 con 0x7f173813f5c0 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 1 -- 172.21.15.1:0/2560705284 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1730004e50 con 0x7f173813f5c0 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 4 set_mon_vals no callback set 2024-08-19T09:57:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173559a640 10 monclient: dump: 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 10 monclient: shutdown 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.118+0000 7f173cbc3640 1 -- 172.21.15.1:0/2560705284 >> v1:172.21.15.82:6791/0 conn(0x7f173813f5c0 legacy=0x7f17381419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.119+0000 7f173cbc3640 1 -- 172.21.15.1:0/2560705284 shutdown_connections 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.119+0000 7f173cbc3640 1 -- 172.21.15.1:0/2560705284 >> 172.21.15.1:0/2560705284 conn(0x7f17380ac4b0 msgr2=0x7f17380a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.119+0000 7f173cbc3640 1 -- 172.21.15.1:0/2560705284 shutdown_connections 2024-08-19T09:57:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.119+0000 7f173cbc3640 1 -- 172.21.15.1:0/2560705284 wait complete. 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.119+0000 7f173cbc3640 10 monclient: build_initial_monmap 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.119+0000 7f173cbc3640 10 monclient: monmap: 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 librados: starting msgr at 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 librados: starting objecter 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 Processor -- start 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 -- start start 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 librados: setting wanted keys 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 librados: calling monclient init 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient: init 2024-08-19T09:57:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient: _add_conns ranks=[7,6,3,5,8,0,1,4,2] 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient(hunting): picked mon.noname-i con 0x7f17380a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient(hunting): picked mon.noname-d con 0x7f17380a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient(hunting): picked mon.noname-g con 0x7f173813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173813c590 con 0x7f173813f5c0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17380a9ee0 con 0x7f17380a6ba0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17380a2520 con 0x7f17380a67c0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f1735d9b640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f17380a6ba0 legacy=0x7f17381d29d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f173cbc3640 10 monclient(hunting): authenticate will time out at 1115.196646s 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f1736d9d640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f173813f5c0 0x7f17381d6100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:54262/0 (socket says 172.21.15.1:54262) 2024-08-19T09:57:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.120+0000 7f1736d9d640 1 -- 172.21.15.1:0/3976464379 learned_addr learned my addr 172.21.15.1:0/3976464379 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173659c640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6792/0 conn(0x7f17380a67c0 legacy=0x7f173813e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 client.?.objecter ms_handle_connect 0x7f17380a67c0 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 client.?.objecter ms_handle_connect 0x7f173813f5c0 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f1736d9d640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6790/0 conn(0x7f173813f5c0 legacy=0x7f17381d6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660964853 0 0) 0x7f173813c590 con 0x7f173813f5c0 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): my global_id is 135093 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1718004eb0 con 0x7f173813f5c0 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 251494194 0 0) 0x7f17380a2520 con 0x7f17380a67c0 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): my global_id is 144556 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f173813c590 con 0x7f17380a67c0 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3229568894 0 0) 0x7f1718004eb0 con 0x7f173813f5c0 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17380a2520 con 0x7f173813f5c0 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1730001d20 con 0x7f173813f5c0 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): dump: 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2317596535 0 0) 0x7f173813c590 con 0x7f17380a67c0 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1718004eb0 con 0x7f17380a67c0 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f17280048d0 con 0x7f17380a67c0 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient(hunting): dump: 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 997663300 0 0) 0x7f17380a2520 con 0x7f173813f5c0 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6792/0 conn(0x7f17380a67c0 legacy=0x7f173813e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.1:6792/0 conn(0x7f17380a6ba0 legacy=0x7f17381d29d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 monclient: found mon.g 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173813f0e0 con 0x7f173813f5c0 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: _finish_auth 0 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 5 monclient: authenticate success, global_id 135093 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 10 monclient: _renew_subs 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f173813f370 con 0x7f173813f5c0 2024-08-19T09:57:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 10 client.135093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 10 monclient: _renew_subs 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f17380a1d40 con 0x7f173813f5c0 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f173cbc3640 1 librados: init done 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17300048a0 con 0x7f173813f5c0 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1730005380 con 0x7f173813f5c0 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f1734d99640 4 set_mon_vals no callback set 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.123+0000 7f172effd640 10 monclient: dump: 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.144 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.124+0000 7f173cbc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.124+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f171c0065c0 con 0x7f171c002860 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.124+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f17300117e0 con 0x7f173813f5c0 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.125+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(299..299 src has 1..299) ==== 5441+0+0 (unknown 2075475729 0 0) 0x7f173004e160 con 0x7f173813f5c0 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.125+0000 7f172effd640 10 client.135093.objecter ms_dispatch 0x7f1738000b90 osd_map(299..299 src has 1..299) 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.125+0000 7f172effd640 3 client.135093.objecter handle_osd_map got epochs [299,299] > 0 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.125+0000 7f172effd640 3 client.135093.objecter handle_osd_map decoding full epoch 299 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.125+0000 7f172effd640 20 client.135093.objecter dump_active .. 0 homeless 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.125+0000 7f1735d9b640 10 monclient: get_auth_request con 0x7f1718041520 auth_method 0 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.126+0000 7f173659c640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6791/0 conn(0x7f171c002860 legacy=0x7f171c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.126+0000 7f172effd640 10 client.135093.objecter ms_handle_connect 0x7f171c002860 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.126+0000 7f172effd640 10 client.135093.objecter resend_mon_ops 2024-08-19T09:57:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2016144474 0 0) 0x7f171c0065c0 con 0x7f171c002860 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient(hunting): my global_id is 27077 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17380a2520 con 0x7f171c002860 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3174245147 0 0) 0x7f17380a2520 con 0x7f171c002860 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f171c0065c0 con 0x7f171c002860 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1728005030 con 0x7f171c002860 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient: dump: 2024-08-19T09:57:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1603179759 0 0) 0x7f171c0065c0 con 0x7f171c002860 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f171c00a160 con 0x7f171c002860 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 client.135093.objecter ms_handle_connect 0x7f1718041520 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1728007310 con 0x7f171c002860 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6791/0 conn(0x7f171c002860 legacy=0x7f171c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.127+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6791/0 conn(0x7f171c002860 legacy=0x7f171c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.134+0000 7f173cbc3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.134+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f171c009e70 con 0x7f171c009130 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f173659c640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6791/0 conn(0x7f171c009130 legacy=0x7f171c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 10 client.135093.objecter ms_handle_connect 0x7f171c009130 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 10 client.135093.objecter resend_mon_ops 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669463461 0 0) 0x7f171c009e70 con 0x7f171c009130 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 10 monclient(hunting): my global_id is 27086 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.136+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f171c0065c0 con 0x7f171c009130 2024-08-19T09:57:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4172393426 0 0) 0x7f171c0065c0 con 0x7f171c009130 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f171c009e70 con 0x7f171c009130 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1728006840 con 0x7f171c009130 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 10 monclient: dump: 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2072326604 0 0) 0x7f171c009e70 con 0x7f171c009130 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f171c00c3f0 con 0x7f171c009130 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7640 (unknown 0 0 12215191) 0x7f1728009de0 con 0x7f171c009130 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6791/0 conn(0x7f171c009130 legacy=0x7f171c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f172effd640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6791/0 conn(0x7f171c009130 legacy=0x7f171c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 465403, 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:39.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:39.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:39.155 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:39.156 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:39.157 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:39.158 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:39.237 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:39.238 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.240 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.240 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.242 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.243 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.244 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:57:39.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.244 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:39.244 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f173cbc3640 10 librados: watch_flush enter 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f173cbc3640 10 librados: watch_flush exit 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f173cbc3640 20 client.135093.objecter shutdown clearing up homeless session... 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.137+0000 7f173cbc3640 10 client.135093.objecter successfully canceled tick 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1718041520 legacy=0x7f17180439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 10 monclient: shutdown 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 >> v1:172.21.15.82:6790/0 conn(0x7f173813f5c0 legacy=0x7f17381d6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f1736d9d640 1 -- 172.21.15.1:0/3976464379 reap_dead start 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f1736d9d640 1 -- 172.21.15.1:0/3976464379 reap_dead start 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 shutdown_connections 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 >> 172.21.15.1:0/3976464379 conn(0x7f17380ac4b0 msgr2=0x7f17381419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 shutdown_connections 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 -- 172.21.15.1:0/3976464379 wait complete. 2024-08-19T09:57:39.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.138+0000 7f173cbc3640 1 librados: shutdown 2024-08-19T09:57:39.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:57:39.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-08-19T09:57:39.248 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.347+0000 7f8f885cf640 10 monclient: get_monmap_and_config 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.347+0000 7f8f885cf640 10 monclient: build_initial_monmap 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.347+0000 7f8f885cf640 10 monclient: monmap: 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:57:39.348233+0000 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:57:39.348233+0000 2024-08-19T09:57:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:39.349 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.347+0000 7f8f885cf640 1 Processor -- start 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.347+0000 7f8f885cf640 1 -- start start 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.347+0000 7f8f885cf640 10 monclient: init 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient: _add_conns ranks=[1,7,8,6,4,3,5,2,0] 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient(hunting): picked mon.noname-f con 0x7f8f800a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient(hunting): picked mon.noname-i con 0x7f8f800a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient(hunting): picked mon.noname-e con 0x7f8f8013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f800a3b40 con 0x7f8f800a67c0 2024-08-19T09:57:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f800a8130 con 0x7f8f800a6ba0 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8006ede0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f885cf640 10 monclient(hunting): authenticate will time out at 1115.424593s 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f86b6c640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f8f8013f5c0 legacy=0x7f8f801419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f8636b640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f8f800a67c0 legacy=0x7f8f800a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f85b6a640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f8f800a6ba0 0x7f8f800a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:59064/0 (socket says 172.21.15.1:59064) 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.348+0000 7f8f85b6a640 1 -- 172.21.15.1:0/2889753270 learned_addr learned my addr 172.21.15.1:0/2889753270 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:39.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.351+0000 7f8f85b6a640 1 -- 172.21.15.1:0/2889753270 >> v1:172.21.15.82:6792/0 conn(0x7f8f800a6ba0 legacy=0x7f8f800a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.550+0000 7f8f86b6c640 1 -- 172.21.15.1:0/2889753270 >> v1:172.21.15.1:6793/0 conn(0x7f8f8013f5c0 legacy=0x7f8f801419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:57:39.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.550+0000 7f8f8636b640 1 -- 172.21.15.1:0/2889753270 >> v1:172.21.15.82:6789/0 conn(0x7f8f800a67c0 legacy=0x7f8f800a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T09:57:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864031156 0 0) 0x7f8f800a8130 con 0x7f8f800a6ba0 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 10 monclient(hunting): my global_id is 144565 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f6c003520 con 0x7f8f800a6ba0 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 184920627 0 0) 0x7f8f6c003520 con 0x7f8f800a6ba0 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 >> v1:172.21.15.1:6793/0 conn(0x7f8f8013f5c0 legacy=0x7f8f801419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 >> v1:172.21.15.82:6789/0 conn(0x7f8f800a67c0 legacy=0x7f8f800a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 10 monclient: _finish_hunting 0 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 1 monclient: found mon.noname-i 2024-08-19T09:57:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.784+0000 7f8f85369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f800a92a0 con 0x7f8f800a6ba0 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: _finish_auth 0 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: _check_auth_tickets 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f70001ac0 con 0x7f8f800a6ba0 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: dump: 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 5 monclient: authenticate success, global_id 144565 2024-08-19T09:57:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f700048a0 con 0x7f8f800a6ba0 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 1 -- 172.21.15.1:0/2889753270 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f70004e10 con 0x7f8f800a6ba0 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f85369640 10 monclient: dump: 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 10 monclient: get_monmap_and_config success 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 4 set_mon_vals no callback set 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 10 monclient: shutdown 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 1 -- 172.21.15.1:0/2889753270 >> v1:172.21.15.82:6792/0 conn(0x7f8f800a6ba0 legacy=0x7f8f800a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 1 -- 172.21.15.1:0/2889753270 shutdown_connections 2024-08-19T09:57:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.785+0000 7f8f885cf640 1 -- 172.21.15.1:0/2889753270 >> 172.21.15.1:0/2889753270 conn(0x7f8f800ac4b0 msgr2=0x7f8f800a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 1 -- 172.21.15.1:0/2889753270 shutdown_connections 2024-08-19T09:57:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 1 -- 172.21.15.1:0/2889753270 wait complete. 2024-08-19T09:57:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 10 monclient: build_initial_monmap 2024-08-19T09:57:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 10 monclient: monmap: 2024-08-19T09:57:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:57:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:57:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:57:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:57:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 1 librados: starting msgr at 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 1 librados: starting objecter 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.786+0000 7f8f885cf640 1 Processor -- start 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 1 -- start start 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 1 librados: setting wanted keys 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 1 librados: calling monclient init 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient: init 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient: _reopen_session rank -1 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient: _add_conns ranks=[6,5,4,0,8,2,3,7,1] 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient(hunting): picked mon.noname-d con 0x7f8f800a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient(hunting): picked mon.noname-h con 0x7f8f800a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:57:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8f8013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f800a26b0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f800a1470 con 0x7f8f800a6ba0 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f8636b640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8f800a67c0 legacy=0x7f8f8013eb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f80139d30 con 0x7f8f800a67c0 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient(hunting): _renew_subs 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f885cf640 10 monclient(hunting): authenticate will time out at 1115.863560s 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f86b6c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f8f8013f5c0 0x7f8f801d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54320/0 (socket says 172.21.15.1:54320) 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.787+0000 7f8f86b6c640 1 -- 172.21.15.1:0/3436495100 learned_addr learned my addr 172.21.15.1:0/3436495100 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.789+0000 7f8f86b6c640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.1:6791/0 conn(0x7f8f8013f5c0 legacy=0x7f8f801d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.789+0000 7f8f76ffd640 10 client.?.objecter ms_handle_connect 0x7f8f8013f5c0 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.789+0000 7f8f76ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f85b6a640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6791/0 conn(0x7f8f800a6ba0 legacy=0x7f8f801d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275473684 0 0) 0x7f8f800a26b0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): my global_id is 27382 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f64004ee0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 client.?.objecter ms_handle_connect 0x7f8f800a6ba0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2776316456 0 0) 0x7f8f64004ee0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f800a26b0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f780048d0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): dump: 2024-08-19T09:57:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.808 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220981954 0 0) 0x7f8f800a1470 con 0x7f8f800a6ba0 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient(hunting): my global_id is 27095 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f64004ee0 con 0x7f8f800a6ba0 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4115405920 0 0) 0x7f8f800a26b0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.1:6792/0 conn(0x7f8f800a67c0 legacy=0x7f8f8013eb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6791/0 conn(0x7f8f800a6ba0 legacy=0x7f8f801d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:57:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 monclient: found mon.c 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8013a5d0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: _finish_auth 0 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 5 monclient: authenticate success, global_id 27382 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 10 monclient: _renew_subs 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f801cac80 con 0x7f8f8013f5c0 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 10 client.27382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 10 monclient: _renew_subs 2024-08-19T09:57:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f8013f240 con 0x7f8f8013f5c0 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f885cf640 1 librados: init done 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f78004be0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f78005150 con 0x7f8f8013f5c0 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f77fff640 4 set_mon_vals no callback set 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.790+0000 7f8f76ffd640 10 monclient: dump: 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.791+0000 7f8f885cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.791+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8f680065c0 con 0x7f8f68002860 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.791+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8f780052f0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.792+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(300..300 src has 1..300) ==== 5770+0+0 (unknown 2901893557 0 0) 0x7f8f7804e6d0 con 0x7f8f8013f5c0 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.792+0000 7f8f76ffd640 10 client.27382.objecter ms_dispatch 0x7f8f80000b90 osd_map(300..300 src has 1..300) 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.792+0000 7f8f76ffd640 3 client.27382.objecter handle_osd_map got epochs [300,300] > 0 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.792+0000 7f8f76ffd640 3 client.27382.objecter handle_osd_map decoding full epoch 300 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.792+0000 7f8f85b6a640 10 monclient: get_auth_request con 0x7f8f64040f50 auth_method 0 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.792+0000 7f8f76ffd640 20 client.27382.objecter dump_active .. 0 homeless 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f8636b640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6792/0 conn(0x7f8f68002860 legacy=0x7f8f68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 client.27382.objecter ms_handle_connect 0x7f8f68002860 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 client.27382.objecter resend_mon_ops 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661989695 0 0) 0x7f8f680065c0 con 0x7f8f68002860 2024-08-19T09:57:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient(hunting): my global_id is 144574 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f800a26b0 con 0x7f8f68002860 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3453944357 0 0) 0x7f8f800a26b0 con 0x7f8f68002860 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f680065c0 con 0x7f8f68002860 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f7c004570 con 0x7f8f68002860 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 monclient: dump: 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 10 client.27382.objecter ms_handle_connect 0x7f8f64040f50 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4003000249 0 0) 0x7f8f680065c0 con 0x7f8f68002860 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.794+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f6800a160 con 0x7f8f68002860 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.795+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8f7c0068d0 con 0x7f8f68002860 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.795+0000 7f8f76ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.795+0000 7f8f76ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.795+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6792/0 conn(0x7f8f68002860 legacy=0x7f8f68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.795+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6792/0 conn(0x7f8f68002860 legacy=0x7f8f68004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.801+0000 7f8f885cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:57:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.801+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8f68009e30 con 0x7f8f68009110 2024-08-19T09:57:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f8636b640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6792/0 conn(0x7f8f68009110 legacy=0x7f8f68009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:57:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 client.27382.objecter ms_handle_connect 0x7f8f68009110 2024-08-19T09:57:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 client.27382.objecter resend_mon_ops 2024-08-19T09:57:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145622829 0 0) 0x7f8f68009e30 con 0x7f8f68009110 2024-08-19T09:57:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient(hunting): my global_id is 144583 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f680065c0 con 0x7f8f68009110 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4237353784 0 0) 0x7f8f680065c0 con 0x7f8f68009110 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f68009e30 con 0x7f8f68009110 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f7c005010 con 0x7f8f68009110 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 10 monclient: dump: 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:57:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:57:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 218975300 0 0) 0x7f8f68009e30 con 0x7f8f68009110 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.804+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f6800c3f0 con 0x7f8f68009110 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7640 (unknown 0 0 1736656017) 0x7f8f7c0063d0 con 0x7f8f68009110 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f76ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f76ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6792/0 conn(0x7f8f68009110 legacy=0x7f8f68009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f76ffd640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6792/0 conn(0x7f8f68009110 legacy=0x7f8f68009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 184184, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:57:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:39.823 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:39.824 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:57:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:57:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:57:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:57:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:57:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:57:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:57:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:57:39.840 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:57:39.842 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:57:39.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:57:39.843 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:57:39.843 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:57:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 10 librados: watch_flush enter 2024-08-19T09:57:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 10 librados: watch_flush exit 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 20 client.27382.objecter shutdown clearing up homeless session... 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 10 client.27382.objecter successfully canceled tick 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8f64040f50 legacy=0x7f8f64043410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 10 monclient: shutdown 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 >> v1:172.21.15.1:6791/0 conn(0x7f8f8013f5c0 legacy=0x7f8f801d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f86b6c640 1 -- 172.21.15.1:0/3436495100 reap_dead start 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f86b6c640 1 -- 172.21.15.1:0/3436495100 reap_dead start 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.805+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 shutdown_connections 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.806+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 >> 172.21.15.1:0/3436495100 conn(0x7f8f800ac4b0 msgr2=0x7f8f800ab120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.806+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 shutdown_connections 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.806+0000 7f8f885cf640 1 -- 172.21.15.1:0/3436495100 wait complete. 2024-08-19T09:57:39.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:57:39.806+0000 7f8f885cf640 1 librados: shutdown 2024-08-19T09:57:39.846 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T09:57:39.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T09:58:03.464 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ 2 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3010 ms) 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6996 ms) 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (18896 ms) 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7067 ms) 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (32742 ms) 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7267 ms) 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-08-19T09:58:03.465 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (10252 ms) 2024-08-19T09:58:03.466 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (217403 ms total) 2024-08-19T09:58:03.466 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: 2024-08-19T09:58:03.466 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-08-19T09:58:03.466 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (404157 ms total) 2024-08-19T09:58:03.466 INFO:tasks.workunit.client.0.smithi082.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-08-19T09:59:09.847 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-08-19T09:59:09.848 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T09:59:09.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T09:59:09.852 INFO:tasks.ceph.mon.f:Started 2024-08-19T09:59:09.852 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-08-19T09:59:09.852 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T09:59:09.853 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T09:59:09.855 INFO:tasks.ceph.mon.e:Started 2024-08-19T09:59:09.856 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T09:59:09.856 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T09:59:09.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T09:59:09.858 INFO:tasks.ceph.mon.d:Started 2024-08-19T09:59:09.858 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T09:59:09.858 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T09:59:09.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T09:59:09.860 INFO:tasks.ceph.mon.a:Started 2024-08-19T09:59:09.860 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-08-19T09:59:09.861 INFO:tasks.ceph.mon.g:Sent signal 19 2024-08-19T09:59:09.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:59:24.862 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-08-19T09:59:24.863 INFO:tasks.ceph.mon.g:Sent signal 18 2024-08-19T09:59:24.864 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:59:24.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:59:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.981+0000 7f4c06fcf640 10 monclient: get_monmap_and_config 2024-08-19T09:59:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.981+0000 7f4c06fcf640 10 monclient: build_initial_monmap 2024-08-19T09:59:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.981+0000 7f4c06fcf640 10 monclient: monmap: 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:24.982390+0000 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:24.982390+0000 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:24.984 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:24.984 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:24.984 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.982+0000 7f4c06fcf640 1 Processor -- start 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.982+0000 7f4c06fcf640 1 -- start start 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.982+0000 7f4c06fcf640 10 monclient: init 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient: _add_conns ranks=[2,1,5,4,0,6,7,3,8] 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f4c0009f310 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient(hunting): picked mon.noname-f con 0x7f4c0009f6f0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient(hunting): picked mon.noname-h con 0x7f4c000a0b70 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c0009e870 con 0x7f4c0009f6f0 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c000a1760 con 0x7f4c0009f310 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c0009feb0 con 0x7f4c000a0b70 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c06fcf640 10 monclient(hunting): authenticate will time out at 1221.059541s 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c04d6b640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f4c0009f310 0x7f4c0009da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39794/0 (socket says 172.21.15.1:39794) 2024-08-19T09:59:24.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.983+0000 7f4c04d6b640 1 -- 172.21.15.1:0/2106444043 learned_addr learned my addr 172.21.15.1:0/2106444043 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:24.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.985+0000 7f4c0556c640 1 -- 172.21.15.1:0/2106444043 >> v1:172.21.15.82:6791/0 conn(0x7f4c000a0b70 legacy=0x7f4c0013e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:24.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.986+0000 7f4bf7fff640 1 -- 172.21.15.1:0/2106444043 >> v1:172.21.15.82:6789/0 conn(0x7f4c0009f6f0 legacy=0x7f4c0009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:24.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.987+0000 7f4c04d6b640 1 -- 172.21.15.1:0/2106444043 >> v1:172.21.15.1:6790/0 conn(0x7f4c0009f310 legacy=0x7f4c0009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362674955 0 0) 0x7f4c000a1760 con 0x7f4c0009f310 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient(hunting): my global_id is 164351 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be8003520 con 0x7f4c0009f310 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2915720162 0 0) 0x7f4be8003520 con 0x7f4c0009f310 2024-08-19T09:59:24.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 >> v1:172.21.15.82:6791/0 conn(0x7f4c000a0b70 legacy=0x7f4c0013e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 >> v1:172.21.15.82:6789/0 conn(0x7f4c0009f6f0 legacy=0x7f4c0009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 monclient: found mon.noname-b 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c000a0190 con 0x7f4c0009f310 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4bfc001ae0 con 0x7f4c0009f310 2024-08-19T09:59:24.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: dump: 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:24.992 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4bfc0048a0 con 0x7f4c0009f310 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 1 -- 172.21.15.1:0/2106444043 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4bfc004d90 con 0x7f4c0009f310 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.988+0000 7f4bf77fe640 10 monclient: dump: 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:24.993 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.989+0000 7f4c06fcf640 5 monclient: authenticate success, global_id 164351 2024-08-19T09:59:24.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.989+0000 7f4c06fcf640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:24.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.989+0000 7f4c06fcf640 4 set_mon_vals no callback set 2024-08-19T09:59:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.991+0000 7f4c06fcf640 10 monclient: shutdown 2024-08-19T09:59:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.991+0000 7f4c06fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.991+0000 7f4c06fcf640 1 -- 172.21.15.1:0/2106444043 >> v1:172.21.15.1:6790/0 conn(0x7f4c0009f310 legacy=0x7f4c0009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.991+0000 7f4c06fcf640 1 -- 172.21.15.1:0/2106444043 shutdown_connections 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.991+0000 7f4c06fcf640 1 -- 172.21.15.1:0/2106444043 >> 172.21.15.1:0/2106444043 conn(0x7f4c00068650 msgr2=0x7f4c00068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.992+0000 7f4c06fcf640 1 -- 172.21.15.1:0/2106444043 shutdown_connections 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.992+0000 7f4c06fcf640 1 -- 172.21.15.1:0/2106444043 wait complete. 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.993+0000 7f4c06fcf640 10 monclient: build_initial_monmap 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.993+0000 7f4c06fcf640 10 monclient: monmap: 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.993+0000 7f4c06fcf640 1 librados: starting msgr at 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.993+0000 7f4c06fcf640 1 librados: starting objecter 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.994+0000 7f4c06fcf640 1 Processor -- start 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.994+0000 7f4c06fcf640 1 -- start start 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.994+0000 7f4c06fcf640 1 librados: setting wanted keys 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.994+0000 7f4c06fcf640 1 librados: calling monclient init 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.994+0000 7f4c06fcf640 10 monclient: init 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.995+0000 7f4c06fcf640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.995+0000 7f4c06fcf640 10 monclient: _add_conns ranks=[1,7,4,8,2,6,3,5,0] 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.995+0000 7f4c06fcf640 10 monclient(hunting): picked mon.noname-f con 0x7f4c0009f310 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.995+0000 7f4c06fcf640 10 monclient(hunting): picked mon.noname-i con 0x7f4c0009f6f0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.995+0000 7f4c04d6b640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f4c0009f310 0x7f4c001c98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:34282/0 (socket says 172.21.15.1:34282) 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.995+0000 7f4c04d6b640 1 -- 172.21.15.1:0/1045601456 learned_addr learned my addr 172.21.15.1:0/1045601456 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.996+0000 7f4c06fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f4c000a0b70 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.996+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c001a5730 con 0x7f4c0009f310 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.996+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c000a6b30 con 0x7f4c000a0b70 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.996+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c001a7a90 con 0x7f4c0009f6f0 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.996+0000 7f4c06fcf640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.996+0000 7f4c06fcf640 10 monclient(hunting): authenticate will time out at 1221.072374s 2024-08-19T09:59:25.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.997+0000 7f4c04d6b640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.82:6789/0 conn(0x7f4c0009f310 legacy=0x7f4c001c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf57fa640 10 client.?.objecter ms_handle_connect 0x7f4c0009f310 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4c0556c640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.1:6791/0 conn(0x7f4c000a0b70 legacy=0x7f4c001d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf7fff640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.82:6792/0 conn(0x7f4c0009f6f0 legacy=0x7f4c001ba0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf57fa640 10 client.?.objecter ms_handle_connect 0x7f4c000a0b70 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf57fa640 10 client.?.objecter ms_handle_connect 0x7f4c0009f6f0 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.998+0000 7f4bf57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.999+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148316198 0 0) 0x7f4c001a5730 con 0x7f4c0009f310 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.999+0000 7f4bf57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.999+0000 7f4bf57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.999+0000 7f4bf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.999+0000 7f4bf57fa640 10 monclient(hunting): my global_id is 174106 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:24.999+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bec004eb0 con 0x7f4c0009f310 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782009222 0 0) 0x7f4c001a7a90 con 0x7f4c0009f6f0 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): my global_id is 144601 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c001a5730 con 0x7f4c0009f6f0 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2867008081 0 0) 0x7f4bec004eb0 con 0x7f4c0009f310 2024-08-19T09:59:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c001a7a90 con 0x7f4c0009f310 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4bfc001a60 con 0x7f4c0009f310 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): dump: 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139056438 0 0) 0x7f4c000a6b30 con 0x7f4c000a0b70 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.000+0000 7f4bf57fa640 10 monclient(hunting): my global_id is 27418 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bec004eb0 con 0x7f4c000a0b70 2024-08-19T09:59:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2911882725 0 0) 0x7f4bec004eb0 con 0x7f4c000a0b70 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c000a6b30 con 0x7f4c000a0b70 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4bf8001ff0 con 0x7f4c000a0b70 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient(hunting): dump: 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.013 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.013 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.013 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.013 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.013 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3623374293 0 0) 0x7f4c000a6b30 con 0x7f4c000a0b70 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.82:6792/0 conn(0x7f4c0009f6f0 legacy=0x7f4c001ba0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.82:6789/0 conn(0x7f4c0009f310 legacy=0x7f4c001c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 monclient: found mon.c 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c001a6290 con 0x7f4c000a0b70 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4bf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.001+0000 7f4c06fcf640 5 monclient: authenticate success, global_id 27418 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 10 monclient: _renew_subs 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c001ba7e0 con 0x7f4c000a0b70 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 10 client.27418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 10 monclient: _renew_subs 2024-08-19T09:59:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c001d34e0 con 0x7f4c000a0b70 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4c06fcf640 1 librados: init done 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4bf8003c40 con 0x7f4c000a0b70 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4bf8004170 con 0x7f4c000a0b70 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf6ffd640 4 set_mon_vals no callback set 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.002+0000 7f4bf57fa640 10 monclient: dump: 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4c06fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4c06fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4c06fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be0005080 con 0x7f4c000a0b70 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4bf8004310 con 0x7f4c000a0b70 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(360..360 src has 299..360) ==== 5477+0+0 (unknown 106524440 0 0) 0x7f4bf804e2b0 con 0x7f4c000a0b70 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4bf57fa640 10 client.27418.objecter ms_dispatch 0x7f4c00000b90 osd_map(360..360 src has 299..360) 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4bf57fa640 3 client.27418.objecter handle_osd_map got epochs [360,360] > 0 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.003+0000 7f4bf57fa640 3 client.27418.objecter handle_osd_map decoding full epoch 360 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.004+0000 7f4bf57fa640 20 client.27418.objecter dump_active .. 0 homeless 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.004+0000 7f4c04d6b640 10 monclient: get_auth_request con 0x7f4bec0414e0 auth_method 0 2024-08-19T09:59:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.006+0000 7f4bf57fa640 10 client.27418.objecter ms_handle_connect 0x7f4bec0414e0 2024-08-19T09:59:25.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.009+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f4bf8017630 con 0x7f4c000a0b70 2024-08-19T09:59:25.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.009+0000 7f4bf57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.009+0000 7f4bf57fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:25.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4c06fcf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4c06fcf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4c06fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4be0005370 con 0x7f4c000a0b70 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4bf57fa640 1 -- 172.21.15.1:0/1045601456 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 3940885478) 0x7f4bf8016f30 con 0x7f4c000a0b70 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4bf57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.266+0000 7f4bf57fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:25.268 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":86,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 10 librados: watch_flush enter 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 10 librados: watch_flush exit 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 20 client.27418.objecter shutdown clearing up homeless session... 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 10 client.27418.objecter successfully canceled tick 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4bec0414e0 legacy=0x7f4bec0439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 10 monclient: shutdown 2024-08-19T09:59:25.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 >> v1:172.21.15.1:6791/0 conn(0x7f4c000a0b70 legacy=0x7f4c001d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 shutdown_connections 2024-08-19T09:59:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 >> 172.21.15.1:0/1045601456 conn(0x7f4c00068650 msgr2=0x7f4c000a5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 shutdown_connections 2024-08-19T09:59:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 -- 172.21.15.1:0/1045601456 wait complete. 2024-08-19T09:59:25.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.268+0000 7f4c06fcf640 1 librados: shutdown 2024-08-19T09:59:25.280 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:59:25.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:59:25.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:59:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.379+0000 7f5a1c898640 10 monclient: get_monmap_and_config 2024-08-19T09:59:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.379+0000 7f5a1c898640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.379+0000 7f5a1c898640 10 monclient: monmap: 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:25.380535+0000 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:25.380535+0000 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.382 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.382 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 1 Processor -- start 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 1 -- start start 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient: init 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient: _add_conns ranks=[5,6,0,2,8,7,1,4,3] 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient(hunting): picked mon.noname-h con 0x7f5a180a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient(hunting): picked mon.noname-d con 0x7f5a180a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a180a3b40 con 0x7f5a1813f5c0 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a180a8130 con 0x7f5a180a67c0 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1806ede0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a1c898640 10 monclient(hunting): authenticate will time out at 1221.456886s 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a0ffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5a180a6ba0 0x7f5a180a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:44822/0 (socket says 172.21.15.1:44822) 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a0ffff640 1 -- 172.21.15.1:0/3358977888 learned_addr learned my addr 172.21.15.1:0/3358977888 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.380+0000 7f5a16d9d640 1 --1- 172.21.15.1:0/3358977888 >> v1:172.21.15.1:6789/0 conn(0x7f5a1813f5c0 0x7f5a181419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57768/0 (socket says 172.21.15.1:57768) 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a0ffff640 1 -- 172.21.15.1:0/3358977888 >> v1:172.21.15.1:6792/0 conn(0x7f5a180a6ba0 legacy=0x7f5a180a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a16d9d640 1 -- 172.21.15.1:0/3358977888 >> v1:172.21.15.1:6789/0 conn(0x7f5a1813f5c0 legacy=0x7f5a181419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a1659c640 1 -- 172.21.15.1:0/3358977888 >> v1:172.21.15.82:6791/0 conn(0x7f5a180a67c0 legacy=0x7f5a180a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210342719 0 0) 0x7f5a180a3b40 con 0x7f5a1813f5c0 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): my global_id is 174105 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a04003520 con 0x7f5a1813f5c0 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319735389 0 0) 0x7f5a1806ede0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): my global_id is 174111 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a180a3b40 con 0x7f5a180a6ba0 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207550915 0 0) 0x7f5a180a8130 con 0x7f5a180a67c0 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient(hunting): my global_id is 27122 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a1806ede0 con 0x7f5a180a67c0 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3929876140 0 0) 0x7f5a180a3b40 con 0x7f5a180a6ba0 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 >> v1:172.21.15.82:6791/0 conn(0x7f5a180a67c0 legacy=0x7f5a180a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 >> v1:172.21.15.1:6789/0 conn(0x7f5a1813f5c0 legacy=0x7f5a181419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 monclient: found mon.noname-d 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a180a92a0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a00001a50 con 0x7f5a180a6ba0 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a1c898640 5 monclient: authenticate success, global_id 174111 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a1c898640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: dump: 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a1c898640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a00001d20 con 0x7f5a180a6ba0 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 1 -- 172.21.15.1:0/3358977888 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a00004be0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a1c898640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a1c898640 4 set_mon_vals no callback set 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.383+0000 7f5a15d9b640 10 monclient: dump: 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 10 monclient: shutdown 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 1 -- 172.21.15.1:0/3358977888 >> v1:172.21.15.1:6792/0 conn(0x7f5a180a6ba0 legacy=0x7f5a180a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 1 -- 172.21.15.1:0/3358977888 shutdown_connections 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 1 -- 172.21.15.1:0/3358977888 >> 172.21.15.1:0/3358977888 conn(0x7f5a180ac4b0 msgr2=0x7f5a180a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 1 -- 172.21.15.1:0/3358977888 shutdown_connections 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.384+0000 7f5a1c898640 1 -- 172.21.15.1:0/3358977888 wait complete. 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient: monmap: 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 librados: starting msgr at 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 librados: starting objecter 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 Processor -- start 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 -- start start 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 librados: setting wanted keys 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 librados: calling monclient init 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient: init 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient: _add_conns ranks=[3,6,4,0,1,5,7,8,2] 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient(hunting): picked mon.noname-g con 0x7f5a180a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient(hunting): picked mon.noname-d con 0x7f5a180a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1813f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a18139970 con 0x7f5a180a67c0 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.385+0000 7f5a1c898640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1813bcd0 con 0x7f5a1813f5c0 2024-08-19T09:59:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.386+0000 7f5a1c898640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a181cf020 con 0x7f5a180a6ba0 2024-08-19T09:59:25.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.386+0000 7f5a1c898640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.386+0000 7f5a1c898640 10 monclient(hunting): authenticate will time out at 1221.462031s 2024-08-19T09:59:25.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.386+0000 7f5a0ffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5a180a6ba0 0x7f5a181ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:44832/0 (socket says 172.21.15.1:44832) 2024-08-19T09:59:25.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.386+0000 7f5a0ffff640 1 -- 172.21.15.1:0/1567142778 learned_addr learned my addr 172.21.15.1:0/1567142778 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0ffff640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6792/0 conn(0x7f5a180a6ba0 legacy=0x7f5a181ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 client.?.objecter ms_handle_connect 0x7f5a180a6ba0 2024-08-19T09:59:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a16d9d640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6791/0 conn(0x7f5a1813f5c0 legacy=0x7f5a181d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 client.?.objecter ms_handle_connect 0x7f5a1813f5c0 2024-08-19T09:59:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1659c640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.82:6790/0 conn(0x7f5a180a67c0 legacy=0x7f5a181ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 client.?.objecter ms_handle_connect 0x7f5a180a67c0 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49178787 0 0) 0x7f5a181cf020 con 0x7f5a180a6ba0 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): my global_id is 174120 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f8004eb0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038444163 0 0) 0x7f5a1813bcd0 con 0x7f5a1813f5c0 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): my global_id is 27427 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a181cf020 con 0x7f5a1813f5c0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892044581 0 0) 0x7f5a18139970 con 0x7f5a180a67c0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): my global_id is 135120 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a1813bcd0 con 0x7f5a180a67c0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 76200299 0 0) 0x7f59f8004eb0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a18139970 con 0x7f5a180a6ba0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a00004b20 con 0x7f5a180a6ba0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): dump: 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1948731373 0 0) 0x7f5a181cf020 con 0x7f5a1813f5c0 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59f8004eb0 con 0x7f5a1813f5c0 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a100020b0 con 0x7f5a1813f5c0 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient(hunting): dump: 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1851677073 0 0) 0x7f5a18139970 con 0x7f5a180a6ba0 2024-08-19T09:59:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6791/0 conn(0x7f5a1813f5c0 legacy=0x7f5a181d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.82:6790/0 conn(0x7f5a180a67c0 legacy=0x7f5a181ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 monclient: found mon.d 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1813b9f0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 5 monclient: authenticate success, global_id 174120 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 10 monclient: _renew_subs 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1813a8a0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 10 client.174120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 10 monclient: _renew_subs 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a180a26d0 con 0x7f5a180a6ba0 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.388+0000 7f5a1c898640 1 librados: init done 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a00001a30 con 0x7f5a180a6ba0 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a00001d20 con 0x7f5a180a6ba0 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a1559a640 4 set_mon_vals no callback set 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 10 monclient: dump: 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.411 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a1c898640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59fc0065c0 con 0x7f59fc002860 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.389+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5a00011810 con 0x7f5a180a6ba0 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.390+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(360..360 src has 299..360) ==== 5477+0+0 (unknown 106524440 0 0) 0x7f5a0004e270 con 0x7f5a180a6ba0 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.390+0000 7f5a0effd640 10 client.174120.objecter ms_dispatch 0x7f5a18000b90 osd_map(360..360 src has 299..360) 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.390+0000 7f5a0effd640 3 client.174120.objecter handle_osd_map got epochs [360,360] > 0 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.390+0000 7f5a0effd640 3 client.174120.objecter handle_osd_map decoding full epoch 360 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.390+0000 7f5a0effd640 20 client.174120.objecter dump_active .. 0 homeless 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.390+0000 7f5a16d9d640 10 monclient: get_auth_request con 0x7f59f80416e0 auth_method 0 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.391+0000 7f5a1659c640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6789/0 conn(0x7f59fc002860 legacy=0x7f59fc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.391+0000 7f5a0effd640 10 client.174120.objecter ms_handle_connect 0x7f59fc002860 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.391+0000 7f5a0effd640 10 client.174120.objecter resend_mon_ops 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3853807343 0 0) 0x7f59fc0065c0 con 0x7f59fc002860 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient(hunting): my global_id is 174114 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a18139970 con 0x7f59fc002860 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2009015280 0 0) 0x7f5a18139970 con 0x7f59fc002860 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59fc0065c0 con 0x7f59fc002860 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a08003e90 con 0x7f59fc002860 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient: dump: 2024-08-19T09:59:25.413 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.414 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1023004520 0 0) 0x7f59fc0065c0 con 0x7f59fc002860 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59fc00a160 con 0x7f59fc002860 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 client.174120.objecter ms_handle_connect 0x7f59f80416e0 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5a080040b0 con 0x7f59fc002860 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6789/0 conn(0x7f59fc002860 legacy=0x7f59fc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.392+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6789/0 conn(0x7f59fc002860 legacy=0x7f59fc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.398+0000 7f5a1c898640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.398+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59fc009e50 con 0x7f59fc009110 2024-08-19T09:59:25.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a1659c640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6789/0 conn(0x7f59fc009110 legacy=0x7f59fc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 client.174120.objecter ms_handle_connect 0x7f59fc009110 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 client.174120.objecter resend_mon_ops 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920410303 0 0) 0x7f59fc009e50 con 0x7f59fc009110 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient(hunting): my global_id is 174123 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59fc0065c0 con 0x7f59fc009110 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2498356033 0 0) 0x7f59fc0065c0 con 0x7f59fc009110 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59fc009e50 con 0x7f59fc009110 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5a08005b00 con 0x7f59fc009110 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient: dump: 2024-08-19T09:59:25.416 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.417 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3365717847 0 0) 0x7f59fc009e50 con 0x7f59fc009110 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59fc00c3a0 con 0x7f59fc009110 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 2086639558) 0x7f5a08007030 con 0x7f59fc009110 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6789/0 conn(0x7f59fc009110 legacy=0x7f59fc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.401+0000 7f5a0effd640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6789/0 conn(0x7f59fc009110 legacy=0x7f59fc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.418 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15441, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:25.439 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:25.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 10 librados: watch_flush enter 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 10 librados: watch_flush exit 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 20 client.174120.objecter shutdown clearing up homeless session... 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 10 client.174120.objecter successfully canceled tick 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.82:6812/2549104415 conn(0x7f59f80416e0 legacy=0x7f59f8043ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 10 monclient: shutdown 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 >> v1:172.21.15.1:6792/0 conn(0x7f5a180a6ba0 legacy=0x7f5a181ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a16d9d640 1 -- 172.21.15.1:0/1567142778 reap_dead start 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a16d9d640 1 -- 172.21.15.1:0/1567142778 reap_dead start 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 shutdown_connections 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 >> 172.21.15.1:0/1567142778 conn(0x7f5a180ac4b0 msgr2=0x7f5a181419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 shutdown_connections 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 -- 172.21.15.1:0/1567142778 wait complete. 2024-08-19T09:59:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.402+0000 7f5a1c898640 1 librados: shutdown 2024-08-19T09:59:25.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:25.442 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:59:25.442 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.541+0000 7f2c7b6c5640 10 monclient: get_monmap_and_config 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.541+0000 7f2c7b6c5640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.541+0000 7f2c7b6c5640 10 monclient: monmap: 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:25.542576+0000 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:25.542576+0000 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 1 Processor -- start 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 1 -- start start 2024-08-19T09:59:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient: init 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient: _add_conns ranks=[3,1,6,8,5,4,7,2,0] 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient(hunting): picked mon.noname-g con 0x7f2c740a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient(hunting): picked mon.noname-f con 0x7f2c740a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient(hunting): picked mon.noname-d con 0x7f2c7413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c740a3b40 con 0x7f2c740a6ba0 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c740a8130 con 0x7f2c740a67c0 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7406ede0 con 0x7f2c7413f5c0 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.542+0000 7f2c7b6c5640 10 monclient(hunting): authenticate will time out at 1221.618953s 2024-08-19T09:59:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.543+0000 7f2c79c62640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f2c7413f5c0 0x7f2c741419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:44846/0 (socket says 172.21.15.1:44846) 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.543+0000 7f2c79c62640 1 -- 172.21.15.1:0/1775511983 learned_addr learned my addr 172.21.15.1:0/1775511983 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c79c62640 1 -- 172.21.15.1:0/1775511983 >> v1:172.21.15.1:6792/0 conn(0x7f2c7413f5c0 legacy=0x7f2c741419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3739469828 0 0) 0x7f2c7406ede0 con 0x7f2c7413f5c0 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient(hunting): my global_id is 174129 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c60003520 con 0x7f2c7413f5c0 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c79461640 1 -- 172.21.15.1:0/1775511983 >> v1:172.21.15.82:6790/0 conn(0x7f2c740a67c0 legacy=0x7f2c740a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c78c60640 1 -- 172.21.15.1:0/1775511983 >> v1:172.21.15.82:6789/0 conn(0x7f2c740a6ba0 legacy=0x7f2c740a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 38875585 0 0) 0x7f2c60003520 con 0x7f2c7413f5c0 2024-08-19T09:59:25.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 >> v1:172.21.15.82:6790/0 conn(0x7f2c740a67c0 legacy=0x7f2c740a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 >> v1:172.21.15.82:6789/0 conn(0x7f2c740a6ba0 legacy=0x7f2c740a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 monclient: found mon.noname-d 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c740a92a0 con 0x7f2c7413f5c0 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c7b6c5640 5 monclient: authenticate success, global_id 174129 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c6c001ad0 con 0x7f2c7413f5c0 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c7b6c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:25.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: dump: 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.552 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c7b6c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c6c0048a0 con 0x7f2c7413f5c0 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 1 -- 172.21.15.1:0/1775511983 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c6c004dd0 con 0x7f2c7413f5c0 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c6bfff640 10 monclient: dump: 2024-08-19T09:59:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c7b6c5640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.545+0000 7f2c7b6c5640 4 set_mon_vals no callback set 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 10 monclient: shutdown 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/1775511983 >> v1:172.21.15.1:6792/0 conn(0x7f2c7413f5c0 legacy=0x7f2c741419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/1775511983 shutdown_connections 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/1775511983 >> 172.21.15.1:0/1775511983 conn(0x7f2c740ac4b0 msgr2=0x7f2c740a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/1775511983 shutdown_connections 2024-08-19T09:59:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/1775511983 wait complete. 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.546+0000 7f2c7b6c5640 10 monclient: monmap: 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 librados: starting msgr at 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 librados: starting objecter 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 Processor -- start 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 -- start start 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 librados: setting wanted keys 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 librados: calling monclient init 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient: init 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient: _add_conns ranks=[1,0,7,6,2,4,5,8,3] 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient(hunting): picked mon.noname-f con 0x7f2c740a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2c740a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient(hunting): picked mon.noname-i con 0x7f2c7413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7413a200 con 0x7f2c740a6ba0 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c741cf020 con 0x7f2c740a67c0 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c74139b60 con 0x7f2c7413f5c0 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c7b6c5640 10 monclient(hunting): authenticate will time out at 1221.623768s 2024-08-19T09:59:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c78c60640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2c740a6ba0 0x7f2c741ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57798/0 (socket says 172.21.15.1:57798) 2024-08-19T09:59:25.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.547+0000 7f2c78c60640 1 -- 172.21.15.1:0/3049006178 learned_addr learned my addr 172.21.15.1:0/3049006178 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c78c60640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6789/0 conn(0x7f2c740a6ba0 legacy=0x7f2c741ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 client.?.objecter ms_handle_connect 0x7f2c740a6ba0 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 client.?.objecter ms_handle_connect 0x7f2c740a67c0 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c79461640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.82:6789/0 conn(0x7f2c740a67c0 legacy=0x7f2c741ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 client.?.objecter ms_handle_connect 0x7f2c7413f5c0 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c79c62640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.82:6792/0 conn(0x7f2c7413f5c0 legacy=0x7f2c741d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337667163 0 0) 0x7f2c7413a200 con 0x7f2c740a6ba0 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 174132 2024-08-19T09:59:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c58004eb0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551437433 0 0) 0x7f2c741cf020 con 0x7f2c740a67c0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 174124 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c7413a200 con 0x7f2c740a67c0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c79c62640 1 --1- 172.21.15.1:0/3049006178 >> v1:172.21.15.82:6792/0 conn(0x7f2c7413f5c0 0x7f2c741d6460 :-1 s=OPENED pgs=359 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f2c74139b60 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3189458641 0 0) 0x7f2c58004eb0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c741cf020 con 0x7f2c740a6ba0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c64003610 con 0x7f2c740a6ba0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.568 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2512114355 0 0) 0x7f2c7413a200 con 0x7f2c740a67c0 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c58004eb0 con 0x7f2c740a67c0 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c70003990 con 0x7f2c740a67c0 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 593095896 0 0) 0x7f2c741cf020 con 0x7f2c740a6ba0 2024-08-19T09:59:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.82:6792/0 conn(0x7f2c7413f5c0 legacy=0x7f2c741d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.82:6789/0 conn(0x7f2c740a67c0 legacy=0x7f2c741ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 monclient: found mon.a 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7413bbc0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 5 monclient: authenticate success, global_id 174132 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825977892 0 0) 0x7f2c74139b60 con 0x7f2c7413f5c0 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 10 monclient: _renew_subs 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c7413bea0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c64003bc0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c640040b0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c69ffb640 10 monclient: dump: 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c6b7fe640 4 set_mon_vals no callback set 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 10 client.174132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 10 monclient: _renew_subs 2024-08-19T09:59:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c7413d2d0 con 0x7f2c740a6ba0 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.550+0000 7f2c7b6c5640 1 librados: init done 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2c64004250 con 0x7f2c740a6ba0 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c7b6c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c740008d0 con 0x7f2c7409da50 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(360..360 src has 299..360) ==== 5477+0+0 (unknown 106524440 0 0) 0x7f2c6404e330 con 0x7f2c740a6ba0 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c69ffb640 10 client.174132.objecter ms_dispatch 0x7f2c74000b90 osd_map(360..360 src has 299..360) 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c69ffb640 3 client.174132.objecter handle_osd_map got epochs [360,360] > 0 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c69ffb640 3 client.174132.objecter handle_osd_map decoding full epoch 360 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.551+0000 7f2c69ffb640 20 client.174132.objecter dump_active .. 0 homeless 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.552+0000 7f2c79c62640 10 monclient: get_auth_request con 0x7f2c580414e0 auth_method 0 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.553+0000 7f2c79461640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6790/0 conn(0x7f2c7409da50 legacy=0x7f2c7409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.553+0000 7f2c69ffb640 10 client.174132.objecter ms_handle_connect 0x7f2c7409da50 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.553+0000 7f2c69ffb640 10 client.174132.objecter resend_mon_ops 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776735670 0 0) 0x7f2c740008d0 con 0x7f2c7409da50 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 164360 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c74139b60 con 0x7f2c7409da50 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 client.174132.objecter ms_handle_connect 0x7f2c580414e0 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1725562147 0 0) 0x7f2c74139b60 con 0x7f2c7409da50 2024-08-19T09:59:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c740008d0 con 0x7f2c7409da50 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c70004440 con 0x7f2c7409da50 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient: dump: 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2895886170 0 0) 0x7f2c740008d0 con 0x7f2c7409da50 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c740aa1d0 con 0x7f2c7409da50 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2c700046b0 con 0x7f2c7409da50 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6790/0 conn(0x7f2c7409da50 legacy=0x7f2c7409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.554+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6790/0 conn(0x7f2c7409da50 legacy=0x7f2c7409de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.561+0000 7f2c7b6c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.561+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c7409e920 con 0x7f2c7409e540 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c79461640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6790/0 conn(0x7f2c7409e540 legacy=0x7f2c741dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 client.174132.objecter ms_handle_connect 0x7f2c7409e540 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 client.174132.objecter resend_mon_ops 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021818421 0 0) 0x7f2c7409e920 con 0x7f2c7409e540 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 164369 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c740008d0 con 0x7f2c7409e540 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2874010755 0 0) 0x7f2c740008d0 con 0x7f2c7409e540 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c7409e920 con 0x7f2c7409e540 2024-08-19T09:59:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2c70006070 con 0x7f2c7409e540 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 10 monclient: dump: 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1054367909 0 0) 0x7f2c7409e920 con 0x7f2c7409e540 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.563+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c741df450 con 0x7f2c7409e540 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 1721974087) 0x7f2c700061a0 con 0x7f2c7409e540 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c69ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c69ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6790/0 conn(0x7f2c7409e540 legacy=0x7f2c741dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c69ffb640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6790/0 conn(0x7f2c7409e540 legacy=0x7f2c741dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 152378, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:25.600 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 10 librados: watch_flush enter 2024-08-19T09:59:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 10 librados: watch_flush exit 2024-08-19T09:59:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 20 client.174132.objecter shutdown clearing up homeless session... 2024-08-19T09:59:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 10 client.174132.objecter successfully canceled tick 2024-08-19T09:59:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2c580414e0 legacy=0x7f2c580439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 10 monclient: shutdown 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 >> v1:172.21.15.1:6789/0 conn(0x7f2c740a6ba0 legacy=0x7f2c741ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c79c62640 1 -- 172.21.15.1:0/3049006178 reap_dead start 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c79c62640 1 -- 172.21.15.1:0/3049006178 reap_dead start 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 shutdown_connections 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 >> 172.21.15.1:0/3049006178 conn(0x7f2c740ac4b0 msgr2=0x7f2c7413eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 shutdown_connections 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 -- 172.21.15.1:0/3049006178 wait complete. 2024-08-19T09:59:25.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.564+0000 7f2c7b6c5640 1 librados: shutdown 2024-08-19T09:59:25.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:25.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:59:25.604 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:59:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.704+0000 7fc67da3f640 10 monclient: get_monmap_and_config 2024-08-19T09:59:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.704+0000 7fc67da3f640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.704+0000 7fc67da3f640 10 monclient: monmap: 2024-08-19T09:59:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:25.705061+0000 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:25.705061+0000 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.706 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.704+0000 7fc67da3f640 1 Processor -- start 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.704+0000 7fc67da3f640 1 -- start start 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.704+0000 7fc67da3f640 10 monclient: init 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient: _add_conns ranks=[8,7,1,6,4,0,5,2,3] 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient(hunting): picked mon.noname-e con 0x7fc6780a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient(hunting): picked mon.noname-i con 0x7fc6780694b0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient(hunting): picked mon.noname-f con 0x7fc6780a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780a3c70 con 0x7fc6780a46a0 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780a33d0 con 0x7fc6780694b0 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780a36b0 con 0x7fc6780a6830 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc67da3f640 10 monclient(hunting): authenticate will time out at 1221.781261s 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc676ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc6780a6830 0x7fc6780a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35766/0 (socket says 172.21.15.1:35766) 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.705+0000 7fc676ffd640 1 -- 172.21.15.1:0/1464482654 learned_addr learned my addr 172.21.15.1:0/1464482654 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc676ffd640 1 -- 172.21.15.1:0/1464482654 >> v1:172.21.15.1:6793/0 conn(0x7fc6780a6830 legacy=0x7fc6780a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc6767fc640 1 -- 172.21.15.1:0/1464482654 >> v1:172.21.15.82:6792/0 conn(0x7fc6780694b0 legacy=0x7fc6780a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 175839266 0 0) 0x7fc6780a36b0 con 0x7fc6780a6830 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient(hunting): my global_id is 174113 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc6777fe640 1 -- 172.21.15.1:0/1464482654 >> v1:172.21.15.82:6789/0 conn(0x7fc6780a46a0 legacy=0x7fc678141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc664003520 con 0x7fc6780a6830 2024-08-19T09:59:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2987045986 0 0) 0x7fc664003520 con 0x7fc6780a6830 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 >> v1:172.21.15.82:6792/0 conn(0x7fc6780694b0 legacy=0x7fc6780a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 >> v1:172.21.15.82:6789/0 conn(0x7fc6780a46a0 legacy=0x7fc678141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 monclient: found mon.noname-e 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67806dbb0 con 0x7fc6780a6830 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc670001e60 con 0x7fc6780a6830 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:59:25.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc675ffb640 10 monclient: dump: 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc67da3f640 5 monclient: authenticate success, global_id 174113 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.707+0000 7fc67da3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc670004c20 con 0x7fc6780a6830 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc675ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc675ffb640 1 -- 172.21.15.1:0/1464482654 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc670005190 con 0x7fc6780a6830 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 4 set_mon_vals no callback set 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc675ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc675ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc675ffb640 10 monclient: dump: 2024-08-19T09:59:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 10 monclient: shutdown 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 1 -- 172.21.15.1:0/1464482654 >> v1:172.21.15.1:6793/0 conn(0x7fc6780a6830 legacy=0x7fc6780a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 1 -- 172.21.15.1:0/1464482654 shutdown_connections 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 1 -- 172.21.15.1:0/1464482654 >> 172.21.15.1:0/1464482654 conn(0x7fc6780a9020 msgr2=0x7fc678068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 1 -- 172.21.15.1:0/1464482654 shutdown_connections 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.708+0000 7fc67da3f640 1 -- 172.21.15.1:0/1464482654 wait complete. 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 10 monclient: monmap: 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 1 librados: starting msgr at 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 1 librados: starting objecter 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 1 Processor -- start 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 1 -- start start 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 1 librados: setting wanted keys 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 1 librados: calling monclient init 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.709+0000 7fc67da3f640 10 monclient: init 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient: _add_conns ranks=[3,4,1,2,6,7,5,8,0] 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient(hunting): picked mon.noname-g con 0x7fc6780694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient(hunting): picked mon.noname-c con 0x7fc6780a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient(hunting): picked mon.noname-f con 0x7fc6780a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc678139630 con 0x7fc6780a6830 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc678138700 con 0x7fc6780694b0 2024-08-19T09:59:25.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc67813ae30 con 0x7fc6780a46a0 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc67da3f640 10 monclient(hunting): authenticate will time out at 1221.786259s 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc6767fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fc6780a46a0 0x7fc6781ceac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54222/0 (socket says 172.21.15.1:54222) 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.710+0000 7fc6767fc640 1 -- 172.21.15.1:0/2391216012 learned_addr learned my addr 172.21.15.1:0/2391216012 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc6767fc640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc6780a46a0 legacy=0x7fc6781ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 client.?.objecter ms_handle_connect 0x7fc6780a46a0 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 client.?.objecter ms_handle_connect 0x7fc6780a6830 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc6777fe640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.82:6789/0 conn(0x7fc6780a6830 legacy=0x7fc6781d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 client.?.objecter ms_handle_connect 0x7fc6780694b0 2024-08-19T09:59:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc676ffd640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.82:6790/0 conn(0x7fc6780694b0 legacy=0x7fc6781ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600650488 0 0) 0x7fc678139630 con 0x7fc6780a6830 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): my global_id is 174142 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc658004eb0 con 0x7fc6780a6830 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235590702 0 0) 0x7fc67813ae30 con 0x7fc6780a46a0 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): my global_id is 27436 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc678139630 con 0x7fc6780a46a0 2024-08-19T09:59:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4205601264 0 0) 0x7fc678138700 con 0x7fc6780694b0 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): my global_id is 135138 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc67813ae30 con 0x7fc6780694b0 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2941302680 0 0) 0x7fc658004eb0 con 0x7fc6780a6830 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc678138700 con 0x7fc6780a6830 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc668001c80 con 0x7fc6780a6830 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): dump: 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.732 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3586247778 0 0) 0x7fc678139630 con 0x7fc6780a46a0 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc658004eb0 con 0x7fc6780a46a0 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc660003610 con 0x7fc6780a46a0 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.712+0000 7fc66ffff640 10 monclient(hunting): dump: 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2536567894 0 0) 0x7fc67813ae30 con 0x7fc6780694b0 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc678139630 con 0x7fc6780694b0 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc6700020b0 con 0x7fc6780694b0 2024-08-19T09:59:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient(hunting): dump: 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.736 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1847129185 0 0) 0x7fc678138700 con 0x7fc6780a6830 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc6780a46a0 legacy=0x7fc6781ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.82:6790/0 conn(0x7fc6780694b0 legacy=0x7fc6781ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 monclient: found mon.f 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6781cf1d0 con 0x7fc6780a6830 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 5 monclient: authenticate success, global_id 174142 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 10 monclient: _renew_subs 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6780a9f30 con 0x7fc6780a6830 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 10 client.174142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 10 monclient: _renew_subs 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc678137cc0 con 0x7fc6780a6830 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 1 librados: init done 2024-08-19T09:59:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6680038c0 con 0x7fc6780a6830 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc668003e30 con 0x7fc6780a6830 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc6757fa640 4 set_mon_vals no callback set 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc66ffff640 10 monclient: dump: 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.713+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc65c0065c0 con 0x7fc65c002860 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc668004080 con 0x7fc6780a6830 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(360..360 src has 299..360) ==== 5477+0+0 (unknown 106524440 0 0) 0x7fc66804e090 con 0x7fc6780a6830 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc66ffff640 10 client.174142.objecter ms_dispatch 0x7fc678000b90 osd_map(360..360 src has 299..360) 2024-08-19T09:59:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc66ffff640 3 client.174142.objecter handle_osd_map got epochs [360,360] > 0 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc66ffff640 3 client.174142.objecter handle_osd_map decoding full epoch 360 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc66ffff640 20 client.174142.objecter dump_active .. 0 homeless 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.714+0000 7fc6767fc640 10 monclient: get_auth_request con 0x7fc658041a70 auth_method 0 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 client.174142.objecter ms_handle_connect 0x7fc65c002860 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc676ffd640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc65c002860 legacy=0x7fc65c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 client.174142.objecter resend_mon_ops 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2731785716 0 0) 0x7fc65c0065c0 con 0x7fc65c002860 2024-08-19T09:59:25.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient(hunting): my global_id is 27445 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc678138700 con 0x7fc65c002860 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3171643740 0 0) 0x7fc678138700 con 0x7fc65c002860 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc65c0065c0 con 0x7fc65c002860 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc670001900 con 0x7fc65c002860 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient: dump: 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.743 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1251893842 0 0) 0x7fc65c0065c0 con 0x7fc65c002860 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc65c00a160 con 0x7fc65c002860 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 client.174142.objecter ms_handle_connect 0x7fc658041a70 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc670005450 con 0x7fc65c002860 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc65c002860 legacy=0x7fc65c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.716+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc65c002860 legacy=0x7fc65c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.723+0000 7fc67da3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:25.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.723+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc65c009e70 con 0x7fc65c009130 2024-08-19T09:59:25.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc676ffd640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc65c009130 legacy=0x7fc65c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 client.174142.objecter ms_handle_connect 0x7fc65c009130 2024-08-19T09:59:25.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 client.174142.objecter resend_mon_ops 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731713271 0 0) 0x7fc65c009e70 con 0x7fc65c009130 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient(hunting): my global_id is 27454 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc65c0065c0 con 0x7fc65c009130 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3962860887 0 0) 0x7fc65c0065c0 con 0x7fc65c009130 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc65c009e70 con 0x7fc65c009130 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc670005bb0 con 0x7fc65c009130 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 10 monclient: dump: 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3937085244 0 0) 0x7fc65c009e70 con 0x7fc65c009130 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.725+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc65c00c3f0 con 0x7fc65c009130 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 227762205) 0x7fc670006030 con 0x7fc65c009130 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc66ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc66ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc65c009130 legacy=0x7fc65c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc66ffff640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.1:6791/0 conn(0x7fc65c009130 legacy=0x7fc65c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.749 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 571996, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:25.750 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.751 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.752 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:25.753 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:25.754 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.755 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:25.757 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.759 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:25.760 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.761 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.765 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.766 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.767 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:25.770 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 10 librados: watch_flush enter 2024-08-19T09:59:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 10 librados: watch_flush exit 2024-08-19T09:59:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 20 client.174142.objecter shutdown clearing up homeless session... 2024-08-19T09:59:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 10 client.174142.objecter successfully canceled tick 2024-08-19T09:59:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc658041a70 legacy=0x7fc658043f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 10 monclient: shutdown 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 >> v1:172.21.15.82:6789/0 conn(0x7fc6780a6830 legacy=0x7fc6781d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc6777fe640 1 -- 172.21.15.1:0/2391216012 reap_dead start 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc6777fe640 1 -- 172.21.15.1:0/2391216012 reap_dead start 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 shutdown_connections 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 >> 172.21.15.1:0/2391216012 conn(0x7fc6780a9020 msgr2=0x7fc6780a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 shutdown_connections 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 -- 172.21.15.1:0/2391216012 wait complete. 2024-08-19T09:59:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.726+0000 7fc67da3f640 1 librados: shutdown 2024-08-19T09:59:25.774 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:25.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:59:25.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:59:25.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.876+0000 7f1393ede640 10 monclient: get_monmap_and_config 2024-08-19T09:59:25.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.876+0000 7f1393ede640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.876+0000 7f1393ede640 10 monclient: monmap: 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:25.877029+0000 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:25.877029+0000 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.876+0000 7f1393ede640 1 Processor -- start 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.876+0000 7f1393ede640 1 -- start start 2024-08-19T09:59:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.876+0000 7f1393ede640 10 monclient: init 2024-08-19T09:59:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient: _add_conns ranks=[2,5,4,3,7,8,1,0,6] 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient(hunting): picked mon.noname-b con 0x7f138c0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient(hunting): picked mon.noname-h con 0x7f138c0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient(hunting): picked mon.noname-c con 0x7f138c0a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c0a3c70 con 0x7f138c0a6830 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c0a33d0 con 0x7f138c0a46a0 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c0a36b0 con 0x7f138c0694b0 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1393ede640 10 monclient(hunting): authenticate will time out at 1221.953414s 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1391c7a640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f138c0a6830 0x7f138c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39830/0 (socket says 172.21.15.1:39830) 2024-08-19T09:59:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f1391c7a640 1 -- 172.21.15.1:0/947267567 learned_addr learned my addr 172.21.15.1:0/947267567 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.877+0000 7f139247b640 1 --1- 172.21.15.1:0/947267567 >> v1:172.21.15.1:6791/0 conn(0x7f138c0a46a0 0x7f138c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54254/0 (socket says 172.21.15.1:54254) 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1391c7a640 1 -- 172.21.15.1:0/947267567 >> v1:172.21.15.1:6790/0 conn(0x7f138c0a6830 legacy=0x7f138c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f139247b640 1 -- 172.21.15.1:0/947267567 >> v1:172.21.15.1:6791/0 conn(0x7f138c0a46a0 legacy=0x7f138c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1391479640 1 -- 172.21.15.1:0/947267567 >> v1:172.21.15.82:6791/0 conn(0x7f138c0694b0 legacy=0x7f138c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598397106 0 0) 0x7f138c0a3c70 con 0x7f138c0a6830 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): my global_id is 164378 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1374003520 con 0x7f138c0a6830 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277155237 0 0) 0x7f138c0a33d0 con 0x7f138c0a46a0 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 10 monclient(hunting): my global_id is 27463 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.879+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138c0a3c70 con 0x7f138c0a46a0 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046860253 0 0) 0x7f138c0a36b0 con 0x7f138c0694b0 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient(hunting): my global_id is 27131 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138c0a33d0 con 0x7f138c0694b0 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 635412676 0 0) 0x7f138c0a3c70 con 0x7f138c0a46a0 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 >> v1:172.21.15.82:6791/0 conn(0x7f138c0694b0 legacy=0x7f138c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 >> v1:172.21.15.1:6790/0 conn(0x7f138c0a6830 legacy=0x7f138c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 monclient: found mon.noname-c 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c06dbb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1384001ac0 con 0x7f138c0a46a0 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: dump: 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 5 monclient: authenticate success, global_id 27463 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13840047e0 con 0x7f138c0a46a0 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 1 -- 172.21.15.1:0/947267567 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1384004d50 con 0x7f138c0a46a0 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1390c78640 10 monclient: dump: 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 4 set_mon_vals no callback set 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 10 monclient: shutdown 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 1 -- 172.21.15.1:0/947267567 >> v1:172.21.15.1:6791/0 conn(0x7f138c0a46a0 legacy=0x7f138c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 1 -- 172.21.15.1:0/947267567 shutdown_connections 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 1 -- 172.21.15.1:0/947267567 >> 172.21.15.1:0/947267567 conn(0x7f138c0a9020 msgr2=0x7f138c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 1 -- 172.21.15.1:0/947267567 shutdown_connections 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.880+0000 7f1393ede640 1 -- 172.21.15.1:0/947267567 wait complete. 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 10 monclient: build_initial_monmap 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 10 monclient: monmap: 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 1 librados: starting msgr at 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 1 librados: starting objecter 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 1 Processor -- start 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 1 -- start start 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 1 librados: setting wanted keys 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 1 librados: calling monclient init 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.881+0000 7f1393ede640 10 monclient: init 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient: _add_conns ranks=[4,6,1,2,3,0,8,5,7] 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient(hunting): picked mon.noname-c con 0x7f138c0694b0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient(hunting): picked mon.noname-d con 0x7f138c0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient(hunting): picked mon.noname-f con 0x7f138c0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c1385a0 con 0x7f138c0a6830 2024-08-19T09:59:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c135cb0 con 0x7f138c0694b0 2024-08-19T09:59:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c142bb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1393ede640 10 monclient(hunting): authenticate will time out at 1221.958278s 2024-08-19T09:59:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1391c7a640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f138c0694b0 0x7f138c134e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54260/0 (socket says 172.21.15.1:54260) 2024-08-19T09:59:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.882+0000 7f1391c7a640 1 -- 172.21.15.1:0/789183772 learned_addr learned my addr 172.21.15.1:0/789183772 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f1391c7a640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6791/0 conn(0x7f138c0694b0 legacy=0x7f138c134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 client.?.objecter ms_handle_connect 0x7f138c0694b0 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 client.?.objecter ms_handle_connect 0x7f138c0a46a0 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f1391479640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f138c0a46a0 legacy=0x7f138c1355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 client.?.objecter ms_handle_connect 0x7f138c0a6830 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f139247b640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.82:6789/0 conn(0x7f138c0a6830 legacy=0x7f138c1d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893725870 0 0) 0x7f138c135cb0 con 0x7f138c0694b0 2024-08-19T09:59:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): my global_id is 27472 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1378004eb0 con 0x7f138c0694b0 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084161949 0 0) 0x7f138c142bb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): my global_id is 174138 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138c135cb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150586234 0 0) 0x7f138c1385a0 con 0x7f138c0a6830 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): my global_id is 174151 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138c142bb0 con 0x7f138c0a6830 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4162683351 0 0) 0x7f1378004eb0 con 0x7f138c0694b0 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f138c1385a0 con 0x7f138c0694b0 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f137c001d50 con 0x7f138c0694b0 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2569064623 0 0) 0x7f138c135cb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1378004eb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1380001cb0 con 0x7f138c0a46a0 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.884+0000 7f138a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1819599497 0 0) 0x7f138c1385a0 con 0x7f138c0694b0 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f138c0a46a0 legacy=0x7f138c1355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.82:6789/0 conn(0x7f138c0a6830 legacy=0x7f138c1d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 monclient: found mon.c 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c138b60 con 0x7f138c0694b0 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 5 monclient: authenticate success, global_id 27472 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 10 monclient: _renew_subs 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f138c136ee0 con 0x7f138c0694b0 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 10 client.27472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 10 monclient: _renew_subs 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f138c142570 con 0x7f138c0694b0 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 1 librados: init done 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f137c005150 con 0x7f138c0694b0 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f137c005640 con 0x7f138c0694b0 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138bfff640 4 set_mon_vals no callback set 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f138a7fc640 10 monclient: dump: 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.908 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.885+0000 7f1393ede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f136c0065c0 con 0x7f136c002860 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f137c011aa0 con 0x7f138c0694b0 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(360..360 src has 299..360) ==== 5477+0+0 (unknown 106524440 0 0) 0x7f137c04e630 con 0x7f138c0694b0 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f138a7fc640 10 client.27472.objecter ms_dispatch 0x7f138c000b90 osd_map(360..360 src has 299..360) 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f138a7fc640 3 client.27472.objecter handle_osd_map got epochs [360,360] > 0 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f138a7fc640 3 client.27472.objecter handle_osd_map decoding full epoch 360 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f138a7fc640 20 client.27472.objecter dump_active .. 0 homeless 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.886+0000 7f139247b640 10 monclient: get_auth_request con 0x7f1378041720 auth_method 0 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 client.27472.objecter ms_handle_connect 0x7f136c002860 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 client.27472.objecter resend_mon_ops 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f1391479640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f136c002860 legacy=0x7f136c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035438143 0 0) 0x7f136c0065c0 con 0x7f136c002860 2024-08-19T09:59:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient(hunting): my global_id is 174147 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138c1385a0 con 0x7f136c002860 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4062377528 0 0) 0x7f138c1385a0 con 0x7f136c002860 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f136c0065c0 con 0x7f136c002860 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1380003c30 con 0x7f136c002860 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 monclient: dump: 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.911 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3726269063 0 0) 0x7f136c0065c0 con 0x7f136c002860 2024-08-19T09:59:25.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f136c00a160 con 0x7f136c002860 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.888+0000 7f138a7fc640 10 client.27472.objecter ms_handle_connect 0x7f1378041720 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.889+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1380003f80 con 0x7f136c002860 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.889+0000 7f138a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.889+0000 7f138a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.889+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f136c002860 legacy=0x7f136c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.889+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f136c002860 legacy=0x7f136c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.895+0000 7f1393ede640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.895+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f136c009e50 con 0x7f136c009110 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f1391479640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f136c009110 legacy=0x7f136c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 10 client.27472.objecter ms_handle_connect 0x7f136c009110 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 10 client.27472.objecter resend_mon_ops 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841377911 0 0) 0x7f136c009e50 con 0x7f136c009110 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:25.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 10 monclient(hunting): my global_id is 174156 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f136c0065c0 con 0x7f136c009110 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.897+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 371147029 0 0) 0x7f136c0065c0 con 0x7f136c009110 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f136c009e50 con 0x7f136c009110 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1380004430 con 0x7f136c009110 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 10 monclient: dump: 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:25.914 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:25.919 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3222635896 0 0) 0x7f136c009e50 con 0x7f136c009110 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f136c00c3a0 con 0x7f136c009110 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 1074333462) 0x7f1380007bc0 con 0x7f136c009110 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f136c009110 legacy=0x7f136c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f138a7fc640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6792/0 conn(0x7f136c009110 legacy=0x7f136c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:25.920 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15931, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:25.921 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.922 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.924 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.924 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.924 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:25.925 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:25.926 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:25.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:25.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.937 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:25.939 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:25.940 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:25.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:25.942 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 10 librados: watch_flush enter 2024-08-19T09:59:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 10 librados: watch_flush exit 2024-08-19T09:59:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 20 client.27472.objecter shutdown clearing up homeless session... 2024-08-19T09:59:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 10 client.27472.objecter successfully canceled tick 2024-08-19T09:59:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1378041720 legacy=0x7f1378043be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 10 monclient: shutdown 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 >> v1:172.21.15.1:6791/0 conn(0x7f138c0694b0 legacy=0x7f138c134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f139247b640 1 -- 172.21.15.1:0/789183772 reap_dead start 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f139247b640 1 -- 172.21.15.1:0/789183772 reap_dead start 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 shutdown_connections 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 >> 172.21.15.1:0/789183772 conn(0x7f138c0a9020 msgr2=0x7f138c133930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 shutdown_connections 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.898+0000 7f1393ede640 1 -- 172.21.15.1:0/789183772 wait complete. 2024-08-19T09:59:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:25.899+0000 7f1393ede640 1 librados: shutdown 2024-08-19T09:59:25.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:25.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:59:25.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:59:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.051+0000 7f821b9cf640 10 monclient: get_monmap_and_config 2024-08-19T09:59:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.051+0000 7f821b9cf640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.051+0000 7f821b9cf640 10 monclient: monmap: 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:26.052694+0000 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:26.052694+0000 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.054 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.052+0000 7f821b9cf640 1 Processor -- start 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.052+0000 7f821b9cf640 1 -- start start 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.052+0000 7f821b9cf640 10 monclient: init 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.052+0000 7f821b9cf640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.052+0000 7f821b9cf640 10 monclient: _add_conns ranks=[3,7,6,8,4,0,1,2,5] 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.052+0000 7f821b9cf640 10 monclient(hunting): picked mon.noname-g con 0x7f82140a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 10 monclient(hunting): picked mon.noname-i con 0x7f82140a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 10 monclient(hunting): picked mon.noname-d con 0x7f821413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140a3b40 con 0x7f82140a67c0 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140a8130 con 0x7f821413f5c0 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821406ede0 con 0x7f82140a6ba0 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f821b9cf640 10 monclient(hunting): authenticate will time out at 1222.129060s 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f8219f6c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f821413f5c0 0x7f82141419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:44898/0 (socket says 172.21.15.1:44898) 2024-08-19T09:59:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.053+0000 7f8219f6c640 1 -- 172.21.15.1:0/2161008640 learned_addr learned my addr 172.21.15.1:0/2161008640 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f8219f6c640 1 -- 172.21.15.1:0/2161008640 >> v1:172.21.15.1:6792/0 conn(0x7f821413f5c0 legacy=0x7f82141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f8218f6a640 1 -- 172.21.15.1:0/2161008640 >> v1:172.21.15.82:6792/0 conn(0x7f82140a6ba0 legacy=0x7f82140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f821976b640 1 -- 172.21.15.1:0/2161008640 >> v1:172.21.15.82:6790/0 conn(0x7f82140a67c0 legacy=0x7f82140a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201592295 0 0) 0x7f82140a8130 con 0x7f821413f5c0 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): my global_id is 174165 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8200003520 con 0x7f821413f5c0 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404780442 0 0) 0x7f821406ede0 con 0x7f82140a6ba0 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): my global_id is 144628 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82140a8130 con 0x7f82140a6ba0 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4283320149 0 0) 0x7f82140a3b40 con 0x7f82140a67c0 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient(hunting): my global_id is 135147 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f821406ede0 con 0x7f82140a67c0 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1141698512 0 0) 0x7f8200003520 con 0x7f821413f5c0 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 >> v1:172.21.15.82:6792/0 conn(0x7f82140a6ba0 legacy=0x7f82140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 >> v1:172.21.15.82:6790/0 conn(0x7f82140a67c0 legacy=0x7f82140a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 monclient: found mon.noname-d 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82140a92a0 con 0x7f821413f5c0 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f821b9cf640 5 monclient: authenticate success, global_id 174165 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f821b9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f820c001e60 con 0x7f821413f5c0 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: dump: 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f821b9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f820c004c20 con 0x7f821413f5c0 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f821b9cf640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 1 -- 172.21.15.1:0/2161008640 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f820c005190 con 0x7f821413f5c0 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f821b9cf640 4 set_mon_vals no callback set 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.055+0000 7f820bfff640 10 monclient: dump: 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 10 monclient: shutdown 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 1 -- 172.21.15.1:0/2161008640 >> v1:172.21.15.1:6792/0 conn(0x7f821413f5c0 legacy=0x7f82141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 1 -- 172.21.15.1:0/2161008640 shutdown_connections 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 1 -- 172.21.15.1:0/2161008640 >> 172.21.15.1:0/2161008640 conn(0x7f82140ac4b0 msgr2=0x7f82140a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 1 -- 172.21.15.1:0/2161008640 shutdown_connections 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.056+0000 7f821b9cf640 1 -- 172.21.15.1:0/2161008640 wait complete. 2024-08-19T09:59:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient: monmap: 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 librados: starting msgr at 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 librados: starting objecter 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 Processor -- start 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 -- start start 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 librados: setting wanted keys 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 librados: calling monclient init 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient: init 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient: _add_conns ranks=[0,6,7,4,2,1,8,5,3] 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient(hunting): picked mon.noname-a con 0x7f82140a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient(hunting): picked mon.noname-d con 0x7f82140a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient(hunting): picked mon.noname-i con 0x7f821413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821413ae60 con 0x7f82140a67c0 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821413bcd0 con 0x7f82140a6ba0 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82141cf020 con 0x7f821413f5c0 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.057+0000 7f821b9cf640 10 monclient(hunting): authenticate will time out at 1222.133991s 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.058+0000 7f821976b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82140a67c0 0x7f82141ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57824/0 (socket says 172.21.15.1:57824) 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.058+0000 7f821976b640 1 -- 172.21.15.1:0/354593286 learned_addr learned my addr 172.21.15.1:0/354593286 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.058+0000 7f8218f6a640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f82140a6ba0 0x7f82141ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:44906/0 (socket says 172.21.15.1:44906) 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8218f6a640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6792/0 conn(0x7f82140a6ba0 legacy=0x7f82141ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f821976b640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6789/0 conn(0x7f82140a67c0 legacy=0x7f82141ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 client.?.objecter ms_handle_connect 0x7f82140a6ba0 2024-08-19T09:59:26.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 client.?.objecter ms_handle_connect 0x7f82140a67c0 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8219f6c640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.82:6792/0 conn(0x7f821413f5c0 legacy=0x7f82141d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 client.?.objecter ms_handle_connect 0x7f821413f5c0 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247695951 0 0) 0x7f821413bcd0 con 0x7f82140a6ba0 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): my global_id is 174174 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81f8004eb0 con 0x7f82140a6ba0 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834796148 0 0) 0x7f821413ae60 con 0x7f82140a67c0 2024-08-19T09:59:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): my global_id is 174141 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f821413bcd0 con 0x7f82140a67c0 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728986847 0 0) 0x7f82141cf020 con 0x7f821413f5c0 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): my global_id is 144637 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f821413ae60 con 0x7f821413f5c0 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 457303156 0 0) 0x7f81f8004eb0 con 0x7f82140a6ba0 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82141cf020 con 0x7f82140a6ba0 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8204003610 con 0x7f82140a6ba0 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3257689044 0 0) 0x7f821413bcd0 con 0x7f82140a67c0 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81f8004eb0 con 0x7f82140a67c0 2024-08-19T09:59:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3368202893 0 0) 0x7f821413ae60 con 0x7f821413f5c0 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f821413bcd0 con 0x7f821413f5c0 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8210003950 con 0x7f82140a67c0 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f820c0020b0 con 0x7f821413f5c0 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2294691066 0 0) 0x7f82141cf020 con 0x7f82140a6ba0 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.82:6792/0 conn(0x7f821413f5c0 legacy=0x7f82141d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6789/0 conn(0x7f82140a67c0 legacy=0x7f82141ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 monclient: found mon.d 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8214139970 con 0x7f82140a6ba0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f8209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f821b9cf640 5 monclient: authenticate success, global_id 174174 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f821b9cf640 10 monclient: _renew_subs 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f821b9cf640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.060+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f821413b430 con 0x7f82140a6ba0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8204003b60 con 0x7f82140a6ba0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8204004090 con 0x7f82140a6ba0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 10 monclient: dump: 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f820b7fe640 4 set_mon_vals no callback set 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 10 client.174174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 10 monclient: _renew_subs 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f821413e540 con 0x7f82140a6ba0 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 1 librados: init done 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f82140008d0 con 0x7f821413b650 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.061+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8204004230 con 0x7f82140a6ba0 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.062+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(361..361 src has 299..361) ==== 5064+0+0 (unknown 3986656978 0 0) 0x7f820404dff0 con 0x7f82140a6ba0 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.062+0000 7f8209ffb640 10 client.174174.objecter ms_dispatch 0x7f8214000b90 osd_map(361..361 src has 299..361) 2024-08-19T09:59:26.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.062+0000 7f8209ffb640 3 client.174174.objecter handle_osd_map got epochs [361,361] > 0 2024-08-19T09:59:26.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.062+0000 7f8209ffb640 3 client.174174.objecter handle_osd_map decoding full epoch 361 2024-08-19T09:59:26.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.062+0000 7f8209ffb640 20 client.174174.objecter dump_active .. 0 homeless 2024-08-19T09:59:26.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.062+0000 7f8219f6c640 10 monclient: get_auth_request con 0x7f81f8041b50 auth_method 0 2024-08-19T09:59:26.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.064+0000 7f8209ffb640 10 client.174174.objecter ms_handle_connect 0x7f821413b650 2024-08-19T09:59:26.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.064+0000 7f8209ffb640 10 client.174174.objecter resend_mon_ops 2024-08-19T09:59:26.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.064+0000 7f821976b640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6793/0 conn(0x7f821413b650 legacy=0x7f821409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.064+0000 7f8209ffb640 10 client.174174.objecter ms_handle_connect 0x7f81f8041b50 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024803769 0 0) 0x7f82140008d0 con 0x7f821413b650 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient(hunting): my global_id is 174122 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82141cf020 con 0x7f821413b650 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 840481096 0 0) 0x7f82141cf020 con 0x7f821413b650 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82140008d0 con 0x7f821413b650 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8210003e30 con 0x7f821413b650 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 10 monclient: dump: 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 475172202 0 0) 0x7f82140008d0 con 0x7f821413b650 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.066+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f821413d100 con 0x7f821413b650 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.067+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f82100040a0 con 0x7f821413b650 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.067+0000 7f8209ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.067+0000 7f8209ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.067+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6793/0 conn(0x7f821413b650 legacy=0x7f821409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.067+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6793/0 conn(0x7f821413b650 legacy=0x7f821409da50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.073+0000 7f821b9cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.073+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f821413d2b0 con 0x7f821402e1d0 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 client.174174.objecter ms_handle_connect 0x7f821402e1d0 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f821976b640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6793/0 conn(0x7f821402e1d0 legacy=0x7f821409e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 client.174174.objecter resend_mon_ops 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012485100 0 0) 0x7f821413d2b0 con 0x7f821402e1d0 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient(hunting): my global_id is 174131 2024-08-19T09:59:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82140008d0 con 0x7f821402e1d0 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 215340398 0 0) 0x7f82140008d0 con 0x7f821402e1d0 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f821413d2b0 con 0x7f821402e1d0 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8210005b00 con 0x7f821402e1d0 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.075+0000 7f8209ffb640 10 monclient: dump: 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 286220164 0 0) 0x7f821413d2b0 con 0x7f821402e1d0 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f821402e5b0 con 0x7f821402e1d0 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3888857261) 0x7f8210005e50 con 0x7f821402e1d0 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6793/0 conn(0x7f821402e1d0 legacy=0x7f821409e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8209ffb640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6793/0 conn(0x7f821402e1d0 legacy=0x7f821409e160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16124, 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:26.095 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.096 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.097 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:26.098 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:26.099 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.100 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.101 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.103 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.104 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.106 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:26.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:26.110 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.113 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.114 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.114 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.114 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:26.114 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 10 librados: watch_flush enter 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 10 librados: watch_flush exit 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 20 client.174174.objecter shutdown clearing up homeless session... 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 10 client.174174.objecter successfully canceled tick 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.82:6812/2549104415 conn(0x7f81f8041b50 legacy=0x7f81f8044010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 10 monclient: shutdown 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 >> v1:172.21.15.1:6792/0 conn(0x7f82140a6ba0 legacy=0x7f82141ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8219f6c640 1 -- 172.21.15.1:0/354593286 reap_dead start 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f8219f6c640 1 -- 172.21.15.1:0/354593286 reap_dead start 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 shutdown_connections 2024-08-19T09:59:26.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 >> 172.21.15.1:0/354593286 conn(0x7f82140ac4b0 msgr2=0x7f8214141cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 shutdown_connections 2024-08-19T09:59:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 -- 172.21.15.1:0/354593286 wait complete. 2024-08-19T09:59:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.076+0000 7f821b9cf640 1 librados: shutdown 2024-08-19T09:59:26.117 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:26.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:59:26.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:59:26.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.216+0000 7f9f3036b640 10 monclient: get_monmap_and_config 2024-08-19T09:59:26.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.216+0000 7f9f3036b640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.216+0000 7f9f3036b640 10 monclient: monmap: 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:26.217578+0000 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:26.217578+0000 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.219 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.219 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.219 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 1 Processor -- start 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 1 -- start start 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient: init 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient: _add_conns ranks=[5,2,6,4,7,0,8,1,3] 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient(hunting): picked mon.noname-h con 0x7f9f280a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f280a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient(hunting): picked mon.noname-d con 0x7f9f2813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f280a3b40 con 0x7f9f280a6ba0 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f280a8130 con 0x7f9f280a67c0 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f2806ede0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.217+0000 7f9f3036b640 10 monclient(hunting): authenticate will time out at 1222.293929s 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.218+0000 7f9f2d906640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f9f280a6ba0 0x7f9f280a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39834/0 (socket says 172.21.15.1:39834) 2024-08-19T09:59:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.218+0000 7f9f2d906640 1 -- 172.21.15.1:0/75866676 learned_addr learned my addr 172.21.15.1:0/75866676 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d906640 1 -- 172.21.15.1:0/75866676 >> v1:172.21.15.1:6790/0 conn(0x7f9f280a6ba0 legacy=0x7f9f280a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2e908640 1 -- 172.21.15.1:0/75866676 >> v1:172.21.15.1:6792/0 conn(0x7f9f2813f5c0 legacy=0x7f9f281419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2e107640 1 -- 172.21.15.1:0/75866676 >> v1:172.21.15.82:6791/0 conn(0x7f9f280a67c0 legacy=0x7f9f280a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009992243 0 0) 0x7f9f280a3b40 con 0x7f9f280a6ba0 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): my global_id is 164387 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f14003520 con 0x7f9f280a6ba0 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873949133 0 0) 0x7f9f2806ede0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): my global_id is 174183 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f280a3b40 con 0x7f9f2813f5c0 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444955336 0 0) 0x7f9f280a8130 con 0x7f9f280a67c0 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient(hunting): my global_id is 27140 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f2806ede0 con 0x7f9f280a67c0 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2880732113 0 0) 0x7f9f280a3b40 con 0x7f9f2813f5c0 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 >> v1:172.21.15.82:6791/0 conn(0x7f9f280a67c0 legacy=0x7f9f280a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 >> v1:172.21.15.1:6790/0 conn(0x7f9f280a6ba0 legacy=0x7f9f280a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 monclient: found mon.noname-d 2024-08-19T09:59:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f280a92a0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f20001cd0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f3036b640 5 monclient: authenticate success, global_id 174183 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f3036b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: dump: 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f3036b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f20001fa0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 1 -- 172.21.15.1:0/75866676 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f20003cd0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f3036b640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f3036b640 4 set_mon_vals no callback set 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.220+0000 7f9f2d105640 10 monclient: dump: 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 10 monclient: shutdown 2024-08-19T09:59:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 1 -- 172.21.15.1:0/75866676 >> v1:172.21.15.1:6792/0 conn(0x7f9f2813f5c0 legacy=0x7f9f281419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 1 -- 172.21.15.1:0/75866676 shutdown_connections 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 1 -- 172.21.15.1:0/75866676 >> 172.21.15.1:0/75866676 conn(0x7f9f280ac4b0 msgr2=0x7f9f280a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 1 -- 172.21.15.1:0/75866676 shutdown_connections 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 1 -- 172.21.15.1:0/75866676 wait complete. 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.221+0000 7f9f3036b640 10 monclient: monmap: 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 librados: starting msgr at 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 librados: starting objecter 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 Processor -- start 2024-08-19T09:59:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 -- start start 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 librados: setting wanted keys 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 librados: calling monclient init 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient: init 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient: _add_conns ranks=[1,6,8,0,4,5,2,7,3] 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient(hunting): picked mon.noname-f con 0x7f9f280a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient(hunting): picked mon.noname-d con 0x7f9f280a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient(hunting): picked mon.noname-e con 0x7f9f2813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f28142c60 con 0x7f9f280a67c0 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f281cf020 con 0x7f9f280a6ba0 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f280a9dc0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f3036b640 10 monclient(hunting): authenticate will time out at 1222.298779s 2024-08-19T09:59:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f2d906640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f9f280a6ba0 0x7f9f281ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:44922/0 (socket says 172.21.15.1:44922) 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.222+0000 7f9f2d906640 1 -- 172.21.15.1:0/3546860339 learned_addr learned my addr 172.21.15.1:0/3546860339 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.224+0000 7f9f2d906640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.1:6792/0 conn(0x7f9f280a6ba0 legacy=0x7f9f281ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 client.?.objecter ms_handle_connect 0x7f9f280a6ba0 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 client.?.objecter ms_handle_connect 0x7f9f2813f5c0 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f2e908640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.1:6793/0 conn(0x7f9f2813f5c0 legacy=0x7f9f281d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f2e107640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f280a67c0 legacy=0x7f9f281ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 client.?.objecter ms_handle_connect 0x7f9f280a67c0 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139184039 0 0) 0x7f9f281cf020 con 0x7f9f280a6ba0 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): my global_id is 174192 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f0c004eb0 con 0x7f9f280a6ba0 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873651677 0 0) 0x7f9f280a9dc0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): my global_id is 174140 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f281cf020 con 0x7f9f2813f5c0 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485661459 0 0) 0x7f9f28142c60 con 0x7f9f280a67c0 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): my global_id is 174160 2024-08-19T09:59:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f280a9dc0 con 0x7f9f280a67c0 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3600389507 0 0) 0x7f9f0c004eb0 con 0x7f9f280a6ba0 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f28142c60 con 0x7f9f280a6ba0 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f180047e0 con 0x7f9f280a6ba0 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): dump: 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1177300556 0 0) 0x7f9f280a9dc0 con 0x7f9f280a67c0 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f0c004eb0 con 0x7f9f280a67c0 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f24003610 con 0x7f9f280a67c0 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): dump: 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1504838258 0 0) 0x7f9f281cf020 con 0x7f9f2813f5c0 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f280a9dc0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f20003cd0 con 0x7f9f2813f5c0 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient(hunting): dump: 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1072580143 0 0) 0x7f9f28142c60 con 0x7f9f280a6ba0 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.1:6793/0 conn(0x7f9f2813f5c0 legacy=0x7f9f281d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f280a67c0 legacy=0x7f9f281ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 monclient: found mon.d 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f280a1830 con 0x7f9f280a6ba0 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 5 monclient: authenticate success, global_id 174192 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 10 monclient: _renew_subs 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2813aa80 con 0x7f9f280a6ba0 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 10 client.174192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:26.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 10 monclient: _renew_subs 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f2813a7a0 con 0x7f9f280a6ba0 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f3036b640 1 librados: init done 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f18004ce0 con 0x7f9f280a6ba0 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f18005250 con 0x7f9f280a6ba0 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f2c904640 4 set_mon_vals no callback set 2024-08-19T09:59:26.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.225+0000 7f9f1effd640 10 monclient: dump: 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.247 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.226+0000 7f9f3036b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.226+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f100065c0 con 0x7f9f10002860 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.226+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9f180053f0 con 0x7f9f280a6ba0 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.227+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(361..361 src has 299..361) ==== 5064+0+0 (unknown 3986656978 0 0) 0x7f9f1804e150 con 0x7f9f280a6ba0 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.227+0000 7f9f1effd640 10 client.174192.objecter ms_dispatch 0x7f9f28000b90 osd_map(361..361 src has 299..361) 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.227+0000 7f9f1effd640 3 client.174192.objecter handle_osd_map got epochs [361,361] > 0 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.227+0000 7f9f1effd640 3 client.174192.objecter handle_osd_map decoding full epoch 361 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.227+0000 7f9f1effd640 20 client.174192.objecter dump_active .. 0 homeless 2024-08-19T09:59:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.227+0000 7f9f2e908640 10 monclient: get_auth_request con 0x7f9f0c0418e0 auth_method 0 2024-08-19T09:59:26.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.228+0000 7f9f2e107640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f10002860 legacy=0x7f9f10004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.228+0000 7f9f1effd640 10 client.174192.objecter ms_handle_connect 0x7f9f10002860 2024-08-19T09:59:26.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.228+0000 7f9f1effd640 10 client.174192.objecter resend_mon_ops 2024-08-19T09:59:26.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2708677593 0 0) 0x7f9f100065c0 con 0x7f9f10002860 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient(hunting): my global_id is 174169 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f28142c60 con 0x7f9f10002860 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3039442233 0 0) 0x7f9f28142c60 con 0x7f9f10002860 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f100065c0 con 0x7f9f10002860 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f24003d60 con 0x7f9f10002860 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:26.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient: dump: 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 client.174192.objecter ms_handle_connect 0x7f9f0c0418e0 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1423132590 0 0) 0x7f9f100065c0 con 0x7f9f10002860 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f1000a160 con 0x7f9f10002860 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9f24003fe0 con 0x7f9f10002860 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f10002860 legacy=0x7f9f10004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.229+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f10002860 legacy=0x7f9f10004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.235+0000 7f9f3036b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.235+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f10009e50 con 0x7f9f10009110 2024-08-19T09:59:26.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f2e107640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f10009110 legacy=0x7f9f100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 client.174192.objecter ms_handle_connect 0x7f9f10009110 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 client.174192.objecter resend_mon_ops 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707080626 0 0) 0x7f9f10009e50 con 0x7f9f10009110 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient(hunting): my global_id is 174178 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f100065c0 con 0x7f9f10009110 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2425509203 0 0) 0x7f9f100065c0 con 0x7f9f10009110 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f10009e50 con 0x7f9f10009110 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f24005b00 con 0x7f9f10009110 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:26.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 10 monclient: dump: 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.256 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4100409483 0 0) 0x7f9f10009e50 con 0x7f9f10009110 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.238+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f1000c3f0 con 0x7f9f10009110 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 1225757641) 0x7f9f24007d70 con 0x7f9f10009110 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f1effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f1effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f10009110 legacy=0x7f9f100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f1effd640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6789/0 conn(0x7f9f10009110 legacy=0x7f9f100095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.257 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16296, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:26.258 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.259 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:26.261 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.262 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.263 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.264 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:26.267 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:26.270 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:26.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.272 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.276 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.277 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.277 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.277 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:26.277 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 10 librados: watch_flush enter 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 10 librados: watch_flush exit 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 20 client.174192.objecter shutdown clearing up homeless session... 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 10 client.174192.objecter successfully canceled tick 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9f0c0418e0 legacy=0x7f9f0c043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 10 monclient: shutdown 2024-08-19T09:59:26.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 >> v1:172.21.15.1:6792/0 conn(0x7f9f280a6ba0 legacy=0x7f9f281ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f2e908640 1 -- 172.21.15.1:0/3546860339 reap_dead start 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f2e908640 1 -- 172.21.15.1:0/3546860339 reap_dead start 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 shutdown_connections 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 >> 172.21.15.1:0/3546860339 conn(0x7f9f280ac4b0 msgr2=0x7f9f280ab2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 shutdown_connections 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.239+0000 7f9f3036b640 1 -- 172.21.15.1:0/3546860339 wait complete. 2024-08-19T09:59:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.240+0000 7f9f3036b640 1 librados: shutdown 2024-08-19T09:59:26.280 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:26.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:59:26.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:59:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.379+0000 7f89a76fd640 10 monclient: get_monmap_and_config 2024-08-19T09:59:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.379+0000 7f89a76fd640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.379+0000 7f89a76fd640 10 monclient: monmap: 2024-08-19T09:59:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:26.380419+0000 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:26.380419+0000 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.382 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.382 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.382 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 1 Processor -- start 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 1 -- start start 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient: init 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient: _add_conns ranks=[1,8,0,6,3,7,4,2,5] 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient(hunting): picked mon.noname-f con 0x7f89a00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient(hunting): picked mon.noname-e con 0x7f89a00a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient(hunting): picked mon.noname-a con 0x7f89a013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a00a3b40 con 0x7f89a013f5c0 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a00a8130 con 0x7f89a00a67c0 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a006ede0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a76fd640 10 monclient(hunting): authenticate will time out at 1222.456767s 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a4c98640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f89a00a6ba0 0x7f89a00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35788/0 (socket says 172.21.15.1:35788) 2024-08-19T09:59:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.380+0000 7f89a4c98640 1 -- 172.21.15.1:0/3641451744 learned_addr learned my addr 172.21.15.1:0/3641451744 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a4c98640 1 -- 172.21.15.1:0/3641451744 >> v1:172.21.15.1:6793/0 conn(0x7f89a00a6ba0 legacy=0x7f89a00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a5c9a640 1 -- 172.21.15.1:0/3641451744 >> v1:172.21.15.1:6789/0 conn(0x7f89a013f5c0 legacy=0x7f89a01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259587961 0 0) 0x7f89a006ede0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a5499640 1 -- 172.21.15.1:0/3641451744 >> v1:172.21.15.82:6789/0 conn(0x7f89a00a67c0 legacy=0x7f89a00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): my global_id is 174149 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f898c003520 con 0x7f89a00a6ba0 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135208589 0 0) 0x7f89a00a3b40 con 0x7f89a013f5c0 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): my global_id is 174150 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a006ede0 con 0x7f89a013f5c0 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130885296 0 0) 0x7f89a00a8130 con 0x7f89a00a67c0 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient(hunting): my global_id is 174187 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a00a3b40 con 0x7f89a00a67c0 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1667304879 0 0) 0x7f898c003520 con 0x7f89a00a6ba0 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 >> v1:172.21.15.82:6789/0 conn(0x7f89a00a67c0 legacy=0x7f89a00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 >> v1:172.21.15.1:6789/0 conn(0x7f89a013f5c0 legacy=0x7f89a01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 monclient: found mon.noname-e 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a00a92a0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8990001ad0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: dump: 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.396 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a76fd640 5 monclient: authenticate success, global_id 174149 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a76fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89900048a0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 1 -- 172.21.15.1:0/3641451744 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8990004dd0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f8997fff640 10 monclient: dump: 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.398 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.399 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a76fd640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.383+0000 7f89a76fd640 4 set_mon_vals no callback set 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 10 monclient: shutdown 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 1 -- 172.21.15.1:0/3641451744 >> v1:172.21.15.1:6793/0 conn(0x7f89a00a6ba0 legacy=0x7f89a00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 1 -- 172.21.15.1:0/3641451744 shutdown_connections 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 1 -- 172.21.15.1:0/3641451744 >> 172.21.15.1:0/3641451744 conn(0x7f89a00ac4b0 msgr2=0x7f89a00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 1 -- 172.21.15.1:0/3641451744 shutdown_connections 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 1 -- 172.21.15.1:0/3641451744 wait complete. 2024-08-19T09:59:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.384+0000 7f89a76fd640 10 monclient: monmap: 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 librados: starting msgr at 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 librados: starting objecter 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 Processor -- start 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 -- start start 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 librados: setting wanted keys 2024-08-19T09:59:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 librados: calling monclient init 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient: init 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient: _add_conns ranks=[3,4,7,6,8,2,5,0,1] 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient(hunting): picked mon.noname-g con 0x7f89a00a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient(hunting): picked mon.noname-c con 0x7f89a00a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient(hunting): picked mon.noname-i con 0x7f89a013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a01cf020 con 0x7f89a00a67c0 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a013e400 con 0x7f89a00a6ba0 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a0139bb0 con 0x7f89a013f5c0 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a76fd640 10 monclient(hunting): authenticate will time out at 1222.461635s 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a4c98640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f89a00a6ba0 0x7f89a01ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54272/0 (socket says 172.21.15.1:54272) 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.385+0000 7f89a4c98640 1 -- 172.21.15.1:0/4184813889 learned_addr learned my addr 172.21.15.1:0/4184813889 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f89a4c98640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.1:6791/0 conn(0x7f89a00a6ba0 legacy=0x7f89a01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f8995ffb640 10 client.?.objecter ms_handle_connect 0x7f89a00a6ba0 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f8995ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f8995ffb640 10 client.?.objecter ms_handle_connect 0x7f89a013f5c0 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f8995ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f89a5c9a640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6792/0 conn(0x7f89a013f5c0 legacy=0x7f89a01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f8995ffb640 10 client.?.objecter ms_handle_connect 0x7f89a00a67c0 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f8995ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.387+0000 7f89a5499640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a00a67c0 legacy=0x7f89a01ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431614155 0 0) 0x7f89a013e400 con 0x7f89a00a6ba0 2024-08-19T09:59:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): my global_id is 27481 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8984004eb0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1637500107 0 0) 0x7f89a0139bb0 con 0x7f89a013f5c0 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): my global_id is 144646 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a013e400 con 0x7f89a013f5c0 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1989017448 0 0) 0x7f89a01cf020 con 0x7f89a00a67c0 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): my global_id is 135156 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a0139bb0 con 0x7f89a00a67c0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3207112090 0 0) 0x7f8984004eb0 con 0x7f89a00a6ba0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89a01cf020 con 0x7f89a00a6ba0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 47678414 0 0) 0x7f89a013e400 con 0x7f89a013f5c0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8984004eb0 con 0x7f89a013f5c0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8998003990 con 0x7f89a013f5c0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4042744688 0 0) 0x7f89a0139bb0 con 0x7f89a00a67c0 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89a013e400 con 0x7f89a00a67c0 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f899c003610 con 0x7f89a00a67c0 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.409 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8990004d90 con 0x7f89a00a6ba0 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3169067737 0 0) 0x7f8984004eb0 con 0x7f89a013f5c0 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.1:6791/0 conn(0x7f89a00a6ba0 legacy=0x7f89a01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a00a67c0 legacy=0x7f89a01ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 monclient: found mon.i 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a013bf10 con 0x7f89a013f5c0 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 5 monclient: authenticate success, global_id 144646 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 10 monclient: _renew_subs 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89a013b670 con 0x7f89a013f5c0 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8998003de0 con 0x7f89a013f5c0 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8998004310 con 0x7f89a013f5c0 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f8995ffb640 10 monclient: dump: 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.413 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89977fe640 4 set_mon_vals no callback set 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 10 client.144646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 10 monclient: _renew_subs 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f89a00a2890 con 0x7f89a013f5c0 2024-08-19T09:59:26.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.388+0000 7f89a76fd640 1 librados: init done 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f89a76fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f89a0143070 con 0x7f89a009da50 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f89980044b0 con 0x7f89a013f5c0 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(361..361 src has 299..361) ==== 5064+0+0 (unknown 3986656978 0 0) 0x7f899804e210 con 0x7f89a013f5c0 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f8995ffb640 10 client.144646.objecter ms_dispatch 0x7f89a0000b90 osd_map(361..361 src has 299..361) 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f8995ffb640 3 client.144646.objecter handle_osd_map got epochs [361,361] > 0 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.389+0000 7f8995ffb640 3 client.144646.objecter handle_osd_map decoding full epoch 361 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.390+0000 7f8995ffb640 20 client.144646.objecter dump_active .. 0 homeless 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.390+0000 7f89a4c98640 10 monclient: get_auth_request con 0x7f8984041900 auth_method 0 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.391+0000 7f89a5499640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a009da50 legacy=0x7f89a009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.391+0000 7f8995ffb640 10 client.144646.objecter ms_handle_connect 0x7f89a009da50 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.391+0000 7f8995ffb640 10 client.144646.objecter resend_mon_ops 2024-08-19T09:59:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101094362 0 0) 0x7f89a013e400 con 0x7f89a009da50 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient(hunting): my global_id is 135165 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8984004eb0 con 0x7f89a009da50 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2868860752 0 0) 0x7f8984004eb0 con 0x7f89a009da50 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89a013e400 con 0x7f89a009da50 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f899c003e50 con 0x7f89a009da50 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient: dump: 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.417 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 client.144646.objecter ms_handle_connect 0x7f8984041900 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3761697593 0 0) 0x7f89a013e400 con 0x7f89a009da50 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89a01427b0 con 0x7f89a009da50 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f899c004080 con 0x7f89a009da50 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a009da50 legacy=0x7f89a009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.392+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a009da50 legacy=0x7f89a009de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.398+0000 7f89a76fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.399+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f89a0068f60 con 0x7f89a009e540 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f89a5499640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a009e540 legacy=0x7f89a01dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 client.144646.objecter ms_handle_connect 0x7f89a009e540 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 client.144646.objecter resend_mon_ops 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975734559 0 0) 0x7f89a0068f60 con 0x7f89a009e540 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient(hunting): my global_id is 135174 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a013e400 con 0x7f89a009e540 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2552099218 0 0) 0x7f89a013e400 con 0x7f89a009e540 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89a0068f60 con 0x7f89a009e540 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f899c005b00 con 0x7f89a009e540 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 10 monclient: dump: 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3984108016 0 0) 0x7f89a0068f60 con 0x7f89a009e540 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.401+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f89a009e920 con 0x7f89a009e540 2024-08-19T09:59:26.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7851 (unknown 0 0 4179538415) 0x7f899c007ec0 con 0x7f89a009e540 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f8995ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f8995ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a009e540 legacy=0x7f89a01dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f8995ffb640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6790/0 conn(0x7f89a009e540 legacy=0x7f89a01dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:26.423 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 423462, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:26.424 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.425 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.426 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:26.427 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.428 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.429 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:26.430 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.431 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.432 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:26.433 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:26.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.435 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.436 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.439 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:26.440 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.441 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:26.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:26.443 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.444 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.446 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.446 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:26.446 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 10 librados: watch_flush enter 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 10 librados: watch_flush exit 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 20 client.144646.objecter shutdown clearing up homeless session... 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 10 client.144646.objecter successfully canceled tick 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8984041900 legacy=0x7f8984043dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 10 monclient: shutdown 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 >> v1:172.21.15.82:6792/0 conn(0x7f89a013f5c0 legacy=0x7f89a01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a5c9a640 1 -- 172.21.15.1:0/4184813889 reap_dead start 2024-08-19T09:59:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a5c9a640 1 -- 172.21.15.1:0/4184813889 reap_dead start 2024-08-19T09:59:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 shutdown_connections 2024-08-19T09:59:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 >> 172.21.15.1:0/4184813889 conn(0x7f89a00ac4b0 msgr2=0x7f89a01419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 shutdown_connections 2024-08-19T09:59:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 -- 172.21.15.1:0/4184813889 wait complete. 2024-08-19T09:59:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.402+0000 7f89a76fd640 1 librados: shutdown 2024-08-19T09:59:26.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:26.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:59:26.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:59:26.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.548+0000 7fae27572640 10 monclient: get_monmap_and_config 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.548+0000 7fae27572640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.548+0000 7fae27572640 10 monclient: monmap: 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:26.549405+0000 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:26.549405+0000 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 1 Processor -- start 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 1 -- start start 2024-08-19T09:59:26.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient: init 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient: _add_conns ranks=[4,2,3,5,6,8,0,1,7] 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient(hunting): picked mon.noname-c con 0x7fae200a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient(hunting): picked mon.noname-b con 0x7fae200a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient(hunting): picked mon.noname-g con 0x7fae2013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae200a3b40 con 0x7fae200a6ba0 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae200a8130 con 0x7fae2013f5c0 2024-08-19T09:59:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2006ede0 con 0x7fae200a67c0 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae27572640 10 monclient(hunting): authenticate will time out at 1222.625775s 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae24b0d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fae200a6ba0 0x7fae200a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39836/0 (socket says 172.21.15.1:39836) 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae2530e640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fae200a67c0 0x7fae200a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:54282/0 (socket says 172.21.15.1:54282) 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.549+0000 7fae24b0d640 1 -- 172.21.15.1:0/2304671834 learned_addr learned my addr 172.21.15.1:0/2304671834 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae2530e640 1 -- 172.21.15.1:0/2304671834 >> v1:172.21.15.1:6791/0 conn(0x7fae200a67c0 legacy=0x7fae200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae24b0d640 1 -- 172.21.15.1:0/2304671834 >> v1:172.21.15.1:6790/0 conn(0x7fae200a6ba0 legacy=0x7fae200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355127485 0 0) 0x7fae200a3b40 con 0x7fae200a6ba0 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): my global_id is 164396 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae0c003520 con 0x7fae200a6ba0 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366220942 0 0) 0x7fae2006ede0 con 0x7fae200a67c0 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient(hunting): my global_id is 27490 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae200a3b40 con 0x7fae200a67c0 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae25b0f640 1 -- 172.21.15.1:0/2304671834 >> v1:172.21.15.82:6790/0 conn(0x7fae2013f5c0 legacy=0x7fae201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3248512544 0 0) 0x7fae0c003520 con 0x7fae200a6ba0 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 >> v1:172.21.15.1:6791/0 conn(0x7fae200a67c0 legacy=0x7fae200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 >> v1:172.21.15.82:6790/0 conn(0x7fae2013f5c0 legacy=0x7fae201419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 monclient: found mon.noname-b 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae200a92a0 con 0x7fae200a6ba0 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae27572640 5 monclient: authenticate success, global_id 164396 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae10001a10 con 0x7fae200a6ba0 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae27572640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:59:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: dump: 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae27572640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae10001ce0 con 0x7fae200a6ba0 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 1 -- 172.21.15.1:0/2304671834 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae10004be0 con 0x7fae200a6ba0 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae17fff640 10 monclient: dump: 2024-08-19T09:59:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.564 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae27572640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.552+0000 7fae27572640 4 set_mon_vals no callback set 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 10 monclient: shutdown 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 1 -- 172.21.15.1:0/2304671834 >> v1:172.21.15.1:6790/0 conn(0x7fae200a6ba0 legacy=0x7fae200a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 1 -- 172.21.15.1:0/2304671834 shutdown_connections 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 1 -- 172.21.15.1:0/2304671834 >> 172.21.15.1:0/2304671834 conn(0x7fae200ac4b0 msgr2=0x7fae200a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 1 -- 172.21.15.1:0/2304671834 shutdown_connections 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 1 -- 172.21.15.1:0/2304671834 wait complete. 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.553+0000 7fae27572640 10 monclient: monmap: 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.566 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 librados: starting msgr at 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 librados: starting objecter 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 Processor -- start 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 -- start start 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 librados: setting wanted keys 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 librados: calling monclient init 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient: init 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient: _add_conns ranks=[0,5,8,2,4,1,3,6,7] 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient(hunting): picked mon.noname-a con 0x7fae200a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient(hunting): picked mon.noname-h con 0x7fae200a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient(hunting): picked mon.noname-e con 0x7fae2013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae20139990 con 0x7fae200a67c0 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2013a6f0 con 0x7fae200a6ba0 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2013b210 con 0x7fae2013f5c0 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae27572640 10 monclient(hunting): authenticate will time out at 1222.630626s 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae2530e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fae200a67c0 0x7fae201ca870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57842/0 (socket says 172.21.15.1:57842) 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae2530e640 1 -- 172.21.15.1:0/1274967444 learned_addr learned my addr 172.21.15.1:0/1274967444 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.554+0000 7fae25b0f640 1 --1- 172.21.15.1:0/1274967444 >> v1:172.21.15.1:6793/0 conn(0x7fae2013f5c0 0x7fae201d63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35802/0 (socket says 172.21.15.1:35802) 2024-08-19T09:59:26.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.556+0000 7fae2530e640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.1:6789/0 conn(0x7fae200a67c0 legacy=0x7fae201ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.556+0000 7fae25b0f640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.1:6793/0 conn(0x7fae2013f5c0 legacy=0x7fae201d63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.556+0000 7fae15ffb640 10 client.?.objecter ms_handle_connect 0x7fae200a67c0 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.556+0000 7fae15ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.556+0000 7fae15ffb640 10 client.?.objecter ms_handle_connect 0x7fae2013f5c0 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.556+0000 7fae15ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 client.?.objecter ms_handle_connect 0x7fae200a6ba0 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae24b0d640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae200a6ba0 legacy=0x7fae201d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4109870280 0 0) 0x7fae2013b210 con 0x7fae2013f5c0 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): my global_id is 174158 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae04004eb0 con 0x7fae2013f5c0 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620838163 0 0) 0x7fae20139990 con 0x7fae200a67c0 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): my global_id is 174159 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae2013b210 con 0x7fae200a67c0 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120227732 0 0) 0x7fae2013a6f0 con 0x7fae200a6ba0 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): my global_id is 27149 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae20139990 con 0x7fae200a6ba0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2723494058 0 0) 0x7fae2013b210 con 0x7fae200a67c0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae2013a6f0 con 0x7fae200a67c0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 779072933 0 0) 0x7fae04004eb0 con 0x7fae2013f5c0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae2013b210 con 0x7fae2013f5c0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae1c0047e0 con 0x7fae200a67c0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.576 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.577 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae18002010 con 0x7fae2013f5c0 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.578 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.579 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1290378725 0 0) 0x7fae20139990 con 0x7fae200a6ba0 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae04004eb0 con 0x7fae200a6ba0 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae10004b20 con 0x7fae200a6ba0 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.580 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3161027959 0 0) 0x7fae2013b210 con 0x7fae2013f5c0 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae200a6ba0 legacy=0x7fae201d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.1:6789/0 conn(0x7fae200a67c0 legacy=0x7fae201ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 monclient: found mon.e 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2013b4f0 con 0x7fae2013f5c0 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 5 monclient: authenticate success, global_id 174158 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 10 monclient: _renew_subs 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fae2013a2b0 con 0x7fae2013f5c0 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae18003d60 con 0x7fae2013f5c0 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae18004290 con 0x7fae2013f5c0 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae15ffb640 10 monclient: dump: 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.583 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae177fe640 4 set_mon_vals no callback set 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 10 client.174158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 10 monclient: _renew_subs 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fae201cb160 con 0x7fae2013f5c0 2024-08-19T09:59:26.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.557+0000 7fae27572640 1 librados: init done 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae27572640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fae18004430 con 0x7fae2013f5c0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fae200a26a0 con 0x7fae200a22c0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(361..361 src has 299..361) ==== 5064+0+0 (unknown 3986656978 0 0) 0x7fae1804e1e0 con 0x7fae2013f5c0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae15ffb640 10 client.174158.objecter ms_dispatch 0x7fae20000b90 osd_map(361..361 src has 299..361) 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae15ffb640 3 client.174158.objecter handle_osd_map got epochs [361,361] > 0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.558+0000 7fae15ffb640 3 client.174158.objecter handle_osd_map decoding full epoch 361 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.559+0000 7fae15ffb640 20 client.174158.objecter dump_active .. 0 homeless 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.559+0000 7fae24b0d640 10 monclient: get_auth_request con 0x7fae040417f0 auth_method 0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.560+0000 7fae2530e640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae200a22c0 legacy=0x7fae2009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.560+0000 7fae15ffb640 10 client.174158.objecter ms_handle_connect 0x7fae200a22c0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 client.174158.objecter resend_mon_ops 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907709945 0 0) 0x7fae200a26a0 con 0x7fae200a22c0 2024-08-19T09:59:26.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient(hunting): my global_id is 27158 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae2013b210 con 0x7fae200a22c0 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2474286820 0 0) 0x7fae2013b210 con 0x7fae200a22c0 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae200a26a0 con 0x7fae200a22c0 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae1c005360 con 0x7fae200a22c0 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient: dump: 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.586 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 client.174158.objecter ms_handle_connect 0x7fae040417f0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3704584584 0 0) 0x7fae200a26a0 con 0x7fae200a22c0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae2013ae80 con 0x7fae200a22c0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fae1c0055d0 con 0x7fae200a22c0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae200a22c0 legacy=0x7fae2009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.561+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae200a22c0 legacy=0x7fae2009da50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.568+0000 7fae27572640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.568+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fae20068f60 con 0x7fae2002e1d0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae2530e640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae2002e1d0 legacy=0x7fae2009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 10 client.174158.objecter ms_handle_connect 0x7fae2002e1d0 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 10 client.174158.objecter resend_mon_ops 2024-08-19T09:59:26.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110737409 0 0) 0x7fae20068f60 con 0x7fae2002e1d0 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 10 monclient(hunting): my global_id is 27167 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.570+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae200a26a0 con 0x7fae2002e1d0 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2494613462 0 0) 0x7fae200a26a0 con 0x7fae2002e1d0 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae20068f60 con 0x7fae2002e1d0 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae1c005e40 con 0x7fae2002e1d0 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 10 monclient: dump: 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.589 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 203214741 0 0) 0x7fae20068f60 con 0x7fae2002e1d0 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae2002e5b0 con 0x7fae2002e1d0 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 2962530974) 0x7fae1c0073e0 con 0x7fae2002e1d0 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae2002e1d0 legacy=0x7fae2009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae15ffb640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6791/0 conn(0x7fae2002e1d0 legacy=0x7fae2009e160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:26.591 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 572837, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:26.592 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.593 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.594 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.596 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:26.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:26.600 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:26.605 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.606 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.607 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:26.609 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.610 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:26.611 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae27572640 10 librados: watch_flush enter 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae27572640 10 librados: watch_flush exit 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae27572640 20 client.174158.objecter shutdown clearing up homeless session... 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.571+0000 7fae27572640 10 client.174158.objecter successfully canceled tick 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.82:6812/2549104415 conn(0x7fae040417f0 legacy=0x7fae04043cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 10 monclient: shutdown 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 >> v1:172.21.15.1:6793/0 conn(0x7fae2013f5c0 legacy=0x7fae201d63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae25b0f640 1 -- 172.21.15.1:0/1274967444 reap_dead start 2024-08-19T09:59:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae25b0f640 1 -- 172.21.15.1:0/1274967444 reap_dead start 2024-08-19T09:59:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 shutdown_connections 2024-08-19T09:59:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 >> 172.21.15.1:0/1274967444 conn(0x7fae200ac4b0 msgr2=0x7fae20141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 shutdown_connections 2024-08-19T09:59:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 -- 172.21.15.1:0/1274967444 wait complete. 2024-08-19T09:59:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.572+0000 7fae27572640 1 librados: shutdown 2024-08-19T09:59:26.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:26.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:59:26.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.715+0000 7fcd45627640 10 monclient: get_monmap_and_config 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.715+0000 7fcd45627640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.715+0000 7fcd45627640 10 monclient: monmap: 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:26.716094+0000 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:26.716094+0000 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.717 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.715+0000 7fcd45627640 1 Processor -- start 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.715+0000 7fcd45627640 1 -- start start 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.715+0000 7fcd45627640 10 monclient: init 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient: _add_conns ranks=[6,3,0,7,2,5,4,1,8] 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient(hunting): picked mon.noname-d con 0x7fcd400a6830 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient(hunting): picked mon.noname-g con 0x7fcd400694b0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient(hunting): picked mon.noname-a con 0x7fcd400a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd400a3c70 con 0x7fcd400a46a0 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd400a33d0 con 0x7fcd400694b0 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd400a36b0 con 0x7fcd400a6830 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd45627640 10 monclient(hunting): authenticate will time out at 1222.792470s 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd3effd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fcd400a6830 0x7fcd400a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34940/0 (socket says 172.21.15.1:34940) 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd3effd640 1 -- 172.21.15.1:0/2867508744 learned_addr learned my addr 172.21.15.1:0/2867508744 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd3f7fe640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.1:6789/0 conn(0x7fcd400a46a0 legacy=0x7fcd40141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd3f7fe640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.1:6789/0 conn(0x7fcd400a46a0 legacy=0x7fcd40141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.716+0000 7fcd3f7fe640 1 --1- 172.21.15.1:0/2867508744 >> v1:172.21.15.1:6789/0 conn(0x7fcd400a46a0 0x7fcd40141850 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-08-19T09:59:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3effd640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.1:6792/0 conn(0x7fcd400a6830 legacy=0x7fcd400a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3e7fc640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.82:6790/0 conn(0x7fcd400694b0 legacy=0x7fcd400a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791501426 0 0) 0x7fcd400a36b0 con 0x7fcd400a6830 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3dffb640 10 monclient(hunting): my global_id is 174201 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.718+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd2c003520 con 0x7fcd400a6830 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967138176 0 0) 0x7fcd400a33d0 con 0x7fcd400694b0 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient(hunting): my global_id is 135192 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd400a36b0 con 0x7fcd400694b0 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3511541348 0 0) 0x7fcd2c003520 con 0x7fcd400a6830 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.82:6790/0 conn(0x7fcd400694b0 legacy=0x7fcd400a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.1:6789/0 conn(0x7fcd400a46a0 legacy=0x7fcd40141850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 monclient: found mon.noname-d 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4006dbb0 con 0x7fcd400a6830 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 5 monclient: authenticate success, global_id 174201 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd38001e60 con 0x7fcd400a6830 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: dump: 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.730 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd38004c20 con 0x7fcd400a6830 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 4 set_mon_vals no callback set 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 1 -- 172.21.15.1:0/2867508744 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd38005190 con 0x7fcd400a6830 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd3dffb640 10 monclient: dump: 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.732 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 10 monclient: shutdown 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 1 -- 172.21.15.1:0/2867508744 >> v1:172.21.15.1:6792/0 conn(0x7fcd400a6830 legacy=0x7fcd400a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 1 -- 172.21.15.1:0/2867508744 shutdown_connections 2024-08-19T09:59:26.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 1 -- 172.21.15.1:0/2867508744 >> 172.21.15.1:0/2867508744 conn(0x7fcd400a9020 msgr2=0x7fcd40068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 1 -- 172.21.15.1:0/2867508744 shutdown_connections 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.719+0000 7fcd45627640 1 -- 172.21.15.1:0/2867508744 wait complete. 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 10 monclient: monmap: 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.734 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 1 librados: starting msgr at 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 1 librados: starting objecter 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 1 Processor -- start 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 1 -- start start 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 1 librados: setting wanted keys 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 1 librados: calling monclient init 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.720+0000 7fcd45627640 10 monclient: init 2024-08-19T09:59:26.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient: _add_conns ranks=[6,8,4,2,3,0,7,1,5] 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient(hunting): picked mon.noname-d con 0x7fcd400694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient(hunting): picked mon.noname-e con 0x7fcd400a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient(hunting): picked mon.noname-c con 0x7fcd400a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd401cf1d0 con 0x7fcd400a6830 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd400aa2d0 con 0x7fcd400694b0 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd40136c00 con 0x7fcd400a46a0 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd45627640 10 monclient(hunting): authenticate will time out at 1222.797305s 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd3effd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fcd400694b0 0x7fcd401ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34942/0 (socket says 172.21.15.1:34942) 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.721+0000 7fcd3effd640 1 -- 172.21.15.1:0/4023446164 learned_addr learned my addr 172.21.15.1:0/4023446164 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd3effd640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.1:6792/0 conn(0x7fcd400694b0 legacy=0x7fcd401ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 client.?.objecter ms_handle_connect 0x7fcd400694b0 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd3f7fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.1:6791/0 conn(0x7fcd400a6830 legacy=0x7fcd401d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 client.?.objecter ms_handle_connect 0x7fcd400a6830 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 client.?.objecter ms_handle_connect 0x7fcd400a46a0 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd3e7fc640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.1:6793/0 conn(0x7fcd400a46a0 legacy=0x7fcd401ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801678570 0 0) 0x7fcd400aa2d0 con 0x7fcd400694b0 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): my global_id is 174210 2024-08-19T09:59:26.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd20004eb0 con 0x7fcd400694b0 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488607835 0 0) 0x7fcd40136c00 con 0x7fcd400a46a0 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): my global_id is 174167 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd400aa2d0 con 0x7fcd400a46a0 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2489256301 0 0) 0x7fcd401cf1d0 con 0x7fcd400a6830 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 10 monclient(hunting): my global_id is 27499 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd40136c00 con 0x7fcd400a6830 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1658662507 0 0) 0x7fcd400aa2d0 con 0x7fcd400a46a0 2024-08-19T09:59:26.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.723+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd401cf1d0 con 0x7fcd400a46a0 2024-08-19T09:59:26.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd28003610 con 0x7fcd400a46a0 2024-08-19T09:59:26.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:59:26.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): dump: 2024-08-19T09:59:26.742 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.742 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.742 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.743 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3408730530 0 0) 0x7fcd20004eb0 con 0x7fcd400694b0 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd400aa2d0 con 0x7fcd400694b0 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3406586146 0 0) 0x7fcd40136c00 con 0x7fcd400a6830 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd20004eb0 con 0x7fcd400a6830 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd380020b0 con 0x7fcd400694b0 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): dump: 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.745 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd30004250 con 0x7fcd400a6830 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient(hunting): dump: 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2949316778 0 0) 0x7fcd401cf1d0 con 0x7fcd400a46a0 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.1:6792/0 conn(0x7fcd400694b0 legacy=0x7fcd401ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.1:6791/0 conn(0x7fcd400a6830 legacy=0x7fcd401d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 monclient: found mon.e 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd401386c0 con 0x7fcd400a46a0 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 5 monclient: authenticate success, global_id 174167 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 10 monclient: _renew_subs 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd401427d0 con 0x7fcd400a46a0 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 10 client.174167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 10 monclient: _renew_subs 2024-08-19T09:59:26.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd40137940 con 0x7fcd400a46a0 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd45627640 1 librados: init done 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd28003ae0 con 0x7fcd400a46a0 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd28004010 con 0x7fcd400a46a0 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd3d7fa640 4 set_mon_vals no callback set 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.724+0000 7fcd377fe640 10 monclient: dump: 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.749 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd45627640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcd240065c0 con 0x7fcd24002860 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcd280042e0 con 0x7fcd400a46a0 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(361..361 src has 299..361) ==== 5064+0+0 (unknown 3986656978 0 0) 0x7fcd2804e090 con 0x7fcd400a46a0 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd377fe640 10 client.174167.objecter ms_dispatch 0x7fcd40000b90 osd_map(361..361 src has 299..361) 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd377fe640 3 client.174167.objecter handle_osd_map got epochs [361,361] > 0 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd377fe640 3 client.174167.objecter handle_osd_map decoding full epoch 361 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd377fe640 20 client.174167.objecter dump_active .. 0 homeless 2024-08-19T09:59:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.725+0000 7fcd3f7fe640 10 monclient: get_auth_request con 0x7fcd20041bb0 auth_method 0 2024-08-19T09:59:26.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd3effd640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6792/0 conn(0x7fcd24002860 legacy=0x7fcd24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 client.174167.objecter ms_handle_connect 0x7fcd24002860 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 client.174167.objecter resend_mon_ops 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686107587 0 0) 0x7fcd240065c0 con 0x7fcd24002860 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient(hunting): my global_id is 144655 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd401cf1d0 con 0x7fcd24002860 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1336978596 0 0) 0x7fcd401cf1d0 con 0x7fcd24002860 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd240065c0 con 0x7fcd24002860 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd38004c20 con 0x7fcd24002860 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.727+0000 7fcd377fe640 10 monclient: dump: 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 10 client.174167.objecter ms_handle_connect 0x7fcd20041bb0 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2462651893 0 0) 0x7fcd240065c0 con 0x7fcd24002860 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd2400a160 con 0x7fcd24002860 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcd38007510 con 0x7fcd24002860 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6792/0 conn(0x7fcd24002860 legacy=0x7fcd24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.728+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6792/0 conn(0x7fcd24002860 legacy=0x7fcd24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.734+0000 7fcd45627640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.734+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcd24009e50 con 0x7fcd24009110 2024-08-19T09:59:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd3effd640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6792/0 conn(0x7fcd24009110 legacy=0x7fcd240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 client.174167.objecter ms_handle_connect 0x7fcd24009110 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 client.174167.objecter resend_mon_ops 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3291713202 0 0) 0x7fcd24009e50 con 0x7fcd24009110 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient(hunting): my global_id is 144664 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd240065c0 con 0x7fcd24009110 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2109385563 0 0) 0x7fcd240065c0 con 0x7fcd24009110 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd24009e50 con 0x7fcd24009110 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcd38005e20 con 0x7fcd24009110 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 10 monclient: dump: 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4179923566 0 0) 0x7fcd24009e50 con 0x7fcd24009110 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.737+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd2400c3a0 con 0x7fcd24009110 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3934138453) 0x7fcd380077e0 con 0x7fcd24009110 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd377fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6792/0 conn(0x7fcd24009110 legacy=0x7fcd240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd377fe640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6792/0 conn(0x7fcd24009110 legacy=0x7fcd240095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 291116, 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:26.761 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.762 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.763 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:26.764 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:26.765 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:26.766 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:26.767 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:26.768 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.771 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.772 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.774 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.776 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:26.780 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:26.781 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:26.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:26.781 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:26.781 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:26.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 10 librados: watch_flush enter 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 10 librados: watch_flush exit 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 20 client.174167.objecter shutdown clearing up homeless session... 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 10 client.174167.objecter successfully canceled tick 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcd20041bb0 legacy=0x7fcd20044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 10 monclient: shutdown 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 >> v1:172.21.15.1:6793/0 conn(0x7fcd400a46a0 legacy=0x7fcd401ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd3f7fe640 1 -- 172.21.15.1:0/4023446164 reap_dead start 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd3f7fe640 1 -- 172.21.15.1:0/4023446164 reap_dead start 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 shutdown_connections 2024-08-19T09:59:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 >> 172.21.15.1:0/4023446164 conn(0x7fcd400a9020 msgr2=0x7fcd400a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 shutdown_connections 2024-08-19T09:59:26.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 -- 172.21.15.1:0/4023446164 wait complete. 2024-08-19T09:59:26.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.738+0000 7fcd45627640 1 librados: shutdown 2024-08-19T09:59:26.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:26.784 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T09:59:26.784 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T09:59:26.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.884+0000 7f24e344f640 10 monclient: get_monmap_and_config 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.884+0000 7f24e344f640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.884+0000 7f24e344f640 10 monclient: monmap: 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:26.885764+0000 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:26.885764+0000 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.887 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.885+0000 7f24e344f640 1 Processor -- start 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.885+0000 7f24e344f640 1 -- start start 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.885+0000 7f24e344f640 10 monclient: init 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient: _add_conns ranks=[2,4,0,3,1,8,7,6,5] 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient(hunting): picked mon.noname-b con 0x7f24dc0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient(hunting): picked mon.noname-c con 0x7f24dc0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient(hunting): picked mon.noname-a con 0x7f24dc13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24dc0a3b40 con 0x7f24dc13f5c0 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24dc0a8130 con 0x7f24dc0a67c0 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24dc06ede0 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e344f640 10 monclient(hunting): authenticate will time out at 1222.962205s 2024-08-19T09:59:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e11eb640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f24dc0a67c0 0x7f24dc0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33956/0 (socket says 172.21.15.1:33956) 2024-08-19T09:59:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e11eb640 1 -- 172.21.15.1:0/4119722054 learned_addr learned my addr 172.21.15.1:0/4119722054 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.886+0000 7f24e19ec640 1 --1- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6789/0 conn(0x7f24dc13f5c0 0x7f24dc1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37134/0 (socket says 172.21.15.1:37134) 2024-08-19T09:59:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24e11eb640 1 -- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6790/0 conn(0x7f24dc0a67c0 legacy=0x7f24dc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24e09ea640 1 -- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6791/0 conn(0x7f24dc0a6ba0 legacy=0x7f24dc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24e19ec640 1 -- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6789/0 conn(0x7f24dc13f5c0 legacy=0x7f24dc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732351399 0 0) 0x7f24dc0a8130 con 0x7f24dc0a67c0 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): my global_id is 164405 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24c8003520 con 0x7f24dc0a67c0 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4192226406 0 0) 0x7f24dc06ede0 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): my global_id is 27508 2024-08-19T09:59:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24dc0a8130 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660118854 0 0) 0x7f24dc0a3b40 con 0x7f24dc13f5c0 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient(hunting): my global_id is 174168 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24dc06ede0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2853215530 0 0) 0x7f24c8003520 con 0x7f24dc0a67c0 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24e09ea640 1 --1- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6791/0 conn(0x7f24dc0a6ba0 0x7f24dc0a6fb0 :-1 s=OPENED pgs=3509 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f24cc001950 mon_map magic: 0 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6791/0 conn(0x7f24dc0a6ba0 legacy=0x7f24dc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6789/0 conn(0x7f24dc13f5c0 legacy=0x7f24dc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 monclient: found mon.noname-b 2024-08-19T09:59:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24dc0a92a0 con 0x7f24dc0a67c0 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24d8001ae0 con 0x7f24dc0a67c0 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.888+0000 7f24d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: dump: 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24d4001e00 con 0x7f24dc13f5c0 2024-08-19T09:59:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 5 monclient: authenticate success, global_id 164405 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24d80048a0 con 0x7f24dc0a67c0 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 1 -- 172.21.15.1:0/4119722054 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24d8004e10 con 0x7f24dc0a67c0 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 4 set_mon_vals no callback set 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24d3fff640 10 monclient: dump: 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.903 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 10 monclient: shutdown 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 1 -- 172.21.15.1:0/4119722054 >> v1:172.21.15.1:6790/0 conn(0x7f24dc0a67c0 legacy=0x7f24dc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 1 -- 172.21.15.1:0/4119722054 shutdown_connections 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 1 -- 172.21.15.1:0/4119722054 >> 172.21.15.1:0/4119722054 conn(0x7f24dc0ac4b0 msgr2=0x7f24dc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 1 -- 172.21.15.1:0/4119722054 shutdown_connections 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.889+0000 7f24e344f640 1 -- 172.21.15.1:0/4119722054 wait complete. 2024-08-19T09:59:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 10 monclient: build_initial_monmap 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 10 monclient: monmap: 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:26.907 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 1 librados: starting msgr at 2024-08-19T09:59:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 1 librados: starting objecter 2024-08-19T09:59:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 1 Processor -- start 2024-08-19T09:59:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 1 -- start start 2024-08-19T09:59:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 1 librados: setting wanted keys 2024-08-19T09:59:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 1 librados: calling monclient init 2024-08-19T09:59:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.890+0000 7f24e344f640 10 monclient: init 2024-08-19T09:59:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient: _add_conns ranks=[7,5,6,4,1,8,2,0,3] 2024-08-19T09:59:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient(hunting): picked mon.noname-i con 0x7f24dc0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient(hunting): picked mon.noname-h con 0x7f24dc0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient(hunting): picked mon.noname-d con 0x7f24dc13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24dc139ad0 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24dc13a990 con 0x7f24dc13f5c0 2024-08-19T09:59:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24dc142e70 con 0x7f24dc0a67c0 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e344f640 10 monclient(hunting): authenticate will time out at 1222.967301s 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e19ec640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f24dc13f5c0 0x7f24dc1d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34952/0 (socket says 172.21.15.1:34952) 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.891+0000 7f24e19ec640 1 -- 172.21.15.1:0/3077391628 learned_addr learned my addr 172.21.15.1:0/3077391628 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24e19ec640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.1:6792/0 conn(0x7f24dc13f5c0 legacy=0x7f24dc1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 client.?.objecter ms_handle_connect 0x7f24dc13f5c0 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1887387865 0 0) 0x7f24dc13a990 con 0x7f24dc13f5c0 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): my global_id is 174219 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24c0004eb0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 client.?.objecter ms_handle_connect 0x7f24dc0a6ba0 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24e09ea640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.82:6791/0 conn(0x7f24dc0a6ba0 legacy=0x7f24dc0a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 client.?.objecter ms_handle_connect 0x7f24dc0a67c0 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24e11eb640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.82:6792/0 conn(0x7f24dc0a67c0 legacy=0x7f24dc0a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1199285730 0 0) 0x7f24dc142e70 con 0x7f24dc0a67c0 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): my global_id is 144673 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24dc13a990 con 0x7f24dc0a67c0 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 537596981 0 0) 0x7f24dc139ad0 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): my global_id is 27176 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24dc142e70 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2700906599 0 0) 0x7f24c0004eb0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24dc139ad0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24d4002090 con 0x7f24dc13f5c0 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.893+0000 7f24d1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4265124599 0 0) 0x7f24dc13a990 con 0x7f24dc0a67c0 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24c0004eb0 con 0x7f24dc0a67c0 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24d8004d90 con 0x7f24dc0a67c0 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 471423392 0 0) 0x7f24dc142e70 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24dc13a990 con 0x7f24dc0a6ba0 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1590512570 0 0) 0x7f24dc139ad0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.82:6792/0 conn(0x7f24dc0a67c0 legacy=0x7f24dc0a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.82:6791/0 conn(0x7f24dc0a6ba0 legacy=0x7f24dc0a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 monclient: found mon.d 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24dc0a17d0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 5 monclient: authenticate success, global_id 174219 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 10 monclient: _renew_subs 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f24dc13a6b0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24d4001e20 con 0x7f24dc13f5c0 2024-08-19T09:59:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24d4005490 con 0x7f24dc13f5c0 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d1ffb640 10 monclient: dump: 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 10 client.174219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 10 monclient: _renew_subs 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24d37fe640 4 set_mon_vals no callback set 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f24dc0a0e30 con 0x7f24dc13f5c0 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.894+0000 7f24e344f640 1 librados: init done 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f24d40118f0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24e344f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24e344f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24e344f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24dc139da0 con 0x7f24dc13f5c0 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(361..361 src has 299..361) ==== 5064+0+0 (unknown 3986656978 0 0) 0x7f24d404e290 con 0x7f24dc13f5c0 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24d1ffb640 10 client.174219.objecter ms_dispatch 0x7f24dc000b90 osd_map(361..361 src has 299..361) 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24d1ffb640 3 client.174219.objecter handle_osd_map got epochs [361,361] > 0 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24d1ffb640 3 client.174219.objecter handle_osd_map decoding full epoch 361 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24d1ffb640 20 client.174219.objecter dump_active .. 0 homeless 2024-08-19T09:59:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.895+0000 7f24e11eb640 10 monclient: get_auth_request con 0x7f24c0041850 auth_method 0 2024-08-19T09:59:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.897+0000 7f24d1ffb640 10 client.174219.objecter ms_handle_connect 0x7f24c0041850 2024-08-19T09:59:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.900+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f24d4017810 con 0x7f24dc13f5c0 2024-08-19T09:59:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.900+0000 7f24d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:26.900+0000 7f24d1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:27.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.164+0000 7f24e344f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T09:59:27.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.165+0000 7f24e344f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T09:59:27.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.165+0000 7f24e344f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:27.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.165+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f24dc068f60 con 0x7f24dc13f5c0 2024-08-19T09:59:27.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f24d1ffb640 1 -- 172.21.15.1:0/3077391628 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f24d4017110 con 0x7f24dc13f5c0 2024-08-19T09:59:27.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f24d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T09:59:27.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f24d1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=3594561153,config=2586702516,health=1235890812,kv=1455257465,logm=1041072099}) 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.166+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2446637898,config=2586702516,health=1235890812,kv=1455257465,logm=4119006589}) 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=2870308617,config=2586702516,health=1235890812,kv=3949047626,logm=927064180}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=22,config=2,health=10,kv=11,logm=55} crc {auth=4220289858,config=2586702516,health=1920683110,kv=487780811,logm=823209475}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=2250441699,config=2586702516,health=1235890812,kv=3798231245,logm=14038527}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=799642440,config=2586702516,health=1235890812,kv=1455257465,logm=2430045211}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=2008279443,config=2586702516,health=292816977,kv=1595672624,logm=3140111389}) 2024-08-19T09:59:27.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3215920790,config=2586702516,health=1235890812,kv=1828024094,logm=3802443992}) 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.167+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=23,config=2,health=10,kv=12,logm=53} crc {auth=2934052278,config=2586702516,health=1920683110,kv=178558022,logm=2147616579}) 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4073529742}) 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2608875323}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2261463312}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2434362817}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3951659570}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1479776603}) 2024-08-19T09:59:27.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=646653097}) 2024-08-19T09:59:27.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605117214}) 2024-08-19T09:59:27.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2105900961}) 2024-08-19T09:59:27.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f24e344f640 10 librados: watch_flush enter 2024-08-19T09:59:27.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f24e344f640 10 librados: watch_flush exit 2024-08-19T09:59:27.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f24e344f640 20 client.174219.objecter shutdown clearing up homeless session... 2024-08-19T09:59:27.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.169+0000 7f24e344f640 10 client.174219.objecter successfully canceled tick 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.82:6812/2549104415 conn(0x7f24c0041850 legacy=0x7f24c0043d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 10 monclient: shutdown 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 >> v1:172.21.15.1:6792/0 conn(0x7f24dc13f5c0 legacy=0x7f24dc1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 shutdown_connections 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 >> 172.21.15.1:0/3077391628 conn(0x7f24dc0ac4b0 msgr2=0x7f24dc0aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 shutdown_connections 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 -- 172.21.15.1:0/3077391628 wait complete. 2024-08-19T09:59:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f24e344f640 1 librados: shutdown 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1097631573}) 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482514477}) 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279783294}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1931018067}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1178782400}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3252942910}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2943239472}) 2024-08-19T09:59:27.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478063812}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2609859453}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2500876200}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297256407}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1036930228}) 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=948595735}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2689284814}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=993876449}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=372265316}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793166012}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1576291336}) 2024-08-19T09:59:27.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657562524}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972091251}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1728668298}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=836519214}) 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3672384475}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1946117041}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2192973121}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735417540}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.174+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4075091714}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548891557}) 2024-08-19T09:59:27.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=808999064}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3483020391}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=781421686}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1291115322}) 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=631862802}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2375475925}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605967912}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.175+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2196382954}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1541842165}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3675841201}) 2024-08-19T09:59:27.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1523003856}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3574432491}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3979863548}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.176+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1824210213}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1157989809}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255887180}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.177+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1815334367}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1664547554}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722728664}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=75791219}) 2024-08-19T09:59:27.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3360749748}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3486143938}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1215345519}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3117051998}) 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892917002}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.178+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3327428270}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1464514867}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531412052}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=885481111}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2564466282}) 2024-08-19T09:59:27.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1791016272}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=21496516}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2951528029}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450310984}) 2024-08-19T09:59:27.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.180+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732037403}) 2024-08-19T09:59:27.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699882031}) 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3085234329}) 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1557374637}) 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=80591061}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1805276}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=978368278}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=797615802}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935203454}) 2024-08-19T09:59:27.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.190+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2832214689}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2823918088}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065694952}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4292694780}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3624040385}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3472522722}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=987539614}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1265855353}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713851803}) 2024-08-19T09:59:27.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.191+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=430218434}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2274837472}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3303021206}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4272939907}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1580708921}) 2024-08-19T09:59:27.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3108482383}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2208486378}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2624970870}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.192+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.193+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338671691}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.193+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4038552630}) 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2392205298}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147724348}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1702561017}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1423051}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3410308875}) 2024-08-19T09:59:27.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=58561374}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=211810044}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492132167}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.194+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=815369435}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=735166738}) 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712581530}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4291425998}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=198868142}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=360288307}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3419158803}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=368069369}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375169517}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.195+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3949896765}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.196+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.196+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.196+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225707301}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.196+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937769836}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1955867226}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1211336870}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3653323164}) 2024-08-19T09:59:27.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2654602567}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4234446101}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881652976}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.197+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3073170254}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4241974197}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848680873}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3332146116}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=724437606}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3196909020}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3119379561}) 2024-08-19T09:59:27.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3874044931}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048944925}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.198+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=769126384}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=421568604}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1957287430}) 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2444798764}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=720223979}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=922545501}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1681274265}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1070294123}) 2024-08-19T09:59:27.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558088070}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.199+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1401269626}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225402506}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327277490}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=798106986}) 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1692845130}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2448471995}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1549180750}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1658286942}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119322806}) 2024-08-19T09:59:27.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3955285147}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758305170}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=523232703}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2111523836}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3569750808}) 2024-08-19T09:59:27.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=987373988}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3283175481}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2364440167}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285122140}) 2024-08-19T09:59:27.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.202+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3219881206}) 2024-08-19T09:59:27.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417321056}) 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825379470}) 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=101919865}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=45523103}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3035077021}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1826677534}) 2024-08-19T09:59:27.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3745629255}) 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366709616}) 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.491+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=272737361}) 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3236953425}) 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675255319}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=341976578}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3700326973}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.492+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4047120317}) 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1037712168}) 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=246344464}) 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056091179}) 2024-08-19T09:59:27.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.493+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1012981648}) 2024-08-19T09:59:27.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2847261330}) 2024-08-19T09:59:27.513 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.513 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.513 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575322127}) 2024-08-19T09:59:27.514 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.514 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.514 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2430169879}) 2024-08-19T09:59:27.514 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.515 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.515 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2473518403}) 2024-08-19T09:59:27.515 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.516 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.516 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2684281535}) 2024-08-19T09:59:27.516 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.516 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.516 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=504904321}) 2024-08-19T09:59:27.516 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=436473504}) 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297716079}) 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.494+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=715493521}) 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.517 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3302362452}) 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396912140}) 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2750849447}) 2024-08-19T09:59:27.518 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.495+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.519 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.519 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=852909456}) 2024-08-19T09:59:27.519 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.519 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.519 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2387712149}) 2024-08-19T09:59:27.519 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.520 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.520 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3413104016}) 2024-08-19T09:59:27.520 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.520 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.520 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=903245440}) 2024-08-19T09:59:27.520 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.521 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781369915}) 2024-08-19T09:59:27.521 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.496+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1068093283}) 2024-08-19T09:59:27.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.866+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.866+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.866+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407507479}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246320395}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1037011489}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4048164956}) 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.870 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=792928922}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1028111762}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=154572095}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810243973}) 2024-08-19T09:59:27.871 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3128605606}) 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.870+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.870+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.870+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563973503}) 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.870+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927398012}) 2024-08-19T09:59:27.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1192590589}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3889031903}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1924203346}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1173960381}) 2024-08-19T09:59:27.874 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.875 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.875 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=135901198}) 2024-08-19T09:59:27.875 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:27.875 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940456236}) 2024-08-19T09:59:27.875 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.871+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=614143815}) 2024-08-19T09:59:28.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264216525}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3530142033}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1694386417}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1446322118}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.997+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3621035044}) 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3516798998}) 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4171057993}) 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244729475}) 2024-08-19T09:59:28.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:27.998+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=348838462}) 2024-08-19T09:59:28.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929942604}) 2024-08-19T09:59:28.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589287024}) 2024-08-19T09:59:28.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2487657029}) 2024-08-19T09:59:28.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1456362654}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=928976371}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=776677503}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.010+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2832240272}) 2024-08-19T09:59:28.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.010+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.010+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245239662}) 2024-08-19T09:59:28.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.010+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2637601359}) 2024-08-19T09:59:28.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111549248}) 2024-08-19T09:59:28.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959414647}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763539524}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1942541789}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=194489721}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2922611905}) 2024-08-19T09:59:28.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1257863513}) 2024-08-19T09:59:28.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991803363}) 2024-08-19T09:59:28.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3107604989}) 2024-08-19T09:59:28.205 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T09:59:28.206 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:59:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.306+0000 7f6b928cc640 10 monclient: get_monmap_and_config 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.306+0000 7f6b928cc640 10 monclient: build_initial_monmap 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.306+0000 7f6b928cc640 10 monclient: monmap: 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:28.307856+0000 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:28.307856+0000 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:28.309 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.307+0000 7f6b928cc640 1 Processor -- start 2024-08-19T09:59:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.307+0000 7f6b928cc640 1 -- start start 2024-08-19T09:59:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.307+0000 7f6b928cc640 10 monclient: init 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient: _add_conns ranks=[6,4,5,2,8,3,1,7,0] 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient(hunting): picked mon.noname-d con 0x7f6b8c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient(hunting): picked mon.noname-c con 0x7f6b8c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient(hunting): picked mon.noname-h con 0x7f6b8c13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c0a3b40 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c0a8130 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c06ede0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b928cc640 10 monclient(hunting): authenticate will time out at 1224.384317s 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b8b7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f6b8c0a6ba0 0x7f6b8c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46042/0 (socket says 172.21.15.1:46042) 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b8bfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f6b8c0a67c0 0x7f6b8c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34966/0 (socket says 172.21.15.1:34966) 2024-08-19T09:59:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.308+0000 7f6b8b7fe640 1 -- 172.21.15.1:0/4184205426 learned_addr learned my addr 172.21.15.1:0/4184205426 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:28.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8b7fe640 1 -- 172.21.15.1:0/4184205426 >> v1:172.21.15.1:6791/0 conn(0x7f6b8c0a6ba0 legacy=0x7f6b8c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8bfff640 1 -- 172.21.15.1:0/4184205426 >> v1:172.21.15.1:6792/0 conn(0x7f6b8c0a67c0 legacy=0x7f6b8c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b90e69640 1 -- 172.21.15.1:0/4184205426 >> v1:172.21.15.82:6791/0 conn(0x7f6b8c13f5c0 legacy=0x7f6b8c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3274881944 0 0) 0x7f6b8c0a3b40 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): my global_id is 27517 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b74003520 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1212269020 0 0) 0x7f6b8c06ede0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): my global_id is 174228 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8c0a3b40 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326927916 0 0) 0x7f6b8c0a8130 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 10 monclient(hunting): my global_id is 27185 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8c06ede0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.310+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4038802511 0 0) 0x7f6b74003520 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 >> v1:172.21.15.1:6792/0 conn(0x7f6b8c0a67c0 legacy=0x7f6b8c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 >> v1:172.21.15.82:6791/0 conn(0x7f6b8c13f5c0 legacy=0x7f6b8c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: _finish_hunting 0 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 monclient: found mon.noname-c 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:28.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c0a92a0 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: _finish_auth 0 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: _check_auth_tickets 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 5 monclient: authenticate success, global_id 27517 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6b80001ae0 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: dump: 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.323 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:28.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b800048a0 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 1 -- 172.21.15.1:0/4184205426 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6b80004e10 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 4 set_mon_vals no callback set 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b8affd640 10 monclient: dump: 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.325 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 10 monclient: shutdown 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 1 -- 172.21.15.1:0/4184205426 >> v1:172.21.15.1:6791/0 conn(0x7f6b8c0a6ba0 legacy=0x7f6b8c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 1 -- 172.21.15.1:0/4184205426 shutdown_connections 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 1 -- 172.21.15.1:0/4184205426 >> 172.21.15.1:0/4184205426 conn(0x7f6b8c0ac4b0 msgr2=0x7f6b8c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 1 -- 172.21.15.1:0/4184205426 shutdown_connections 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.311+0000 7f6b928cc640 1 -- 172.21.15.1:0/4184205426 wait complete. 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 10 monclient: build_initial_monmap 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 10 monclient: monmap: 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:28.327 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:28.328 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 1 librados: starting msgr at 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 1 librados: starting objecter 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 1 Processor -- start 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 1 -- start start 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 1 librados: setting wanted keys 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 1 librados: calling monclient init 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.312+0000 7f6b928cc640 10 monclient: init 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient: _add_conns ranks=[6,3,8,7,5,4,2,0,1] 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient(hunting): picked mon.noname-d con 0x7f6b8c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient(hunting): picked mon.noname-g con 0x7f6b8c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient(hunting): picked mon.noname-e con 0x7f6b8c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:28.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c139ee0 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c13ace0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c0a0f40 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b8bfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f6b8c0a67c0 0x7f6b8c0a0120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34974/0 (socket says 172.21.15.1:34974) 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b8bfff640 1 -- 172.21.15.1:0/3400583513 learned_addr learned my addr 172.21.15.1:0/3400583513 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.313+0000 7f6b928cc640 10 monclient(hunting): authenticate will time out at 1224.389372s 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b8bfff640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.1:6792/0 conn(0x7f6b8c0a67c0 legacy=0x7f6b8c0a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 client.?.objecter ms_handle_connect 0x7f6b8c0a67c0 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b90e69640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.1:6793/0 conn(0x7f6b8c13f5c0 legacy=0x7f6b8c1d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 client.?.objecter ms_handle_connect 0x7f6b8c13f5c0 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b8b7fe640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.82:6790/0 conn(0x7f6b8c0a6ba0 legacy=0x7f6b8c0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 client.?.objecter ms_handle_connect 0x7f6b8c0a6ba0 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139991023 0 0) 0x7f6b8c13ace0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): my global_id is 174237 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b78004eb0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873012624 0 0) 0x7f6b8c0a0f40 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): my global_id is 174176 2024-08-19T09:59:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8c13ace0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1466595525 0 0) 0x7f6b8c139ee0 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): my global_id is 135201 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8c0a0f40 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2262949572 0 0) 0x7f6b8c13ace0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b8c139ee0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3943198508 0 0) 0x7f6b78004eb0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b8c13ace0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6b84002070 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.315+0000 7f6b88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): dump: 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6b7c0047e0 con 0x7f6b8c0a67c0 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): dump: 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1536435424 0 0) 0x7f6b8c0a0f40 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b78004eb0 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6b80004d90 con 0x7f6b8c0a6ba0 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient(hunting): dump: 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2094950747 0 0) 0x7f6b8c139ee0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.1:6792/0 conn(0x7f6b8c0a67c0 legacy=0x7f6b8c0a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.82:6790/0 conn(0x7f6b8c0a6ba0 legacy=0x7f6b8c0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:59:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 monclient: found mon.e 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c142bf0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: _finish_auth 0 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 5 monclient: authenticate success, global_id 174176 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 10 monclient: _renew_subs 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c0a1ba0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 10 client.174176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 10 monclient: _renew_subs 2024-08-19T09:59:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b8c0a2a00 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b928cc640 1 librados: init done 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b84003e70 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6b84003a00 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b8a7fc640 4 set_mon_vals no callback set 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.316+0000 7f6b88ff9640 10 monclient: dump: 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6b84004460 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b928cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b928cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b928cc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6c005080 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(363..363 src has 299..363) ==== 5449+0+0 (unknown 2017952525 0 0) 0x7f6b8404e510 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b88ff9640 10 client.174176.objecter ms_dispatch 0x7f6b8c000b90 osd_map(363..363 src has 299..363) 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b88ff9640 3 client.174176.objecter handle_osd_map got epochs [363,363] > 0 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b88ff9640 3 client.174176.objecter handle_osd_map decoding full epoch 363 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b88ff9640 20 client.174176.objecter dump_active .. 0 homeless 2024-08-19T09:59:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.317+0000 7f6b8bfff640 10 monclient: get_auth_request con 0x7f6b780417f0 auth_method 0 2024-08-19T09:59:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.319+0000 7f6b88ff9640 10 client.174176.objecter ms_handle_connect 0x7f6b780417f0 2024-08-19T09:59:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.322+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6b840178b0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.322+0000 7f6b88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.322+0000 7f6b88ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.575+0000 7f6b928cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.575+0000 7f6b928cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.575+0000 7f6b928cc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.575+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6b6c005370 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.576+0000 7f6b88ff9640 1 -- 172.21.15.1:0/3400583513 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2702 (unknown 254746657 0 202104995) 0x7f6b840171b0 con 0x7f6b8c13f5c0 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.576+0000 7f6b88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:59:28.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.576+0000 7f6b88ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:28.578 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":86,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:59:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 10 librados: watch_flush enter 2024-08-19T09:59:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 10 librados: watch_flush exit 2024-08-19T09:59:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 20 client.174176.objecter shutdown clearing up homeless session... 2024-08-19T09:59:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 10 client.174176.objecter successfully canceled tick 2024-08-19T09:59:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6b780417f0 legacy=0x7f6b78043cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 10 monclient: shutdown 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 >> v1:172.21.15.1:6793/0 conn(0x7f6b8c13f5c0 legacy=0x7f6b8c1d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 shutdown_connections 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 >> 172.21.15.1:0/3400583513 conn(0x7f6b8c0ac4b0 msgr2=0x7f6b8c0a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 shutdown_connections 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 -- 172.21.15.1:0/3400583513 wait complete. 2024-08-19T09:59:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.578+0000 7f6b928cc640 1 librados: shutdown 2024-08-19T09:59:28.589 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T09:59:28.589 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T09:59:28.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T09:59:28.590 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:59:28.641 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=679069206}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=96602471}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681972841}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2690202482}) 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4274236893}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=207177773}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3486874535}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278888}) 2024-08-19T09:59:28.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.639+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=598152944}) 2024-08-19T09:59:28.645 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.645 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.645 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2684673789}) 2024-08-19T09:59:28.645 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.645 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=299544991}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=841444342}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1122673426}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=127777011}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.646 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1881706973}) 2024-08-19T09:59:28.647 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.647 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.647 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=159303388}) 2024-08-19T09:59:28.647 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.647 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651486923}) 2024-08-19T09:59:28.647 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.641+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=439349211}) 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.689+0000 7f577af8a640 10 monclient: get_monmap_and_config 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.689+0000 7f577af8a640 10 monclient: build_initial_monmap 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.689+0000 7f577af8a640 10 monclient: monmap: 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:28.690166+0000 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:28.690166+0000 2024-08-19T09:59:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:28.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.689+0000 7f577af8a640 1 Processor -- start 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 1 -- start start 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient: init 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient: _add_conns ranks=[4,0,3,2,7,5,8,6,1] 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient(hunting): picked mon.noname-c con 0x7f57740a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient(hunting): picked mon.noname-a con 0x7f57740a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient(hunting): picked mon.noname-g con 0x7f577413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740a3b40 con 0x7f57740a6ba0 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740a8130 con 0x7f577413f5c0 2024-08-19T09:59:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f577406ede0 con 0x7f57740a67c0 2024-08-19T09:59:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f577af8a640 10 monclient(hunting): authenticate will time out at 1224.766599s 2024-08-19T09:59:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f5778d26640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f57740a67c0 0x7f57740a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46048/0 (socket says 172.21.15.1:46048) 2024-08-19T09:59:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f5773fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f57740a6ba0 0x7f57740a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37138/0 (socket says 172.21.15.1:37138) 2024-08-19T09:59:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.690+0000 7f5778d26640 1 -- 172.21.15.1:0/3464243522 learned_addr learned my addr 172.21.15.1:0/3464243522 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.692+0000 7f5773fff640 1 -- 172.21.15.1:0/3464243522 >> v1:172.21.15.1:6789/0 conn(0x7f57740a6ba0 legacy=0x7f57740a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.692+0000 7f5778d26640 1 -- 172.21.15.1:0/3464243522 >> v1:172.21.15.1:6791/0 conn(0x7f57740a67c0 legacy=0x7f57740a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.692+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869789065 0 0) 0x7f577406ede0 con 0x7f57740a67c0 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.692+0000 7f57737fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): my global_id is 27526 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.692+0000 7f5779527640 1 -- 172.21.15.1:0/3464243522 >> v1:172.21.15.82:6790/0 conn(0x7f577413f5c0 legacy=0x7f57741419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5764003520 con 0x7f57740a67c0 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230899562 0 0) 0x7f57740a3b40 con 0x7f57740a6ba0 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): my global_id is 174177 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f577406ede0 con 0x7f57740a6ba0 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675908587 0 0) 0x7f57740a8130 con 0x7f577413f5c0 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient(hunting): my global_id is 135210 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57740a3b40 con 0x7f577413f5c0 2024-08-19T09:59:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1602519631 0 0) 0x7f5764003520 con 0x7f57740a67c0 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 >> v1:172.21.15.82:6790/0 conn(0x7f577413f5c0 legacy=0x7f57741419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 >> v1:172.21.15.1:6789/0 conn(0x7f57740a6ba0 legacy=0x7f57740a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 monclient: found mon.noname-c 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57740a92a0 con 0x7f57740a67c0 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5768001a00 con 0x7f57740a67c0 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: dump: 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 5 monclient: authenticate success, global_id 27526 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5768001cd0 con 0x7f57740a67c0 2024-08-19T09:59:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 1 -- 172.21.15.1:0/3464243522 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5768004be0 con 0x7f57740a67c0 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f57737fe640 10 monclient: dump: 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 4 set_mon_vals no callback set 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 10 monclient: shutdown 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 1 -- 172.21.15.1:0/3464243522 >> v1:172.21.15.1:6791/0 conn(0x7f57740a67c0 legacy=0x7f57740a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 1 -- 172.21.15.1:0/3464243522 shutdown_connections 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 1 -- 172.21.15.1:0/3464243522 >> 172.21.15.1:0/3464243522 conn(0x7f57740ac4b0 msgr2=0x7f57740a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 1 -- 172.21.15.1:0/3464243522 shutdown_connections 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.693+0000 7f577af8a640 1 -- 172.21.15.1:0/3464243522 wait complete. 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 10 monclient: build_initial_monmap 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 10 monclient: monmap: 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 1 librados: starting msgr at 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 1 librados: starting objecter 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 1 Processor -- start 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 1 -- start start 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 1 librados: setting wanted keys 2024-08-19T09:59:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 1 librados: calling monclient init 2024-08-19T09:59:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.694+0000 7f577af8a640 10 monclient: init 2024-08-19T09:59:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient: _add_conns ranks=[0,5,7,8,6,3,4,1,2] 2024-08-19T09:59:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient(hunting): picked mon.noname-a con 0x7f57740a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient(hunting): picked mon.noname-h con 0x7f57740a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient(hunting): picked mon.noname-i con 0x7f577413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5774139a70 con 0x7f57740a67c0 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740a9cc0 con 0x7f57740a6ba0 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f577413bdb0 con 0x7f577413f5c0 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f577af8a640 10 monclient(hunting): authenticate will time out at 1224.771581s 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f5778d26640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f57740a67c0 0x7f57741ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37148/0 (socket says 172.21.15.1:37148) 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.695+0000 7f5778d26640 1 -- 172.21.15.1:0/721663421 learned_addr learned my addr 172.21.15.1:0/721663421 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f5778d26640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f57740a67c0 legacy=0x7f57741ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 client.?.objecter ms_handle_connect 0x7f57740a67c0 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396991560 0 0) 0x7f5774139a70 con 0x7f57740a67c0 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 monclient(hunting): my global_id is 174186 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5758004eb0 con 0x7f57740a67c0 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f5779527640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.82:6792/0 conn(0x7f577413f5c0 legacy=0x7f57741d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 client.?.objecter ms_handle_connect 0x7f577413f5c0 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 client.?.objecter ms_handle_connect 0x7f57740a6ba0 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f57717fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.697+0000 7f5773fff640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.82:6791/0 conn(0x7f57740a6ba0 legacy=0x7f57741ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870856606 0 0) 0x7f577413bdb0 con 0x7f577413f5c0 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): my global_id is 144682 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5774139a70 con 0x7f577413f5c0 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 794821496 0 0) 0x7f5758004eb0 con 0x7f57740a67c0 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f577413bdb0 con 0x7f57740a67c0 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205329465 0 0) 0x7f577406ede0 con 0x7f57740a6ba0 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): my global_id is 27194 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5758004eb0 con 0x7f57740a6ba0 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5768004b20 con 0x7f57740a67c0 2024-08-19T09:59:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): dump: 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3034757139 0 0) 0x7f577413bdb0 con 0x7f57740a67c0 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.82:6792/0 conn(0x7f577413f5c0 legacy=0x7f57741d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.82:6791/0 conn(0x7f57740a6ba0 legacy=0x7f57741ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: _finish_hunting 0 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 monclient: found mon.a 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57741cf020 con 0x7f57740a67c0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: _finish_auth 0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: _check_auth_tickets 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 5 monclient: authenticate success, global_id 174186 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 10 monclient: _renew_subs 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f577413af40 con 0x7f57740a67c0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 10 client.174186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 10 monclient: _renew_subs 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f577413a880 con 0x7f57740a67c0 2024-08-19T09:59:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f577af8a640 1 librados: init done 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5768004cc0 con 0x7f57740a67c0 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5768001cd0 con 0x7f57740a67c0 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f5772ffd640 4 set_mon_vals no callback set 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.698+0000 7f57717fa640 10 monclient: dump: 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f577af8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57540065c0 con 0x7f5754002860 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5768011810 con 0x7f57740a67c0 2024-08-19T09:59:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(363..363 src has 299..363) ==== 5449+0+0 (unknown 2017952525 0 0) 0x7f576804e180 con 0x7f57740a67c0 2024-08-19T09:59:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f57717fa640 10 client.174186.objecter ms_dispatch 0x7f5774000b90 osd_map(363..363 src has 299..363) 2024-08-19T09:59:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f57717fa640 3 client.174186.objecter handle_osd_map got epochs [363,363] > 0 2024-08-19T09:59:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f57717fa640 3 client.174186.objecter handle_osd_map decoding full epoch 363 2024-08-19T09:59:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f57717fa640 20 client.174186.objecter dump_active .. 0 homeless 2024-08-19T09:59:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.699+0000 7f5779527640 10 monclient: get_auth_request con 0x7f5758041280 auth_method 0 2024-08-19T09:59:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f5773fff640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f5754002860 legacy=0x7f5754004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 client.174186.objecter ms_handle_connect 0x7f5754002860 2024-08-19T09:59:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 client.174186.objecter resend_mon_ops 2024-08-19T09:59:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1306288275 0 0) 0x7f57540065c0 con 0x7f5754002860 2024-08-19T09:59:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient(hunting): my global_id is 174195 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f577413bdb0 con 0x7f5754002860 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1393856394 0 0) 0x7f577413bdb0 con 0x7f5754002860 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57540065c0 con 0x7f5754002860 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5760001c10 con 0x7f5754002860 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 10 monclient: dump: 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2332549345 0 0) 0x7f57540065c0 con 0x7f5754002860 2024-08-19T09:59:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.701+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f575400a160 con 0x7f5754002860 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.702+0000 7f57717fa640 10 client.174186.objecter ms_handle_connect 0x7f5758041280 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.702+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f57600047e0 con 0x7f5754002860 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.702+0000 7f57717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.702+0000 7f57717fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.702+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f5754002860 legacy=0x7f5754004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.702+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f5754002860 legacy=0x7f5754004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.708+0000 7f577af8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.708+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5754009e50 con 0x7f5754009110 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f5773fff640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f5754009110 legacy=0x7f57540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 10 client.174186.objecter ms_handle_connect 0x7f5754009110 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 10 client.174186.objecter resend_mon_ops 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024071790 0 0) 0x7f5754009e50 con 0x7f5754009110 2024-08-19T09:59:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 10 monclient(hunting): my global_id is 174204 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.710+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57540065c0 con 0x7f5754009110 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3840319025 0 0) 0x7f57540065c0 con 0x7f5754009110 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5754009e50 con 0x7f5754009110 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5760004f40 con 0x7f5754009110 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 10 monclient: dump: 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2854231747 0 0) 0x7f5754009e50 con 0x7f5754009110 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f575400c3a0 con 0x7f5754009110 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3063543343) 0x7f57600072e0 con 0x7f5754009110 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f5754009110 legacy=0x7f57540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f57717fa640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f5754009110 legacy=0x7f57540095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:28.728 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 18751, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:28.729 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:28.730 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.731 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:28.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:28.733 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:28.734 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.735 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:28.739 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.740 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:28.745 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.746 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.747 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.748 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.749 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:28.750 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 10 librados: watch_flush enter 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 10 librados: watch_flush exit 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 20 client.174186.objecter shutdown clearing up homeless session... 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 10 client.174186.objecter successfully canceled tick 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5758041280 legacy=0x7f5758043740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 10 monclient: shutdown 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 >> v1:172.21.15.1:6789/0 conn(0x7f57740a67c0 legacy=0x7f57741ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f5779527640 1 -- 172.21.15.1:0/721663421 reap_dead start 2024-08-19T09:59:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f5779527640 1 -- 172.21.15.1:0/721663421 reap_dead start 2024-08-19T09:59:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 shutdown_connections 2024-08-19T09:59:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 >> 172.21.15.1:0/721663421 conn(0x7f57740ac4b0 msgr2=0x7f57741421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.711+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 shutdown_connections 2024-08-19T09:59:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.712+0000 7f577af8a640 1 -- 172.21.15.1:0/721663421 wait complete. 2024-08-19T09:59:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.712+0000 7f577af8a640 1 librados: shutdown 2024-08-19T09:59:28.753 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:28.753 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T09:59:28.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.851+0000 7fd151869640 10 monclient: get_monmap_and_config 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.851+0000 7fd151869640 10 monclient: build_initial_monmap 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.851+0000 7fd151869640 10 monclient: monmap: 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:28.852864+0000 2024-08-19T09:59:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:28.852864+0000 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:28.854 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.852+0000 7fd151869640 1 Processor -- start 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.852+0000 7fd151869640 1 -- start start 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.852+0000 7fd151869640 10 monclient: init 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.852+0000 7fd151869640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 10 monclient: _add_conns ranks=[8,0,3,2,5,1,4,7,6] 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 10 monclient(hunting): picked mon.noname-e con 0x7fd14c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 10 monclient(hunting): picked mon.noname-a con 0x7fd14c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 10 monclient(hunting): picked mon.noname-g con 0x7fd14c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c0a3b40 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c0a8130 con 0x7fd14c13f5c0 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c06ede0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd151869640 10 monclient(hunting): authenticate will time out at 1224.929128s 2024-08-19T09:59:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd14a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd14c0a6ba0 0x7fd14c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37164/0 (socket says 172.21.15.1:37164) 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.853+0000 7fd14a7fc640 1 -- 172.21.15.1:0/4201474677 learned_addr learned my addr 172.21.15.1:0/4201474677 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd14affd640 1 -- 172.21.15.1:0/4201474677 >> v1:172.21.15.1:6793/0 conn(0x7fd14c0a67c0 legacy=0x7fd14c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd14a7fc640 1 -- 172.21.15.1:0/4201474677 >> v1:172.21.15.1:6789/0 conn(0x7fd14c0a6ba0 legacy=0x7fd14c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd14b7fe640 1 -- 172.21.15.1:0/4201474677 >> v1:172.21.15.82:6790/0 conn(0x7fd14c13f5c0 legacy=0x7fd14c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3198670880 0 0) 0x7fd14c0a3b40 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): my global_id is 174213 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd138003520 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911819636 0 0) 0x7fd14c06ede0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): my global_id is 174185 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c0a3b40 con 0x7fd14c0a67c0 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158905151 0 0) 0x7fd14c0a8130 con 0x7fd14c13f5c0 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient(hunting): my global_id is 135219 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c06ede0 con 0x7fd14c13f5c0 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 589139053 0 0) 0x7fd138003520 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 >> v1:172.21.15.1:6793/0 conn(0x7fd14c0a67c0 legacy=0x7fd14c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 >> v1:172.21.15.82:6790/0 conn(0x7fd14c13f5c0 legacy=0x7fd14c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 monclient: found mon.noname-a 2024-08-19T09:59:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c0a92a0 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd134001ae0 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd149ffb640 10 monclient: dump: 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd151869640 5 monclient: authenticate success, global_id 174213 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.855+0000 7fd151869640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1340048a0 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd149ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd149ffb640 1 -- 172.21.15.1:0/4201474677 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd134004e10 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 4 set_mon_vals no callback set 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd149ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd149ffb640 10 monclient: dump: 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.874 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 10 monclient: shutdown 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 1 -- 172.21.15.1:0/4201474677 >> v1:172.21.15.1:6789/0 conn(0x7fd14c0a6ba0 legacy=0x7fd14c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 1 -- 172.21.15.1:0/4201474677 shutdown_connections 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 1 -- 172.21.15.1:0/4201474677 >> 172.21.15.1:0/4201474677 conn(0x7fd14c0ac4b0 msgr2=0x7fd14c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 1 -- 172.21.15.1:0/4201474677 shutdown_connections 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.856+0000 7fd151869640 1 -- 172.21.15.1:0/4201474677 wait complete. 2024-08-19T09:59:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 10 monclient: build_initial_monmap 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 10 monclient: monmap: 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 1 librados: starting msgr at 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 1 librados: starting objecter 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 1 Processor -- start 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 1 -- start start 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 1 librados: setting wanted keys 2024-08-19T09:59:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 1 librados: calling monclient init 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.857+0000 7fd151869640 10 monclient: init 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient: _add_conns ranks=[0,7,4,8,6,1,5,3,2] 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient(hunting): picked mon.noname-a con 0x7fd14c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient(hunting): picked mon.noname-i con 0x7fd14c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient(hunting): picked mon.noname-c con 0x7fd14c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c13a6b0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c13a990 con 0x7fd14c13f5c0 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c139c70 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd151869640 10 monclient(hunting): authenticate will time out at 1224.934111s 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd14affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd14c0a67c0 0x7fd14c09fe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37170/0 (socket says 172.21.15.1:37170) 2024-08-19T09:59:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.858+0000 7fd14affd640 1 -- 172.21.15.1:0/3596559076 learned_addr learned my addr 172.21.15.1:0/3596559076 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd14b7fe640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6791/0 conn(0x7fd14c13f5c0 legacy=0x7fd14c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd14affd640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6789/0 conn(0x7fd14c0a67c0 legacy=0x7fd14c09fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 client.?.objecter ms_handle_connect 0x7fd14c13f5c0 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 client.?.objecter ms_handle_connect 0x7fd14c0a67c0 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4113690111 0 0) 0x7fd14c13a6b0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): my global_id is 174222 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd12c004eb0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 client.?.objecter ms_handle_connect 0x7fd14c0a6ba0 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd14a7fc640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.82:6792/0 conn(0x7fd14c0a6ba0 legacy=0x7fd14c0a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1675683571 0 0) 0x7fd14c13a990 con 0x7fd14c13f5c0 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): my global_id is 27535 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c13a6b0 con 0x7fd14c13f5c0 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099020357 0 0) 0x7fd14c139c70 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): my global_id is 144691 2024-08-19T09:59:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c13a990 con 0x7fd14c0a6ba0 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1031582484 0 0) 0x7fd12c004eb0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd14c139c70 con 0x7fd14c0a67c0 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd144003990 con 0x7fd14c0a67c0 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): dump: 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 855397473 0 0) 0x7fd14c13a6b0 con 0x7fd14c13f5c0 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd12c004eb0 con 0x7fd14c13f5c0 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd13c001cb0 con 0x7fd14c13f5c0 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient(hunting): dump: 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3271611493 0 0) 0x7fd14c139c70 con 0x7fd14c0a67c0 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.82:6792/0 conn(0x7fd14c0a6ba0 legacy=0x7fd14c0a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6791/0 conn(0x7fd14c13f5c0 legacy=0x7fd14c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 monclient: found mon.a 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c0a0c70 con 0x7fd14c0a67c0 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient: _finish_auth 0 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd143fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.860+0000 7fd151869640 5 monclient: authenticate success, global_id 174222 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 10 monclient: _renew_subs 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c0a0f00 con 0x7fd14c0a67c0 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 10 client.174222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 10 monclient: _renew_subs 2024-08-19T09:59:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd14c1d7b50 con 0x7fd14c0a67c0 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 1 librados: init done 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd144003ee0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd144004450 con 0x7fd14c0a67c0 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd1497fa640 4 set_mon_vals no callback set 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 10 monclient: dump: 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd1300065c0 con 0x7fd130002860 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.861+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd1440045f0 con 0x7fd14c0a67c0 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.862+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(363..363 src has 299..363) ==== 5449+0+0 (unknown 2017952525 0 0) 0x7fd14404e690 con 0x7fd14c0a67c0 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.862+0000 7fd143fff640 10 client.174222.objecter ms_dispatch 0x7fd14c000b90 osd_map(363..363 src has 299..363) 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.862+0000 7fd143fff640 3 client.174222.objecter handle_osd_map got epochs [363,363] > 0 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.862+0000 7fd143fff640 3 client.174222.objecter handle_osd_map decoding full epoch 363 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.862+0000 7fd143fff640 20 client.174222.objecter dump_active .. 0 homeless 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.862+0000 7fd14b7fe640 10 monclient: get_auth_request con 0x7fd12c041410 auth_method 0 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd14a7fc640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6790/0 conn(0x7fd130002860 legacy=0x7fd130004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 client.174222.objecter ms_handle_connect 0x7fd130002860 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 client.174222.objecter resend_mon_ops 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161186150 0 0) 0x7fd1300065c0 con 0x7fd130002860 2024-08-19T09:59:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient(hunting): my global_id is 164414 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c139c70 con 0x7fd130002860 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4271020347 0 0) 0x7fd14c139c70 con 0x7fd130002860 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1300065c0 con 0x7fd130002860 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd134001d30 con 0x7fd130002860 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient: dump: 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.889 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.890 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1919333561 0 0) 0x7fd1300065c0 con 0x7fd130002860 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd13000a160 con 0x7fd130002860 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 client.174222.objecter ms_handle_connect 0x7fd12c041410 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd1340048a0 con 0x7fd130002860 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6790/0 conn(0x7fd130002860 legacy=0x7fd130004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.864+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6790/0 conn(0x7fd130002860 legacy=0x7fd130004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:28.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.871+0000 7fd151869640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:28.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.871+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd130009e70 con 0x7fd130009130 2024-08-19T09:59:28.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd14a7fc640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6790/0 conn(0x7fd130009130 legacy=0x7fd1300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:28.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 client.174222.objecter ms_handle_connect 0x7fd130009130 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 client.174222.objecter resend_mon_ops 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187756197 0 0) 0x7fd130009e70 con 0x7fd130009130 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient(hunting): my global_id is 164423 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1300065c0 con 0x7fd130009130 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2496197222 0 0) 0x7fd1300065c0 con 0x7fd130009130 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd130009e70 con 0x7fd130009130 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd134005ad0 con 0x7fd130009130 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 10 monclient: dump: 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:28.895 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:28.896 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:28.896 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:28.896 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:28.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3530474104 0 0) 0x7fd130009e70 con 0x7fd130009130 2024-08-19T09:59:28.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.873+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd13000c3f0 con 0x7fd130009130 2024-08-19T09:59:28.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 272227847) 0x7fd134005ea0 con 0x7fd130009130 2024-08-19T09:59:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd143fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd143fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6790/0 conn(0x7fd130009130 legacy=0x7fd1300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd143fff640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6790/0 conn(0x7fd130009130 legacy=0x7fd1300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390690377}) 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649758782}) 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1192685748}) 2024-08-19T09:59:28.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 155688, 2024-08-19T09:59:28.899 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:28.900 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:28.901 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:28.902 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:28.903 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:28.904 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:28.905 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:28.909 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:28.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.914 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:28.915 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.916 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:28.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:28.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:28.919 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:28.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:28.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:28.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:28.920 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:28.920 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:28.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2222953191}) 2024-08-19T09:59:28.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2669717326}) 2024-08-19T09:59:28.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4051303254}) 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3092628836}) 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612966526}) 2024-08-19T09:59:28.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.874+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=166000605}) 2024-08-19T09:59:28.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 10 librados: watch_flush enter 2024-08-19T09:59:28.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 10 librados: watch_flush exit 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 20 client.174222.objecter shutdown clearing up homeless session... 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 10 client.174222.objecter successfully canceled tick 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd12c041410 legacy=0x7fd12c0438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 10 monclient: shutdown 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.874+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 >> v1:172.21.15.1:6789/0 conn(0x7fd14c0a67c0 legacy=0x7fd14c09fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd14b7fe640 1 -- 172.21.15.1:0/3596559076 reap_dead start 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd14b7fe640 1 -- 172.21.15.1:0/3596559076 reap_dead start 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 shutdown_connections 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 >> 172.21.15.1:0/3596559076 conn(0x7fd14c0ac4b0 msgr2=0x7fd14c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 shutdown_connections 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd151869640 1 -- 172.21.15.1:0/3596559076 wait complete. 2024-08-19T09:59:28.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:28.875+0000 7fd151869640 1 librados: shutdown 2024-08-19T09:59:28.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856712654}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=580905449}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3325847069}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=392080785}) 2024-08-19T09:59:28.927 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.875+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4015647945}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2920132826}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=215099171}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432738654}) 2024-08-19T09:59:28.928 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.876+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4023121738}) 2024-08-19T09:59:28.929 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:28.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T09:59:28.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875143156}) 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2589059187}) 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3706784842}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2167131259}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1193107570}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=242727406}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3081826531}) 2024-08-19T09:59:28.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575247469}) 2024-08-19T09:59:28.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.906+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1612170244}) 2024-08-19T09:59:28.979 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1746303269}) 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144417994}) 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.977+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3071309486}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1443409701}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=887768625}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2163533792}) 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3786090771}) 2024-08-19T09:59:28.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:28.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136308145}) 2024-08-19T09:59:28.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.978+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1364010815}) 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259305743}) 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3818705944}) 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2210107723}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2363101766}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=803644905}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2083534356}) 2024-08-19T09:59:29.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1615316298}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876009859}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.979+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1221309207}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807533064}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110819281}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2335335097}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=722552280}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3937661440}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=397853013}) 2024-08-19T09:59:29.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=602652399}) 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.980+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726677282}) 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=602808297}) 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107372070}) 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2983485808}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=826376904}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=366178025}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1225267445}) 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=214277101}) 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3686566297}) 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053192015}) 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4200915204}) 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993302443}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316145384}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4100971737}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1281235463}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=260956384}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1342781363}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3287346819}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321966476}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3433139909}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642504405}) 2024-08-19T09:59:29.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1217109296}) 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1865934033}) 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3095970264}) 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3712822738}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4070643229}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1030104264}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560879762}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=34269196}) 2024-08-19T09:59:29.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=417384159}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3607846587}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2114571771}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893436139}) 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3793970250}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2659754105}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2990168659}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993706031}) 2024-08-19T09:59:29.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.986+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1151338197}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1688764014}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376419956}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3529631030}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2045582658}) 2024-08-19T09:59:29.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1046514222}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=898813296}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2110807180}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856659599}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.987+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2424299064}) 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209481204}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180853239}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3817790867}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3437510883}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2157143123}) 2024-08-19T09:59:29.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3086120985}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.988+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.989+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.989+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=481743709}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.989+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.989+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756190082}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.989+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1724681008}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856554408}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690510277}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2622970903}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2787090903}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3572765536}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1646486837}) 2024-08-19T09:59:29.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2343720123}) 2024-08-19T09:59:29.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557981847}) 2024-08-19T09:59:29.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:28.990+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2124912442}) 2024-08-19T09:59:29.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147309867}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606259552}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1277873152}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=878589936}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2657348230}) 2024-08-19T09:59:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3464863490}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=613295347}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319148058}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=796232777}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687876718}) 2024-08-19T09:59:29.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396616026}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1653045094}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3275532981}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1669239309}) 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2372924305}) 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812941467}) 2024-08-19T09:59:29.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121503171}) 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=894886674}) 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50582221}) 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036479132}) 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1110696538}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2720077634}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1253295064}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1245011265}) 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2778134415}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157380731}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2109600633}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1342186310}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117620622}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=66073352}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=522674441}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1765688164}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2682647926}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4003002587}) 2024-08-19T09:59:29.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529707812}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.020+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=143070199}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524059501}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554991865}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2692392436}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4015316571}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1540183687}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1909455963}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2313631231}) 2024-08-19T09:59:29.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407074718}) 2024-08-19T09:59:29.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.022+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3494343547}) 2024-08-19T09:59:29.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1896092617}) 2024-08-19T09:59:29.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3994839912}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1443228683}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=752704915}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1742190646}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=423745508}) 2024-08-19T09:59:29.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1687638412}) 2024-08-19T09:59:29.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696792445}) 2024-08-19T09:59:29.041 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.030+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2616029201}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649165869}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1350019069}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3931434378}) 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.060 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2452950336}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=226080969}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.057+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1606010601}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.058+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.058+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.058+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=213788865}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.058+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.058+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197405721}) 2024-08-19T09:59:29.061 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.058+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2041069062}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3025839443}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310153499}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3965371335}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.065 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3418971575}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1383412191}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1357500752}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=425026843}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565393267}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.060+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2381475838}) 2024-08-19T09:59:29.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=617229175}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220551924}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1942601940}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2141635752}) 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2708323903}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=657378377}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1657450122}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662479312}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.061+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3536350281}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1815344690}) 2024-08-19T09:59:29.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1329606065}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3166338074}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2959641674}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2463284278}) 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.069 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2762556320}) 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2305820289}) 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030574178}) 2024-08-19T09:59:29.070 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.063+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3656421494}) 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.065+0000 7f50a5c67640 10 monclient: get_monmap_and_config 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.065+0000 7f50a5c67640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.065+0000 7f50a5c67640 10 monclient: monmap: 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:29.066385+0000 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:29.066385+0000 2024-08-19T09:59:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 1 Processor -- start 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 1 -- start start 2024-08-19T09:59:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient: init 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient: _add_conns ranks=[8,6,0,3,4,1,2,7,5] 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient(hunting): picked mon.noname-e con 0x7f50a00a6830 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient(hunting): picked mon.noname-d con 0x7f50a00694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient(hunting): picked mon.noname-a con 0x7f50a00a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a00a3c70 con 0x7f50a00a46a0 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a00a33d0 con 0x7f50a00694b0 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a00a36b0 con 0x7f50a00a6830 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f50a5c67640 10 monclient(hunting): authenticate will time out at 1225.142872s 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f509effd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f50a00694b0 0x7f50a00a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34990/0 (socket says 172.21.15.1:34990) 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f509effd640 1 -- 172.21.15.1:0/1356272745 learned_addr learned my addr 172.21.15.1:0/1356272745 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.066+0000 7f509f7fe640 1 --1- 172.21.15.1:0/1356272745 >> v1:172.21.15.1:6793/0 conn(0x7f50a00a6830 0x7f50a00a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:60892/0 (socket says 172.21.15.1:60892) 2024-08-19T09:59:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509effd640 1 -- 172.21.15.1:0/1356272745 >> v1:172.21.15.1:6792/0 conn(0x7f50a00694b0 legacy=0x7f50a00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509f7fe640 1 -- 172.21.15.1:0/1356272745 >> v1:172.21.15.1:6793/0 conn(0x7f50a00a6830 legacy=0x7f50a00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983644312}) 2024-08-19T09:59:29.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283625048 0 0) 0x7f50a00a33d0 con 0x7f50a00694b0 2024-08-19T09:59:29.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1958208803}) 2024-08-19T09:59:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): my global_id is 174246 2024-08-19T09:59:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5088003520 con 0x7f50a00694b0 2024-08-19T09:59:29.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895561644 0 0) 0x7f50a00a36b0 con 0x7f50a00a6830 2024-08-19T09:59:29.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1203212081}) 2024-08-19T09:59:29.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient(hunting): my global_id is 174194 2024-08-19T09:59:29.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50a00a33d0 con 0x7f50a00a6830 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2689970911}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3412444284}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3285120436}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3802883918}) 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966302475}) 2024-08-19T09:59:29.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4199870601}) 2024-08-19T09:59:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3532051129 0 0) 0x7f50a00a33d0 con 0x7f50a00a6830 2024-08-19T09:59:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 >> v1:172.21.15.1:6792/0 conn(0x7f50a00694b0 legacy=0x7f50a00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 >> v1:172.21.15.1:6789/0 conn(0x7f50a00a46a0 legacy=0x7f50a0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T09:59:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 monclient: found mon.noname-e 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a006dbb0 con 0x7f50a00a6830 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5090001ac0 con 0x7f50a00a6830 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f509e7fc640 10 monclient: dump: 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.088 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.089 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f50a5c67640 5 monclient: authenticate success, global_id 174194 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.069+0000 7f50a5c67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50900047e0 con 0x7f50a00a6830 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f509e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f509e7fc640 1 -- 172.21.15.1:0/1356272745 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5090004cd0 con 0x7f50a00a6830 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f50a5c67640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f50a5c67640 4 set_mon_vals no callback set 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f509e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f509e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.070+0000 7f509e7fc640 10 monclient: dump: 2024-08-19T09:59:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 10 monclient: shutdown 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 1 -- 172.21.15.1:0/1356272745 >> v1:172.21.15.1:6793/0 conn(0x7f50a00a6830 legacy=0x7f50a00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 1 -- 172.21.15.1:0/1356272745 shutdown_connections 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 1 -- 172.21.15.1:0/1356272745 >> 172.21.15.1:0/1356272745 conn(0x7f50a00a9020 msgr2=0x7f50a0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 1 -- 172.21.15.1:0/1356272745 shutdown_connections 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.071+0000 7f50a5c67640 1 -- 172.21.15.1:0/1356272745 wait complete. 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 10 monclient: monmap: 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 1 librados: starting msgr at 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 1 librados: starting objecter 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 1 Processor -- start 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 1 -- start start 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 1 librados: setting wanted keys 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 1 librados: calling monclient init 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.072+0000 7f50a5c67640 10 monclient: init 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient: _add_conns ranks=[0,8,4,2,7,6,1,5,3] 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient(hunting): picked mon.noname-a con 0x7f50a00694b0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient(hunting): picked mon.noname-e con 0x7f50a00a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient(hunting): picked mon.noname-c con 0x7f50a00a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a0139520 con 0x7f50a00694b0 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a013b5a0 con 0x7f50a00a6830 2024-08-19T09:59:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a013b880 con 0x7f50a00a46a0 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f50a5c67640 10 monclient(hunting): authenticate will time out at 1225.149173s 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f509effd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f50a00a46a0 0x7f50a0141e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:60906/0 (socket says 172.21.15.1:60906) 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.073+0000 7f509effd640 1 -- 172.21.15.1:0/4062164250 learned_addr learned my addr 172.21.15.1:0/4062164250 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f509effd640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6793/0 conn(0x7f50a00a46a0 legacy=0x7f50a0141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 client.?.objecter ms_handle_connect 0x7f50a00a46a0 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 client.?.objecter ms_handle_connect 0x7f50a00a6830 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f509ffff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f50a00a6830 legacy=0x7f50a01d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230622042 0 0) 0x7f50a013b880 con 0x7f50a00a46a0 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): my global_id is 174203 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f508c004eb0 con 0x7f50a00a46a0 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865811646 0 0) 0x7f50a013b5a0 con 0x7f50a00a6830 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): my global_id is 27544 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50a013b880 con 0x7f50a00a6830 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f509f7fe640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6789/0 conn(0x7f50a00694b0 legacy=0x7f50a01cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 client.?.objecter ms_handle_connect 0x7f50a00694b0 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2573641514 0 0) 0x7f50a013b880 con 0x7f50a00a6830 2024-08-19T09:59:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50a013b5a0 con 0x7f50a00a6830 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1212830652 0 0) 0x7f50a0139520 con 0x7f50a00694b0 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): my global_id is 174231 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50a013b880 con 0x7f50a00694b0 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3418792948 0 0) 0x7f508c004eb0 con 0x7f50a00a46a0 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50a0139520 con 0x7f50a00a46a0 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f50980047d0 con 0x7f50a00a6830 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): dump: 2024-08-19T09:59:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f50940047e0 con 0x7f50a00a46a0 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.075+0000 7f5087fff640 10 monclient(hunting): dump: 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1525402922 0 0) 0x7f50a013b5a0 con 0x7f50a00a6830 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6793/0 conn(0x7f50a00a46a0 legacy=0x7f50a0141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6789/0 conn(0x7f50a00694b0 legacy=0x7f50a01cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 monclient: found mon.c 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a0142a20 con 0x7f50a00a6830 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 5 monclient: authenticate success, global_id 27544 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 10 monclient: _renew_subs 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50a00a9f30 con 0x7f50a00a6830 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 10 client.27544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 10 monclient: _renew_subs 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f50a01426f0 con 0x7f50a00a6830 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f50a5c67640 1 librados: init done 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5098001500 con 0x7f50a00a6830 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f50980019f0 con 0x7f50a00a6830 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f509dffb640 4 set_mon_vals no callback set 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.076+0000 7f5087fff640 10 monclient: dump: 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.102 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f50a5c67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f50800065c0 con 0x7f5080002860 2024-08-19T09:59:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5098001c90 con 0x7f50a00a6830 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(364..364 src has 299..364) ==== 5064+0+0 (unknown 3328354567 0 0) 0x7f509804e310 con 0x7f50a00a6830 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f5087fff640 10 client.27544.objecter ms_dispatch 0x7f50a0000b90 osd_map(364..364 src has 299..364) 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f5087fff640 3 client.27544.objecter handle_osd_map got epochs [364,364] > 0 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f5087fff640 3 client.27544.objecter handle_osd_map decoding full epoch 364 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f5087fff640 20 client.27544.objecter dump_active .. 0 homeless 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.077+0000 7f509effd640 10 monclient: get_auth_request con 0x7f508c0415e0 auth_method 0 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 client.27544.objecter ms_handle_connect 0x7f5080002860 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 client.27544.objecter resend_mon_ops 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f509f7fe640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f5080002860 legacy=0x7f5080004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78029827 0 0) 0x7f50800065c0 con 0x7f5080002860 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient(hunting): my global_id is 27553 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50a013b5a0 con 0x7f5080002860 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3994227249 0 0) 0x7f50a013b5a0 con 0x7f5080002860 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50800065c0 con 0x7f5080002860 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5090001d10 con 0x7f5080002860 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 10 monclient: dump: 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3718333935 0 0) 0x7f50800065c0 con 0x7f5080002860 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.079+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f508000a160 con 0x7f5080002860 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.080+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5090004bb0 con 0x7f5080002860 2024-08-19T09:59:29.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.080+0000 7f5087fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.080+0000 7f5087fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:29.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.080+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f5080002860 legacy=0x7f5080004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.080+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f5080002860 legacy=0x7f5080004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.080+0000 7f5087fff640 10 client.27544.objecter ms_handle_connect 0x7f508c0415e0 2024-08-19T09:59:29.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4040836037}) 2024-08-19T09:59:29.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.115 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529310916}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=980557131}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528373926}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3185474205}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1330814036}) 2024-08-19T09:59:29.116 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.083+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.083+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2113003059}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.083+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.083+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736042529}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.083+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1000336572}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513182068}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852810281}) 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.117 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994788440}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1124244834}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=445907751}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2276262406}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.118 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1527317073}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030509612}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.084+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=119103751}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976413759}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516677319}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805710379}) 2024-08-19T09:59:29.119 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2461322578}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=959531066}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=775460761}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1489650469}) 2024-08-19T09:59:29.120 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.085+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.086+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540127644}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.086+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2006848608}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3107159439}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034999248}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2016344793}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3218713409}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3921914048}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=196174347}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2825315091}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288562924}) 2024-08-19T09:59:29.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.087+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3588199649}) 2024-08-19T09:59:29.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f50a5c67640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:29.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5080009e50 con 0x7f5080009110 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1767311394}) 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216479089}) 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2828138204}) 2024-08-19T09:59:29.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3805275523}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=730056886}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1340120571}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2732599724}) 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.088+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012798650}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.089+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2101468294}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408148625}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463988461}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472387515}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3507852578}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2720068851}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2906682903}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1279865933}) 2024-08-19T09:59:29.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:29.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 10 client.27544.objecter ms_handle_connect 0x7f5080009110 2024-08-19T09:59:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f509f7fe640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f5080009110 legacy=0x7f50800095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945424321}) 2024-08-19T09:59:29.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1878206085}) 2024-08-19T09:59:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 10 client.27544.objecter resend_mon_ops 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053041964 0 0) 0x7f5080009e50 con 0x7f5080009110 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 10 monclient(hunting): my global_id is 27562 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.090+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50800065c0 con 0x7f5080009110 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2671053832 0 0) 0x7f50800065c0 con 0x7f5080009110 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5080009e50 con 0x7f5080009110 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5090005370 con 0x7f5080009110 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 10 monclient: dump: 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 949684554 0 0) 0x7f5080009e50 con 0x7f5080009110 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f508000c3f0 con 0x7f5080009110 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3266079994) 0x7f50900056c0 con 0x7f5080009110 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:29.136 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 575362, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:29.137 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.138 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:29.140 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.141 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:29.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.143 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:29.144 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.145 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.148 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.150 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:29.151 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:29.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.153 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:29.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.156 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:29.157 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f5080009110 legacy=0x7f50800095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.091+0000 7f5087fff640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f5080009110 legacy=0x7f50800095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 10 librados: watch_flush enter 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 10 librados: watch_flush exit 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 20 client.27544.objecter shutdown clearing up homeless session... 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 10 client.27544.objecter successfully canceled tick 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.82:6812/2549104415 conn(0x7f508c0415e0 legacy=0x7f508c043aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 10 monclient: shutdown 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 >> v1:172.21.15.1:6791/0 conn(0x7f50a00a6830 legacy=0x7f50a01d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f509ffff640 1 -- 172.21.15.1:0/4062164250 reap_dead start 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f509ffff640 1 -- 172.21.15.1:0/4062164250 reap_dead start 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 shutdown_connections 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 >> 172.21.15.1:0/4062164250 conn(0x7f50a00a9020 msgr2=0x7f50a00a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 shutdown_connections 2024-08-19T09:59:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 -- 172.21.15.1:0/4062164250 wait complete. 2024-08-19T09:59:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.092+0000 7f50a5c67640 1 librados: shutdown 2024-08-19T09:59:29.161 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:29.161 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T09:59:29.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:59:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.259+0000 7f2df3867640 10 monclient: get_monmap_and_config 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.259+0000 7f2df3867640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.259+0000 7f2df3867640 10 monclient: monmap: 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:29.260733+0000 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:29.260733+0000 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.262 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.260+0000 7f2df3867640 1 Processor -- start 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.260+0000 7f2df3867640 1 -- start start 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.260+0000 7f2df3867640 10 monclient: init 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient: _add_conns ranks=[4,7,1,0,2,8,5,6,3] 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient(hunting): picked mon.noname-c con 0x7f2dec0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient(hunting): picked mon.noname-i con 0x7f2dec0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient(hunting): picked mon.noname-f con 0x7f2dec13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec0a3b40 con 0x7f2dec13f5c0 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec0a8130 con 0x7f2dec0a67c0 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec06ede0 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df3867640 10 monclient(hunting): authenticate will time out at 1225.337166s 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df1603640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 0x7f2dec0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46088/0 (socket says 172.21.15.1:46088) 2024-08-19T09:59:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.261+0000 7f2df1603640 1 -- 172.21.15.1:0/236526463 learned_addr learned my addr 172.21.15.1:0/236526463 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2df1603640 1 -- 172.21.15.1:0/236526463 >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 legacy=0x7f2dec0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2df0e02640 1 -- 172.21.15.1:0/236526463 >> v1:172.21.15.82:6792/0 conn(0x7f2dec0a6ba0 legacy=0x7f2dec0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2df1e04640 1 -- 172.21.15.1:0/236526463 >> v1:172.21.15.82:6789/0 conn(0x7f2dec13f5c0 legacy=0x7f2dec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555674564 0 0) 0x7f2dec0a8130 con 0x7f2dec0a67c0 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): my global_id is 27571 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd4003520 con 0x7f2dec0a67c0 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169413362 0 0) 0x7f2dec06ede0 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): my global_id is 144700 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dec0a8130 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165086230 0 0) 0x7f2dec0a3b40 con 0x7f2dec13f5c0 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 10 monclient(hunting): my global_id is 174196 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dec06ede0 con 0x7f2dec13f5c0 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3511869273 0 0) 0x7f2dd4003520 con 0x7f2dec0a67c0 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 >> v1:172.21.15.82:6792/0 conn(0x7f2dec0a6ba0 legacy=0x7f2dec0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.263+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 >> v1:172.21.15.82:6789/0 conn(0x7f2dec13f5c0 legacy=0x7f2dec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 1 monclient: found mon.noname-c 2024-08-19T09:59:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec0a92a0 con 0x7f2dec0a67c0 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ddc001e60 con 0x7f2dec0a67c0 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 5 monclient: authenticate success, global_id 27571 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: dump: 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.272 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df1603640 1 --1- 172.21.15.1:0/236526463 >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 0x7f2dec0a3f60 :-1 s=OPENED pgs=3519 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f2ddc005190 mon_map magic: 0 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ddc004c20 con 0x7f2dec0a67c0 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 4 set_mon_vals no callback set 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 10 monclient: shutdown 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 1 -- 172.21.15.1:0/236526463 >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 legacy=0x7f2dec0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 1 -- 172.21.15.1:0/236526463 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ddc005190 con 0x7f2dec0a67c0 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2debfff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 1 -- 172.21.15.1:0/236526463 shutdown_connections 2024-08-19T09:59:29.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 1 -- 172.21.15.1:0/236526463 >> 172.21.15.1:0/236526463 conn(0x7f2dec0ac4b0 msgr2=0x7f2dec0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 1 -- 172.21.15.1:0/236526463 shutdown_connections 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.264+0000 7f2df3867640 1 -- 172.21.15.1:0/236526463 wait complete. 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 10 monclient: monmap: 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.274 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 1 librados: starting msgr at 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 1 librados: starting objecter 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 1 Processor -- start 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 1 -- start start 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 1 librados: setting wanted keys 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 1 librados: calling monclient init 2024-08-19T09:59:29.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.265+0000 7f2df3867640 10 monclient: init 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient: _add_conns ranks=[4,8,5,7,6,0,2,1,3] 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient(hunting): picked mon.noname-c con 0x7f2dec0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient(hunting): picked mon.noname-e con 0x7f2dec0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient(hunting): picked mon.noname-h con 0x7f2dec13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec13a900 con 0x7f2dec0a67c0 2024-08-19T09:59:29.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec0a2860 con 0x7f2dec13f5c0 2024-08-19T09:59:29.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec139c50 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df3867640 10 monclient(hunting): authenticate will time out at 1225.342459s 2024-08-19T09:59:29.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df1603640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 0x7f2dec09fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46094/0 (socket says 172.21.15.1:46094) 2024-08-19T09:59:29.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.266+0000 7f2df1603640 1 -- 172.21.15.1:0/987622188 learned_addr learned my addr 172.21.15.1:0/987622188 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2df1603640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 legacy=0x7f2dec09fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 client.?.objecter ms_handle_connect 0x7f2dec0a67c0 2024-08-19T09:59:29.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 client.?.objecter ms_handle_connect 0x7f2dec0a6ba0 2024-08-19T09:59:29.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2df0e02640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6793/0 conn(0x7f2dec0a6ba0 legacy=0x7f2dec0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889533003 0 0) 0x7f2dec13a900 con 0x7f2dec0a67c0 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): my global_id is 27580 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd8004eb0 con 0x7f2dec0a67c0 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1128904736 0 0) 0x7f2dec139c50 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 monclient(hunting): my global_id is 174212 2024-08-19T09:59:29.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dec13a900 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 client.?.objecter ms_handle_connect 0x7f2dec13f5c0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2de9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.268+0000 7f2df1e04640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.82:6791/0 conn(0x7f2dec13f5c0 legacy=0x7f2dec1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3356365029 0 0) 0x7f2dd8004eb0 con 0x7f2dec0a67c0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dec139c50 con 0x7f2dec0a67c0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1333933176 0 0) 0x7f2dec13a900 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dd8004eb0 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ddc005110 con 0x7f2dec0a67c0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2de0003610 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): dump: 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.286 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302668474 0 0) 0x7f2dec0a2860 con 0x7f2dec13f5c0 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient(hunting): my global_id is 27203 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dec13a900 con 0x7f2dec13f5c0 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1351632758 0 0) 0x7f2dd8004eb0 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.82:6791/0 conn(0x7f2dec13f5c0 legacy=0x7f2dec1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6791/0 conn(0x7f2dec0a67c0 legacy=0x7f2dec09fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 monclient: found mon.e 2024-08-19T09:59:29.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec0a1600 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 5 monclient: authenticate success, global_id 174212 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 10 monclient: _renew_subs 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2dec13a620 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2de0003b90 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2de0004080 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2de9ffb640 10 monclient: dump: 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.289 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 10 client.174212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 10 monclient: _renew_subs 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2dec13f250 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2deb7fe640 4 set_mon_vals no callback set 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.269+0000 7f2df3867640 1 librados: init done 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2de0004220 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(364..364 src has 299..364) ==== 5064+0+0 (unknown 3328354567 0 0) 0x7f2de004e0a0 con 0x7f2dec0a6ba0 2024-08-19T09:59:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2de9ffb640 10 client.174212.objecter ms_dispatch 0x7f2dec000b90 osd_map(364..364 src has 299..364) 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2de9ffb640 3 client.174212.objecter handle_osd_map got epochs [364,364] > 0 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2de9ffb640 3 client.174212.objecter handle_osd_map decoding full epoch 364 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2df3867640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2dec0008d0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2de9ffb640 20 client.174212.objecter dump_active .. 0 homeless 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.270+0000 7f2df1603640 10 monclient: get_auth_request con 0x7f2dd8041580 auth_method 0 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2df1e04640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6792/0 conn(0x7f2dec0a0c00 legacy=0x7f2dec13fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 client.174212.objecter ms_handle_connect 0x7f2dec0a0c00 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 client.174212.objecter resend_mon_ops 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1359491862 0 0) 0x7f2dec0008d0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 monclient(hunting): my global_id is 174255 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd8004eb0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.272+0000 7f2de9ffb640 10 client.174212.objecter ms_handle_connect 0x7f2dd8041580 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 331760218 0 0) 0x7f2dd8004eb0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dec0008d0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2de4001bb0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 10 monclient: dump: 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2477135341 0 0) 0x7f2dec0008d0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2dec142bd0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2de40057c0 con 0x7f2dec0a0c00 2024-08-19T09:59:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6792/0 conn(0x7f2dec0a0c00 legacy=0x7f2dec13fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.273+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6792/0 conn(0x7f2dec0a0c00 legacy=0x7f2dec13fce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.279+0000 7f2df3867640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.279+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2dec1430d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.281+0000 7f2df1e04640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6792/0 conn(0x7f2dec02e1d0 legacy=0x7f2dec1403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.281+0000 7f2de9ffb640 10 client.174212.objecter ms_handle_connect 0x7f2dec02e1d0 2024-08-19T09:59:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.281+0000 7f2de9ffb640 10 client.174212.objecter resend_mon_ops 2024-08-19T09:59:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195387300 0 0) 0x7f2dec1430d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient(hunting): my global_id is 174264 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dec0008d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1551864399 0 0) 0x7f2dec0008d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dec1430d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2de4003d40 con 0x7f2dec02e1d0 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient: dump: 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2529782414 0 0) 0x7f2dec1430d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2dec02e5d0 con 0x7f2dec02e1d0 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3469027515) 0x7f2de4005a30 con 0x7f2dec02e1d0 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6792/0 conn(0x7f2dec02e1d0 legacy=0x7f2dec1403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2de9ffb640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6792/0 conn(0x7f2dec02e1d0 legacy=0x7f2dec1403f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19315, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:29.299 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.300 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.301 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:29.302 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.303 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.304 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.307 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.308 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.311 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.314 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:29.319 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:29.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2df3867640 10 librados: watch_flush enter 2024-08-19T09:59:29.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2df3867640 10 librados: watch_flush exit 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2df3867640 20 client.174212.objecter shutdown clearing up homeless session... 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.282+0000 7f2df3867640 10 client.174212.objecter successfully canceled tick 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2dd8041580 legacy=0x7f2dd8043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 10 monclient: shutdown 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 >> v1:172.21.15.1:6793/0 conn(0x7f2dec0a6ba0 legacy=0x7f2dec0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df1e04640 1 -- 172.21.15.1:0/987622188 reap_dead start 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 shutdown_connections 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 >> 172.21.15.1:0/987622188 conn(0x7f2dec0ac4b0 msgr2=0x7f2dec0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 shutdown_connections 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 -- 172.21.15.1:0/987622188 wait complete. 2024-08-19T09:59:29.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.283+0000 7f2df3867640 1 librados: shutdown 2024-08-19T09:59:29.323 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:29.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T09:59:29.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.423+0000 7fb917d16640 10 monclient: get_monmap_and_config 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.423+0000 7fb917d16640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.423+0000 7fb917d16640 10 monclient: monmap: 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:29.424327+0000 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:29.424327+0000 2024-08-19T09:59:29.424 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.425 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.423+0000 7fb917d16640 1 Processor -- start 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 1 -- start start 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient: init 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient: _add_conns ranks=[4,5,7,1,3,2,6,0,8] 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient(hunting): picked mon.noname-c con 0x7fb9100a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient(hunting): picked mon.noname-h con 0x7fb9100a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient(hunting): picked mon.noname-i con 0x7fb91013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9100a3b40 con 0x7fb9100a67c0 2024-08-19T09:59:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9100a8130 con 0x7fb9100a6ba0 2024-08-19T09:59:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb91006ede0 con 0x7fb91013f5c0 2024-08-19T09:59:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb917d16640 10 monclient(hunting): authenticate will time out at 1225.500726s 2024-08-19T09:59:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb915ab2640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb9100a67c0 0x7fb9100a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46102/0 (socket says 172.21.15.1:46102) 2024-08-19T09:59:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.424+0000 7fb915ab2640 1 -- 172.21.15.1:0/3823361275 learned_addr learned my addr 172.21.15.1:0/3823361275 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb915ab2640 1 -- 172.21.15.1:0/3823361275 >> v1:172.21.15.1:6791/0 conn(0x7fb9100a67c0 legacy=0x7fb9100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb9162b3640 1 -- 172.21.15.1:0/3823361275 >> v1:172.21.15.82:6792/0 conn(0x7fb91013f5c0 legacy=0x7fb9101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb9152b1640 1 -- 172.21.15.1:0/3823361275 >> v1:172.21.15.82:6791/0 conn(0x7fb9100a6ba0 legacy=0x7fb9100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047678295 0 0) 0x7fb9100a3b40 con 0x7fb9100a67c0 2024-08-19T09:59:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): my global_id is 27589 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8f8003520 con 0x7fb9100a67c0 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854910330 0 0) 0x7fb91006ede0 con 0x7fb91013f5c0 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): my global_id is 144709 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9100a3b40 con 0x7fb91013f5c0 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808132869 0 0) 0x7fb9100a8130 con 0x7fb9100a6ba0 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient(hunting): my global_id is 27212 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb91006ede0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3653436716 0 0) 0x7fb8f8003520 con 0x7fb9100a67c0 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 >> v1:172.21.15.82:6792/0 conn(0x7fb91013f5c0 legacy=0x7fb9101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 >> v1:172.21.15.82:6791/0 conn(0x7fb9100a6ba0 legacy=0x7fb9100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 monclient: found mon.noname-c 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9100a92a0 con 0x7fb9100a67c0 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb908001ae0 con 0x7fb9100a67c0 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb917d16640 5 monclient: authenticate success, global_id 27589 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb917d16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: dump: 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.435 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb917d16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9080048a0 con 0x7fb9100a67c0 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 1 -- 172.21.15.1:0/3823361275 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb908004e10 con 0x7fb9100a67c0 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb914ab0640 10 monclient: dump: 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb917d16640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.427+0000 7fb917d16640 4 set_mon_vals no callback set 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 10 monclient: shutdown 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 1 -- 172.21.15.1:0/3823361275 >> v1:172.21.15.1:6791/0 conn(0x7fb9100a67c0 legacy=0x7fb9100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 1 -- 172.21.15.1:0/3823361275 shutdown_connections 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 1 -- 172.21.15.1:0/3823361275 >> 172.21.15.1:0/3823361275 conn(0x7fb9100ac4b0 msgr2=0x7fb9100a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 1 -- 172.21.15.1:0/3823361275 shutdown_connections 2024-08-19T09:59:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 1 -- 172.21.15.1:0/3823361275 wait complete. 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.428+0000 7fb917d16640 10 monclient: monmap: 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 librados: starting msgr at 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 librados: starting objecter 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 Processor -- start 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 -- start start 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 librados: setting wanted keys 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 librados: calling monclient init 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient: init 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient: _add_conns ranks=[5,0,3,2,1,4,6,7,8] 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient(hunting): picked mon.noname-h con 0x7fb9100a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient(hunting): picked mon.noname-a con 0x7fb9100a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient(hunting): picked mon.noname-g con 0x7fb91013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9100a28c0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb91013ae80 con 0x7fb91013f5c0 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9101cf020 con 0x7fb9100a67c0 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb917d16640 10 monclient(hunting): authenticate will time out at 1225.505724s 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb9152b1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb9100a6ba0 0x7fb9101ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37222/0 (socket says 172.21.15.1:37222) 2024-08-19T09:59:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.429+0000 7fb9152b1640 1 -- 172.21.15.1:0/3763957941 learned_addr learned my addr 172.21.15.1:0/3763957941 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.431+0000 7fb9152b1640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6789/0 conn(0x7fb9100a6ba0 legacy=0x7fb9101ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.431+0000 7fb90e7fc640 10 client.?.objecter ms_handle_connect 0x7fb9100a6ba0 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.431+0000 7fb915ab2640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.82:6791/0 conn(0x7fb9100a67c0 legacy=0x7fb9101ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.431+0000 7fb90e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.431+0000 7fb90e7fc640 10 client.?.objecter ms_handle_connect 0x7fb9100a67c0 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.431+0000 7fb90e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 client.?.objecter ms_handle_connect 0x7fb91013f5c0 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb9162b3640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.82:6790/0 conn(0x7fb91013f5c0 legacy=0x7fb9101d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128911073 0 0) 0x7fb9100a28c0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 174240 2024-08-19T09:59:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8fc004eb0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338325809 0 0) 0x7fb9101cf020 con 0x7fb9100a67c0 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 27221 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9100a28c0 con 0x7fb9100a67c0 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3412849101 0 0) 0x7fb91013ae80 con 0x7fb91013f5c0 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 135228 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9101cf020 con 0x7fb91013f5c0 2024-08-19T09:59:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1685533996 0 0) 0x7fb8fc004eb0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb91013ae80 con 0x7fb9100a6ba0 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb904003990 con 0x7fb9100a6ba0 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 966994888 0 0) 0x7fb9100a28c0 con 0x7fb9100a67c0 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8fc004eb0 con 0x7fb9100a67c0 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb908004d90 con 0x7fb9100a67c0 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 261526420 0 0) 0x7fb9101cf020 con 0x7fb91013f5c0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9100a28c0 con 0x7fb91013f5c0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 10404570 0 0) 0x7fb91013ae80 con 0x7fb9100a6ba0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.82:6791/0 conn(0x7fb9100a67c0 legacy=0x7fb9101ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.82:6790/0 conn(0x7fb91013f5c0 legacy=0x7fb9101d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 monclient: found mon.a 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb910139990 con 0x7fb9100a6ba0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 5 monclient: authenticate success, global_id 174240 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 10 monclient: _renew_subs 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb91013bcf0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 10 client.174240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 10 monclient: _renew_subs 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb91013e560 con 0x7fb9100a6ba0 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb917d16640 1 librados: init done 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb904003f50 con 0x7fb9100a6ba0 2024-08-19T09:59:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb9040044c0 con 0x7fb9100a6ba0 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90ffff640 4 set_mon_vals no callback set 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.432+0000 7fb90e7fc640 10 monclient: dump: 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb904004660 con 0x7fb9100a6ba0 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb917d16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb8f00065c0 con 0x7fb8f0002860 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(364..364 src has 299..364) ==== 5064+0+0 (unknown 3328354567 0 0) 0x7fb90404e430 con 0x7fb9100a6ba0 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb90e7fc640 10 client.174240.objecter ms_dispatch 0x7fb910000b90 osd_map(364..364 src has 299..364) 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb90e7fc640 3 client.174240.objecter handle_osd_map got epochs [364,364] > 0 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb90e7fc640 3 client.174240.objecter handle_osd_map decoding full epoch 364 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb90e7fc640 20 client.174240.objecter dump_active .. 0 homeless 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.433+0000 7fb9162b3640 10 monclient: get_auth_request con 0x7fb8fc041880 auth_method 0 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb915ab2640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6793/0 conn(0x7fb8f0002860 legacy=0x7fb8f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 10 client.174240.objecter ms_handle_connect 0x7fb8f0002860 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 10 client.174240.objecter resend_mon_ops 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608379067 0 0) 0x7fb8f00065c0 con 0x7fb8f0002860 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.435+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 174221 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb91013ae80 con 0x7fb8f0002860 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2192732067 0 0) 0x7fb91013ae80 con 0x7fb8f0002860 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8f00065c0 con 0x7fb8f0002860 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 10 client.174240.objecter ms_handle_connect 0x7fb8fc041880 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb908001580 con 0x7fb8f0002860 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 10 monclient: dump: 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.459 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3766999784 0 0) 0x7fb8f00065c0 con 0x7fb8f0002860 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8f000a160 con 0x7fb8f0002860 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb9080048a0 con 0x7fb8f0002860 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6793/0 conn(0x7fb8f0002860 legacy=0x7fb8f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.436+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6793/0 conn(0x7fb8f0002860 legacy=0x7fb8f0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.442+0000 7fb917d16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.442+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb8f0009e50 con 0x7fb8f0009110 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb915ab2640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6793/0 conn(0x7fb8f0009110 legacy=0x7fb8f00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 client.174240.objecter ms_handle_connect 0x7fb8f0009110 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 client.174240.objecter resend_mon_ops 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823454541 0 0) 0x7fb8f0009e50 con 0x7fb8f0009110 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 174230 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8f00065c0 con 0x7fb8f0009110 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2041411879 0 0) 0x7fb8f00065c0 con 0x7fb8f0009110 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8f0009e50 con 0x7fb8f0009110 2024-08-19T09:59:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb9080059c0 con 0x7fb8f0009110 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 10 monclient: dump: 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.463 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.463 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.463 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.463 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1869446092 0 0) 0x7fb8f0009e50 con 0x7fb8f0009110 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.445+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb8f000c3a0 con 0x7fb8f0009110 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 3926159969) 0x7fb908005d50 con 0x7fb8f0009110 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb90e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb90e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6793/0 conn(0x7fb8f0009110 legacy=0x7fb8f00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb90e7fc640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6793/0 conn(0x7fb8f0009110 legacy=0x7fb8f00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19493, 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.477 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.480 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.482 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:29.483 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.485 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:29.487 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:29.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 10 librados: watch_flush enter 2024-08-19T09:59:29.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 10 librados: watch_flush exit 2024-08-19T09:59:29.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 20 client.174240.objecter shutdown clearing up homeless session... 2024-08-19T09:59:29.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 10 client.174240.objecter successfully canceled tick 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb8fc041880 legacy=0x7fb8fc043d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 10 monclient: shutdown 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 >> v1:172.21.15.1:6789/0 conn(0x7fb9100a6ba0 legacy=0x7fb9101ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb9162b3640 1 -- 172.21.15.1:0/3763957941 reap_dead start 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb9162b3640 1 -- 172.21.15.1:0/3763957941 reap_dead start 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 shutdown_connections 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 >> 172.21.15.1:0/3763957941 conn(0x7fb9100ac4b0 msgr2=0x7fb91013edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 shutdown_connections 2024-08-19T09:59:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 -- 172.21.15.1:0/3763957941 wait complete. 2024-08-19T09:59:29.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.446+0000 7fb917d16640 1 librados: shutdown 2024-08-19T09:59:29.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:29.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T09:59:29.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.590+0000 7fbdc139f640 10 monclient: get_monmap_and_config 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.590+0000 7fbdc139f640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.590+0000 7fbdc139f640 10 monclient: monmap: 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:29.591138+0000 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:29.591138+0000 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.592 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.590+0000 7fbdc139f640 1 Processor -- start 2024-08-19T09:59:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.590+0000 7fbdc139f640 1 -- start start 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.590+0000 7fbdc139f640 10 monclient: init 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient: _add_conns ranks=[0,6,8,3,1,7,5,4,2] 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient(hunting): picked mon.noname-a con 0x7fbdbc0a6830 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient(hunting): picked mon.noname-d con 0x7fbdbc0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient(hunting): picked mon.noname-e con 0x7fbdbc0a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc0a3c70 con 0x7fbdbc0a6830 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc0a33d0 con 0x7fbdbc0694b0 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc0a36b0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdc139f640 10 monclient(hunting): authenticate will time out at 1225.667368s 2024-08-19T09:59:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdbaffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbdbc0a6830 0x7fbdbc0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37238/0 (socket says 172.21.15.1:37238) 2024-08-19T09:59:29.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdbb7fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbdbc0a46a0 0x7fbdbc141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:60944/0 (socket says 172.21.15.1:60944) 2024-08-19T09:59:29.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdbaffd640 1 -- 172.21.15.1:0/1738360672 learned_addr learned my addr 172.21.15.1:0/1738360672 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.591+0000 7fbdba7fc640 1 --1- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6792/0 conn(0x7fbdbc0694b0 0x7fbdbc0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:35020/0 (socket says 172.21.15.1:35020) 2024-08-19T09:59:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdba7fc640 1 -- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6792/0 conn(0x7fbdbc0694b0 legacy=0x7fbdbc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6793/0 conn(0x7fbdbc0a46a0 legacy=0x7fbdbc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdbaffd640 1 -- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6789/0 conn(0x7fbdbc0a6830 legacy=0x7fbdbc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741423384 0 0) 0x7fbdbc0a33d0 con 0x7fbdbc0694b0 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): my global_id is 174273 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbda8003520 con 0x7fbdbc0694b0 2024-08-19T09:59:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052202028 0 0) 0x7fbdbc0a36b0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): my global_id is 174239 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc0a33d0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106786893 0 0) 0x7fbdbc0a3c70 con 0x7fbdbc0a6830 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 10 monclient(hunting): my global_id is 174249 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.593+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc0a36b0 con 0x7fbdbc0a6830 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2954325383 0 0) 0x7fbda8003520 con 0x7fbdbc0694b0 2024-08-19T09:59:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6793/0 conn(0x7fbdbc0a46a0 legacy=0x7fbdbc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdbb7fe640 1 --1- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6793/0 conn(0x7fbdbc0a46a0 0x7fbdbc141850 :-1 s=OPENED pgs=3478 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fbdb40019f0 mon_map magic: 0 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6789/0 conn(0x7fbdbc0a6830 legacy=0x7fbdbc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 monclient: found mon.noname-d 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc06dbb0 con 0x7fbdbc0694b0 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbda4001ae0 con 0x7fbdbc0694b0 2024-08-19T09:59:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: dump: 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 5 monclient: authenticate success, global_id 174273 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbda40047e0 con 0x7fbdbc0694b0 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 1 -- 172.21.15.1:0/1738360672 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbda4004d50 con 0x7fbdbc0694b0 2024-08-19T09:59:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdb9ffb640 10 monclient: dump: 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 4 set_mon_vals no callback set 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 10 monclient: shutdown 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 1 -- 172.21.15.1:0/1738360672 >> v1:172.21.15.1:6792/0 conn(0x7fbdbc0694b0 legacy=0x7fbdbc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 1 -- 172.21.15.1:0/1738360672 shutdown_connections 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 1 -- 172.21.15.1:0/1738360672 >> 172.21.15.1:0/1738360672 conn(0x7fbdbc0a9020 msgr2=0x7fbdbc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 1 -- 172.21.15.1:0/1738360672 shutdown_connections 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.594+0000 7fbdc139f640 1 -- 172.21.15.1:0/1738360672 wait complete. 2024-08-19T09:59:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 10 monclient: monmap: 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 1 librados: starting msgr at 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 1 librados: starting objecter 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 1 Processor -- start 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 1 -- start start 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 1 librados: setting wanted keys 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 1 librados: calling monclient init 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.595+0000 7fbdc139f640 10 monclient: init 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient: _add_conns ranks=[5,0,4,2,8,3,7,1,6] 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient(hunting): picked mon.noname-h con 0x7fbdbc0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient(hunting): picked mon.noname-a con 0x7fbdbc0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient(hunting): picked mon.noname-c con 0x7fbdbc0a6830 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc1cf280 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc0aa050 con 0x7fbdbc0a6830 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc068b70 con 0x7fbdbc0694b0 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdc139f640 10 monclient(hunting): authenticate will time out at 1225.672325s 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdbb7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fbdbc0a6830 0x7fbdbc1d66c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46116/0 (socket says 172.21.15.1:46116) 2024-08-19T09:59:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.596+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/3263594582 learned_addr learned my addr 172.21.15.1:0/3263594582 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdba7fc640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.1:6789/0 conn(0x7fbdbc0a46a0 legacy=0x7fbdbc1ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 client.?.objecter ms_handle_connect 0x7fbdbc0a46a0 2024-08-19T09:59:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 client.?.objecter ms_handle_connect 0x7fbdbc0a6830 2024-08-19T09:59:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.1:6791/0 conn(0x7fbdbc0a6830 legacy=0x7fbdbc1d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990795514 0 0) 0x7fbdbc1cf280 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): my global_id is 174258 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd9c004eb0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348130607 0 0) 0x7fbdbc0aa050 con 0x7fbdbc0a6830 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): my global_id is 27598 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc1cf280 con 0x7fbdbc0a6830 2024-08-19T09:59:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 client.?.objecter ms_handle_connect 0x7fbdbc0694b0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdbaffd640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6791/0 conn(0x7fbdbc0694b0 legacy=0x7fbdbc1ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869190667 0 0) 0x7fbdbc068b70 con 0x7fbdbc0694b0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): my global_id is 27230 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc0aa050 con 0x7fbdbc0694b0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3806167386 0 0) 0x7fbd9c004eb0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdbc068b70 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbda4001d30 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:29.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.598+0000 7fbdb37fe640 10 monclient(hunting): dump: 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 760704127 0 0) 0x7fbdbc1cf280 con 0x7fbdbc0a6830 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd9c004eb0 con 0x7fbdbc0a6830 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbdb40047e0 con 0x7fbdbc0a6830 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient(hunting): dump: 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.618 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2880494688 0 0) 0x7fbdbc068b70 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6791/0 conn(0x7fbdbc0694b0 legacy=0x7fbdbc1ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.1:6791/0 conn(0x7fbdbc0a6830 legacy=0x7fbdbc1d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 monclient: found mon.a 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc139ab0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 5 monclient: authenticate success, global_id 174258 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 10 monclient: _renew_subs 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbdbc13be10 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 10 client.174258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 10 monclient: _renew_subs 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbdbc13a8b0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdc139f640 1 librados: init done 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbda40047e0 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbda4001580 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb97fa640 4 set_mon_vals no callback set 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.599+0000 7fbdb37fe640 10 monclient: dump: 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdc139f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbda00065c0 con 0x7fbda0002860 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbda4011600 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(364..364 src has 299..364) ==== 5064+0+0 (unknown 3328354567 0 0) 0x7fbda404e000 con 0x7fbdbc0a46a0 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdb37fe640 10 client.174258.objecter ms_dispatch 0x7fbdbc000b90 osd_map(364..364 src has 299..364) 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdb37fe640 3 client.174258.objecter handle_osd_map got epochs [364,364] > 0 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdb37fe640 3 client.174258.objecter handle_osd_map decoding full epoch 364 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdb37fe640 20 client.174258.objecter dump_active .. 0 homeless 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.600+0000 7fbdbb7fe640 10 monclient: get_auth_request con 0x7fbd9c0417c0 auth_method 0 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdbaffd640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6789/0 conn(0x7fbda0002860 legacy=0x7fbda0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 client.174258.objecter ms_handle_connect 0x7fbda0002860 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 client.174258.objecter resend_mon_ops 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047861346 0 0) 0x7fbda00065c0 con 0x7fbda0002860 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient(hunting): my global_id is 174205 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc068b70 con 0x7fbda0002860 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 453018982 0 0) 0x7fbdbc068b70 con 0x7fbda0002860 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbda00065c0 con 0x7fbda0002860 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbdac001bb0 con 0x7fbda0002860 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:29.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.602+0000 7fbdb37fe640 10 monclient: dump: 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.625 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 10 client.174258.objecter ms_handle_connect 0x7fbd9c0417c0 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1987405948 0 0) 0x7fbda00065c0 con 0x7fbda0002860 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbda000a160 con 0x7fbda0002860 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbdac007480 con 0x7fbda0002860 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6789/0 conn(0x7fbda0002860 legacy=0x7fbda0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.603+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6789/0 conn(0x7fbda0002860 legacy=0x7fbda0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.609+0000 7fbdc139f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.609+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbda0009e50 con 0x7fbda0009110 2024-08-19T09:59:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdbaffd640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6789/0 conn(0x7fbda0009110 legacy=0x7fbda00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 client.174258.objecter ms_handle_connect 0x7fbda0009110 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 client.174258.objecter resend_mon_ops 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606643909 0 0) 0x7fbda0009e50 con 0x7fbda0009110 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient(hunting): my global_id is 174214 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbda00065c0 con 0x7fbda0009110 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1930324889 0 0) 0x7fbda00065c0 con 0x7fbda0009110 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbda0009e50 con 0x7fbda0009110 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbdac005670 con 0x7fbda0009110 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 10 monclient: dump: 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2859747304 0 0) 0x7fbda0009e50 con 0x7fbda0009110 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.612+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbda000c3a0 con 0x7fbda0009110 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 597818292) 0x7fbdac007630 con 0x7fbda0009110 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdb37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdb37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6789/0 conn(0x7fbda0009110 legacy=0x7fbda00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdb37fe640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6789/0 conn(0x7fbda0009110 legacy=0x7fbda00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.629 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19670, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:29.630 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.631 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:29.633 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:29.634 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:29.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:29.650 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 10 librados: watch_flush enter 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 10 librados: watch_flush exit 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 20 client.174258.objecter shutdown clearing up homeless session... 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 10 client.174258.objecter successfully canceled tick 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbd9c0417c0 legacy=0x7fbd9c043c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 10 monclient: shutdown 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 >> v1:172.21.15.1:6789/0 conn(0x7fbdbc0a46a0 legacy=0x7fbdbc1ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/3263594582 reap_dead start 2024-08-19T09:59:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/3263594582 reap_dead start 2024-08-19T09:59:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 shutdown_connections 2024-08-19T09:59:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 >> 172.21.15.1:0/3263594582 conn(0x7fbdbc0a9020 msgr2=0x7fbdbc0a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 shutdown_connections 2024-08-19T09:59:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 -- 172.21.15.1:0/3263594582 wait complete. 2024-08-19T09:59:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.613+0000 7fbdc139f640 1 librados: shutdown 2024-08-19T09:59:29.653 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:29.653 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T09:59:29.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T09:59:29.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.752+0000 7efd58112640 10 monclient: get_monmap_and_config 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.752+0000 7efd58112640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.752+0000 7efd58112640 10 monclient: monmap: 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:29.753596+0000 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:29.753596+0000 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.754 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 1 Processor -- start 2024-08-19T09:59:29.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 1 -- start start 2024-08-19T09:59:29.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 10 monclient: init 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 10 monclient: _add_conns ranks=[3,7,1,6,4,0,8,5,2] 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 10 monclient(hunting): picked mon.noname-g con 0x7efd500a5710 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 10 monclient(hunting): picked mon.noname-i con 0x7efd500a6200 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 10 monclient(hunting): picked mon.noname-f con 0x7efd5013dc10 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.753+0000 7efd58112640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd500a2c50 con 0x7efd5013dc10 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd58112640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd500a2f30 con 0x7efd500a5710 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd58112640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd500a1900 con 0x7efd500a6200 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd58112640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd58112640 10 monclient(hunting): authenticate will time out at 1225.830047s 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd566af640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7efd5013dc10 0x7efd50140000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:46770/0 (socket says 172.21.15.1:46770) 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd566af640 1 -- 172.21.15.1:0/277444931 learned_addr learned my addr 172.21.15.1:0/277444931 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.754+0000 7efd55eae640 1 --1- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6790/0 conn(0x7efd500a5710 0x7efd500a5af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:35030/0 (socket says 172.21.15.1:35030) 2024-08-19T09:59:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd55eae640 1 -- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6790/0 conn(0x7efd500a5710 legacy=0x7efd500a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd566af640 1 -- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6789/0 conn(0x7efd5013dc10 legacy=0x7efd50140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd556ad640 1 -- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6792/0 conn(0x7efd500a6200 legacy=0x7efd5013c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203086835 0 0) 0x7efd500a2f30 con 0x7efd500a5710 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): my global_id is 135237 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd3c003520 con 0x7efd500a5710 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998988003 0 0) 0x7efd500a2c50 con 0x7efd5013dc10 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): my global_id is 174223 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd500a2f30 con 0x7efd5013dc10 2024-08-19T09:59:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211812210 0 0) 0x7efd500a1900 con 0x7efd500a6200 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient(hunting): my global_id is 144718 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd500a2c50 con 0x7efd500a6200 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2884071067 0 0) 0x7efd3c003520 con 0x7efd500a5710 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6792/0 conn(0x7efd500a6200 legacy=0x7efd5013c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6789/0 conn(0x7efd5013dc10 legacy=0x7efd50140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 monclient: found mon.noname-g 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd500a0e60 con 0x7efd500a5710 2024-08-19T09:59:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd4c001ae0 con 0x7efd500a5710 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.756+0000 7efd54eac640 10 monclient: dump: 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 5 monclient: authenticate success, global_id 135237 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd4c0047e0 con 0x7efd500a5710 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd54eac640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd54eac640 1 -- 172.21.15.1:0/277444931 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd4c004d50 con 0x7efd500a5710 2024-08-19T09:59:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd54eac640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd54eac640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd54eac640 10 monclient: dump: 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.770 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 4 set_mon_vals no callback set 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 10 monclient: shutdown 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 1 -- 172.21.15.1:0/277444931 >> v1:172.21.15.82:6790/0 conn(0x7efd500a5710 legacy=0x7efd500a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 1 -- 172.21.15.1:0/277444931 shutdown_connections 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 1 -- 172.21.15.1:0/277444931 >> 172.21.15.1:0/277444931 conn(0x7efd500a6800 msgr2=0x7efd500a6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 1 -- 172.21.15.1:0/277444931 shutdown_connections 2024-08-19T09:59:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.757+0000 7efd58112640 1 -- 172.21.15.1:0/277444931 wait complete. 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 10 monclient: monmap: 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 1 librados: starting msgr at 2024-08-19T09:59:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 1 librados: starting objecter 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 1 Processor -- start 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 1 -- start start 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 1 librados: setting wanted keys 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 1 librados: calling monclient init 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.758+0000 7efd58112640 10 monclient: init 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient: _add_conns ranks=[5,2,3,1,8,0,4,6,7] 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient(hunting): picked mon.noname-h con 0x7efd500a5710 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient(hunting): picked mon.noname-b con 0x7efd500a6200 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient(hunting): picked mon.noname-g con 0x7efd5013dc10 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd501259d0 con 0x7efd500a6200 2024-08-19T09:59:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd50125cb0 con 0x7efd5013dc10 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd50124e40 con 0x7efd500a5710 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd58112640 10 monclient(hunting): authenticate will time out at 1225.835217s 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd556ad640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7efd500a6200 0x7efd501d0fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33986/0 (socket says 172.21.15.1:33986) 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.759+0000 7efd556ad640 1 -- 172.21.15.1:0/767393481 learned_addr learned my addr 172.21.15.1:0/767393481 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd556ad640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.1:6790/0 conn(0x7efd500a6200 legacy=0x7efd501d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 client.?.objecter ms_handle_connect 0x7efd500a6200 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 client.?.objecter ms_handle_connect 0x7efd500a5710 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 client.?.objecter ms_handle_connect 0x7efd5013dc10 2024-08-19T09:59:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd55eae640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6791/0 conn(0x7efd500a5710 legacy=0x7efd501cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd566af640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd5013dc10 legacy=0x7efd501d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420305703 0 0) 0x7efd501259d0 con 0x7efd500a6200 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): my global_id is 164432 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd34004ee0 con 0x7efd500a6200 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884677840 0 0) 0x7efd50125cb0 con 0x7efd5013dc10 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): my global_id is 135246 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd501259d0 con 0x7efd5013dc10 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508858398 0 0) 0x7efd50124e40 con 0x7efd500a5710 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): my global_id is 27239 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd50125cb0 con 0x7efd500a5710 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 381064107 0 0) 0x7efd34004ee0 con 0x7efd500a6200 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd50124e40 con 0x7efd500a6200 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd40003950 con 0x7efd500a6200 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T09:59:29.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): dump: 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3368912073 0 0) 0x7efd501259d0 con 0x7efd5013dc10 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd34004ee0 con 0x7efd5013dc10 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd48001f60 con 0x7efd5013dc10 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.761+0000 7efd467fc640 10 monclient(hunting): dump: 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 928829806 0 0) 0x7efd50124e40 con 0x7efd500a6200 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6791/0 conn(0x7efd500a5710 legacy=0x7efd501cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd5013dc10 legacy=0x7efd501d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 monclient: found mon.b 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd501256f0 con 0x7efd500a6200 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 5 monclient: authenticate success, global_id 164432 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 10 monclient: _renew_subs 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:59:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd5013a9a0 con 0x7efd500a6200 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 10 client.164432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 10 monclient: _renew_subs 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efd501244b0 con 0x7efd500a6200 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 1 librados: init done 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd40001ce0 con 0x7efd500a6200 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd40004160 con 0x7efd500a6200 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd47fff640 4 set_mon_vals no callback set 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 10 monclient: dump: 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7efd40011a80 con 0x7efd500a6200 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.762+0000 7efd58112640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efd380065c0 con 0x7efd38002860 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(364..364 src has 299..364) ==== 5064+0+0 (unknown 3328354567 0 0) 0x7efd4004e1a0 con 0x7efd500a6200 2024-08-19T09:59:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd467fc640 10 client.164432.objecter ms_dispatch 0x7efd50000b90 osd_map(364..364 src has 299..364) 2024-08-19T09:59:29.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd467fc640 3 client.164432.objecter handle_osd_map got epochs [364,364] > 0 2024-08-19T09:59:29.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd467fc640 3 client.164432.objecter handle_osd_map decoding full epoch 364 2024-08-19T09:59:29.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd467fc640 20 client.164432.objecter dump_active .. 0 homeless 2024-08-19T09:59:29.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.763+0000 7efd566af640 10 monclient: get_auth_request con 0x7efd340414d0 auth_method 0 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 client.164432.objecter ms_handle_connect 0x7efd38002860 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd55eae640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd38002860 legacy=0x7efd38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 client.164432.objecter resend_mon_ops 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 995769861 0 0) 0x7efd380065c0 con 0x7efd38002860 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient(hunting): my global_id is 135255 2024-08-19T09:59:29.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd50124e40 con 0x7efd38002860 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 client.164432.objecter ms_handle_connect 0x7efd340414d0 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2142866057 0 0) 0x7efd50124e40 con 0x7efd38002860 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd380065c0 con 0x7efd38002860 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd4c001d30 con 0x7efd38002860 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.765+0000 7efd467fc640 10 monclient: dump: 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.789 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1286784599 0 0) 0x7efd380065c0 con 0x7efd38002860 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd3800a160 con 0x7efd38002860 2024-08-19T09:59:29.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efd4c0053d0 con 0x7efd38002860 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd38002860 legacy=0x7efd38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.766+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd38002860 legacy=0x7efd38004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.772+0000 7efd58112640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.772+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efd38009e70 con 0x7efd38009130 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd55eae640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd38009130 legacy=0x7efd380095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 client.164432.objecter ms_handle_connect 0x7efd38009130 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 client.164432.objecter resend_mon_ops 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460318048 0 0) 0x7efd38009e70 con 0x7efd38009130 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient(hunting): my global_id is 135264 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd380065c0 con 0x7efd38009130 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2046913799 0 0) 0x7efd380065c0 con 0x7efd38009130 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd38009e70 con 0x7efd38009130 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7efd4c005b70 con 0x7efd38009130 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 10 monclient: dump: 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.792 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 144075511 0 0) 0x7efd38009e70 con 0x7efd38009130 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.775+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd3800c410 con 0x7efd38009130 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7851 (unknown 0 0 1370135061) 0x7efd4c007fb0 con 0x7efd38009130 2024-08-19T09:59:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd467fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd38009130 legacy=0x7efd380095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd467fc640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6790/0 conn(0x7efd38009130 legacy=0x7efd380095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 426836, 2024-08-19T09:59:29.794 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:29.795 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.796 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.797 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:29.798 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.799 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:29.803 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.805 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.807 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.808 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.809 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.812 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:29.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.814 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.815 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.815 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.815 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:29.815 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 10 librados: watch_flush enter 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 10 librados: watch_flush exit 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 20 client.164432.objecter shutdown clearing up homeless session... 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 10 client.164432.objecter successfully canceled tick 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.82:6812/2549104415 conn(0x7efd340414d0 legacy=0x7efd34043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 10 monclient: shutdown 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 >> v1:172.21.15.1:6790/0 conn(0x7efd500a6200 legacy=0x7efd501d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd566af640 1 -- 172.21.15.1:0/767393481 reap_dead start 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd566af640 1 -- 172.21.15.1:0/767393481 reap_dead start 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 shutdown_connections 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 >> 172.21.15.1:0/767393481 conn(0x7efd500a6800 msgr2=0x7efd500a9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 shutdown_connections 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 -- 172.21.15.1:0/767393481 wait complete. 2024-08-19T09:59:29.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.776+0000 7efd58112640 1 librados: shutdown 2024-08-19T09:59:29.818 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:29.818 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T09:59:29.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:59:29.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.918+0000 7f6d21514640 10 monclient: get_monmap_and_config 2024-08-19T09:59:29.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.918+0000 7f6d21514640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.918+0000 7f6d21514640 10 monclient: monmap: 2024-08-19T09:59:29.921 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:29.919552+0000 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:29.919552+0000 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.922 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 1 Processor -- start 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 1 -- start start 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient: init 2024-08-19T09:59:29.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient: _add_conns ranks=[4,1,2,3,0,8,6,7,5] 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient(hunting): picked mon.noname-f con 0x7f6d1c0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient(hunting): picked mon.noname-b con 0x7f6d1c13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1c0a3b40 con 0x7f6d1c0a6ba0 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1c0a8130 con 0x7f6d1c13f5c0 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1c06ede0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d21514640 10 monclient(hunting): authenticate will time out at 1225.995899s 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d1affd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f6d1c0a67c0 0x7f6d1c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46126/0 (socket says 172.21.15.1:46126) 2024-08-19T09:59:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.919+0000 7f6d1affd640 1 -- 172.21.15.1:0/3607999037 learned_addr learned my addr 172.21.15.1:0/3607999037 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1b7fe640 1 -- 172.21.15.1:0/3607999037 >> v1:172.21.15.1:6790/0 conn(0x7f6d1c13f5c0 legacy=0x7f6d1c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1affd640 1 -- 172.21.15.1:0/3607999037 >> v1:172.21.15.1:6791/0 conn(0x7f6d1c0a67c0 legacy=0x7f6d1c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 402437406 0 0) 0x7f6d1c06ede0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): my global_id is 27607 2024-08-19T09:59:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d08003520 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2456193600 0 0) 0x7f6d1c0a8130 con 0x7f6d1c13f5c0 2024-08-19T09:59:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient(hunting): my global_id is 164441 2024-08-19T09:59:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d1c06ede0 con 0x7f6d1c13f5c0 2024-08-19T09:59:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d13fff640 1 -- 172.21.15.1:0/3607999037 >> v1:172.21.15.82:6789/0 conn(0x7f6d1c0a6ba0 legacy=0x7f6d1c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 926788362 0 0) 0x7f6d08003520 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 >> v1:172.21.15.1:6790/0 conn(0x7f6d1c13f5c0 legacy=0x7f6d1c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 >> v1:172.21.15.82:6789/0 conn(0x7f6d1c0a6ba0 legacy=0x7f6d1c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 monclient: found mon.noname-c 2024-08-19T09:59:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1c0a92a0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6d0c001a00 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d21514640 5 monclient: authenticate success, global_id 27607 2024-08-19T09:59:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d21514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: dump: 2024-08-19T09:59:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.952 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.952 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d21514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:29.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d0c001cd0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 1 -- 172.21.15.1:0/3607999037 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6d0c004be0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d21514640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d21514640 4 set_mon_vals no callback set 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.922+0000 7f6d1a7fc640 10 monclient: dump: 2024-08-19T09:59:29.953 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 10 monclient: shutdown 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 1 -- 172.21.15.1:0/3607999037 >> v1:172.21.15.1:6791/0 conn(0x7f6d1c0a67c0 legacy=0x7f6d1c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 1 -- 172.21.15.1:0/3607999037 shutdown_connections 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 1 -- 172.21.15.1:0/3607999037 >> 172.21.15.1:0/3607999037 conn(0x7f6d1c0ac4b0 msgr2=0x7f6d1c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 1 -- 172.21.15.1:0/3607999037 shutdown_connections 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 1 -- 172.21.15.1:0/3607999037 wait complete. 2024-08-19T09:59:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 10 monclient: build_initial_monmap 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.923+0000 7f6d21514640 10 monclient: monmap: 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 librados: starting msgr at 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 librados: starting objecter 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 Processor -- start 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 -- start start 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 librados: setting wanted keys 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 librados: calling monclient init 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient: init 2024-08-19T09:59:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient: _add_conns ranks=[0,7,2,8,1,3,4,6,5] 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient(hunting): picked mon.noname-a con 0x7f6d1c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient(hunting): picked mon.noname-i con 0x7f6d1c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient(hunting): picked mon.noname-b con 0x7f6d1c13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1c0a16b0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1c0a9dc0 con 0x7f6d1c13f5c0 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1c142c60 con 0x7f6d1c0a6ba0 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d21514640 10 monclient(hunting): authenticate will time out at 1226.000636s 2024-08-19T09:59:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d1b7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f6d1c13f5c0 0x7f6d1c1d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34000/0 (socket says 172.21.15.1:34000) 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d1b7fe640 1 -- 172.21.15.1:0/1116897803 learned_addr learned my addr 172.21.15.1:0/1116897803 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.924+0000 7f6d1affd640 1 --1- 172.21.15.1:0/1116897803 >> v1:172.21.15.1:6789/0 conn(0x7f6d1c0a67c0 0x7f6d1c09ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37248/0 (socket says 172.21.15.1:37248) 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.926+0000 7f6d1b7fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.1:6790/0 conn(0x7f6d1c13f5c0 legacy=0x7f6d1c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.926+0000 7f6d1affd640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.1:6789/0 conn(0x7f6d1c0a67c0 legacy=0x7f6d1c09ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.926+0000 7f6d137fe640 10 client.?.objecter ms_handle_connect 0x7f6d1c13f5c0 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.926+0000 7f6d137fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.926+0000 7f6d137fe640 10 client.?.objecter ms_handle_connect 0x7f6d1c0a67c0 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.926+0000 7f6d137fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d1b7fe640 1 --1- 172.21.15.1:0/1116897803 >> v1:172.21.15.1:6790/0 conn(0x7f6d1c13f5c0 0x7f6d1c1d6250 :-1 s=OPENED pgs=3318 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f6d1c0a9dc0 auth_reply(proto 2 0 (0) Success) 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590930098 0 0) 0x7f6d1c0a16b0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): my global_id is 174267 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cfc004eb0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 client.?.objecter ms_handle_connect 0x7f6d1c0a6ba0 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d13fff640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6792/0 conn(0x7f6d1c0a6ba0 legacy=0x7f6d1c0a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2252010843 0 0) 0x7f6cfc004eb0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d1c0a16b0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6d0c004b20 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:29.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): dump: 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.961 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482335971 0 0) 0x7f6d1c142c60 con 0x7f6d1c0a6ba0 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient(hunting): my global_id is 144727 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cfc004eb0 con 0x7f6d1c0a6ba0 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2888309216 0 0) 0x7f6d1c0a16b0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6792/0 conn(0x7f6d1c0a6ba0 legacy=0x7f6d1c0a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.1:6790/0 conn(0x7f6d1c13f5c0 legacy=0x7f6d1c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 monclient: found mon.a 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1c13a810 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 5 monclient: authenticate success, global_id 174267 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076854598 0 0) 0x7f6d1c0a9dc0 con 0x7f6d1c13f5c0 2024-08-19T09:59:29.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 10 monclient: _renew_subs 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1c139ab0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 10 client.174267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 10 monclient: _renew_subs 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d1c0a0d70 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d21514640 1 librados: init done 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d0c0019a0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6d0c005550 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d19ffb640 4 set_mon_vals no callback set 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.927+0000 7f6d137fe640 10 monclient: dump: 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.965 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d21514640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6d000065c0 con 0x7f6d00002860 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6d0c0119b0 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(364..364 src has 299..364) ==== 5064+0+0 (unknown 3328354567 0 0) 0x7f6d0c04df90 con 0x7f6d1c0a67c0 2024-08-19T09:59:29.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d137fe640 10 client.174267.objecter ms_dispatch 0x7f6d1c000b90 osd_map(364..364 src has 299..364) 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d137fe640 3 client.174267.objecter handle_osd_map got epochs [364,364] > 0 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d137fe640 3 client.174267.objecter handle_osd_map decoding full epoch 364 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d137fe640 20 client.174267.objecter dump_active .. 0 homeless 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.928+0000 7f6d1b7fe640 10 monclient: get_auth_request con 0x7f6cfc040e60 auth_method 0 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.930+0000 7f6d137fe640 10 client.174267.objecter ms_handle_connect 0x7f6d00002860 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.930+0000 7f6d137fe640 10 client.174267.objecter resend_mon_ops 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.930+0000 7f6d13fff640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6791/0 conn(0x7f6d00002860 legacy=0x7f6d00004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271850422 0 0) 0x7f6d000065c0 con 0x7f6d00002860 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient(hunting): my global_id is 27248 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d1c0a9dc0 con 0x7f6d00002860 2024-08-19T09:59:29.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 client.174267.objecter ms_handle_connect 0x7f6cfc040e60 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3565174647 0 0) 0x7f6d1c0a9dc0 con 0x7f6d00002860 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d000065c0 con 0x7f6d00002860 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6d04001bf0 con 0x7f6d00002860 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient: dump: 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.968 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3348111401 0 0) 0x7f6d000065c0 con 0x7f6d00002860 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d0000a160 con 0x7f6d00002860 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6d040057c0 con 0x7f6d00002860 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:29.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6791/0 conn(0x7f6d00002860 legacy=0x7f6d00004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.931+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6791/0 conn(0x7f6d00002860 legacy=0x7f6d00004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.938+0000 7f6d21514640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.938+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6d00009e50 con 0x7f6d00009110 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 client.174267.objecter ms_handle_connect 0x7f6d00009110 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d13fff640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6791/0 conn(0x7f6d00009110 legacy=0x7f6d000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 client.174267.objecter resend_mon_ops 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1379314721 0 0) 0x7f6d00009e50 con 0x7f6d00009110 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient(hunting): my global_id is 27257 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d000065c0 con 0x7f6d00009110 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3188761129 0 0) 0x7f6d000065c0 con 0x7f6d00009110 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d00009e50 con 0x7f6d00009110 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6d04003d00 con 0x7f6d00009110 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.940+0000 7f6d137fe640 10 monclient: dump: 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:29.972 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 625093596 0 0) 0x7f6d00009e50 con 0x7f6d00009110 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d0000c3a0 con 0x7f6d00009110 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 608441956) 0x7f6d04005a30 con 0x7f6d00009110 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6791/0 conn(0x7f6d00009110 legacy=0x7f6d000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d137fe640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6791/0 conn(0x7f6d00009110 legacy=0x7f6d000095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:29.973 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 576207, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:29.974 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.975 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.976 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:29.977 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:29.978 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.979 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.980 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:29.986 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.987 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:29.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.991 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:29.993 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:29.994 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:59:29.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:29.994 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:29.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:29.994 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:29.994 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:29.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 10 librados: watch_flush enter 2024-08-19T09:59:29.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 10 librados: watch_flush exit 2024-08-19T09:59:29.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 20 client.174267.objecter shutdown clearing up homeless session... 2024-08-19T09:59:29.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 10 client.174267.objecter successfully canceled tick 2024-08-19T09:59:29.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6cfc040e60 legacy=0x7f6cfc043320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 10 monclient: shutdown 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 >> v1:172.21.15.1:6789/0 conn(0x7f6d1c0a67c0 legacy=0x7f6d1c09ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d1b7fe640 1 -- 172.21.15.1:0/1116897803 reap_dead start 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d1b7fe640 1 -- 172.21.15.1:0/1116897803 reap_dead start 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 shutdown_connections 2024-08-19T09:59:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 >> 172.21.15.1:0/1116897803 conn(0x7f6d1c0ac4b0 msgr2=0x7f6d1c0ab2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 shutdown_connections 2024-08-19T09:59:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.941+0000 7f6d21514640 1 -- 172.21.15.1:0/1116897803 wait complete. 2024-08-19T09:59:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:29.942+0000 7f6d21514640 1 librados: shutdown 2024-08-19T09:59:30.000 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:30.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T09:59:30.000 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:59:30.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525776066}) 2024-08-19T09:59:30.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2679625450}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=615549560}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=796822748}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=933630930}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3649871729}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3081555928}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033968026}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.976+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510603905}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426093508}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564294418}) 2024-08-19T09:59:30.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1322376829}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1856272266}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3289565800}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1293943739}) 2024-08-19T09:59:30.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=177843205}) 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114840960}) 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4106566818}) 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021288582}) 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866001324}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1463741063}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2179813339}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3348130338}) 2024-08-19T09:59:30.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1034603002}) 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3674682599}) 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954409805}) 2024-08-19T09:59:30.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:29.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3110339266}) 2024-08-19T09:59:30.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=896184246}) 2024-08-19T09:59:30.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=238023401}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2053338949}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2606785021}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2478228051}) 2024-08-19T09:59:30.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1632234323}) 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4002469877}) 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916194970}) 2024-08-19T09:59:30.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.013+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4129708976}) 2024-08-19T09:59:30.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248932303}) 2024-08-19T09:59:30.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1566719667}) 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=291468606}) 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3488895878}) 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2168013029}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2848919573}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=516119284}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777650171}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.014+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=56352909}) 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347560502}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876130693}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1333388866}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=962984715}) 2024-08-19T09:59:30.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1251758713}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3938880950}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3574621435}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254907317}) 2024-08-19T09:59:30.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.015+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3965747720}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310606955}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163923018}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1652328051}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=113528123}) 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=793007223}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=727149923}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1630461843}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169015584}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.016+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3320330261}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780489724}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=137450847}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4250874620}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3934087424}) 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3759357878}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2580170291}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3102272462}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621586907}) 2024-08-19T09:59:30.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.018+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2004167053}) 2024-08-19T09:59:30.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944395253}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3784248154}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851604113}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2957600863}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2070758794}) 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=89256284}) 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270570609}) 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439116557}) 2024-08-19T09:59:30.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.035+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3227686452}) 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=570566152}) 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601387531}) 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.066 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317923087}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=512612219}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3432738453}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4007006191}) 2024-08-19T09:59:30.067 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1717138328}) 2024-08-19T09:59:30.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:30.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594046266}) 2024-08-19T09:59:30.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:30.064+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2252880013}) 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.121+0000 7f5c35020640 10 monclient: get_monmap_and_config 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.121+0000 7f5c35020640 10 monclient: build_initial_monmap 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.122+0000 7f5c35020640 10 monclient: monmap: 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:30.122913+0000 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:30.122913+0000 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:30.123 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:30.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.122+0000 7f5c35020640 1 Processor -- start 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.122+0000 7f5c35020640 1 -- start start 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.122+0000 7f5c35020640 10 monclient: init 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient: _add_conns ranks=[7,5,0,1,4,2,3,6,8] 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient(hunting): picked mon.noname-i con 0x7f5c300a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient(hunting): picked mon.noname-h con 0x7f5c300a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c300a3b40 con 0x7f5c3013f5c0 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c300a8130 con 0x7f5c300a6ba0 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3006ede0 con 0x7f5c300a67c0 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c35020640 10 monclient(hunting): authenticate will time out at 1226.199162s 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c2f59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c3013f5c0 0x7f5c301419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37250/0 (socket says 172.21.15.1:37250) 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.123+0000 7f5c2f59e640 1 -- 172.21.15.1:0/1515036002 learned_addr learned my addr 172.21.15.1:0/1515036002 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2f59e640 1 -- 172.21.15.1:0/1515036002 >> v1:172.21.15.1:6789/0 conn(0x7f5c3013f5c0 legacy=0x7f5c301419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3081488248 0 0) 0x7f5c300a3b40 con 0x7f5c3013f5c0 2024-08-19T09:59:30.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): my global_id is 174276 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c18003520 con 0x7f5c3013f5c0 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2ed9d640 1 -- 172.21.15.1:0/1515036002 >> v1:172.21.15.82:6792/0 conn(0x7f5c300a67c0 legacy=0x7f5c300a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2e59c640 1 -- 172.21.15.1:0/1515036002 >> v1:172.21.15.82:6791/0 conn(0x7f5c300a6ba0 legacy=0x7f5c300a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355611255 0 0) 0x7f5c3006ede0 con 0x7f5c300a67c0 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient(hunting): my global_id is 144736 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c300a3b40 con 0x7f5c300a67c0 2024-08-19T09:59:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1049965619 0 0) 0x7f5c18003520 con 0x7f5c3013f5c0 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 >> v1:172.21.15.82:6792/0 conn(0x7f5c300a67c0 legacy=0x7f5c300a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 >> v1:172.21.15.82:6791/0 conn(0x7f5c300a6ba0 legacy=0x7f5c300a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: _finish_hunting 0 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 monclient: found mon.noname-a 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c300a92a0 con 0x7f5c3013f5c0 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: _finish_auth 0 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: _check_auth_tickets 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c28001e60 con 0x7f5c3013f5c0 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c35020640 5 monclient: authenticate success, global_id 174276 2024-08-19T09:59:30.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c35020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c2dd9b640 10 monclient: dump: 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.139 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.125+0000 7f5c35020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c28004c20 con 0x7f5c3013f5c0 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c2dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c2dd9b640 1 -- 172.21.15.1:0/1515036002 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c28005190 con 0x7f5c3013f5c0 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c2dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c2dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c2dd9b640 10 monclient: dump: 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 4 set_mon_vals no callback set 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 10 monclient: shutdown 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 1 -- 172.21.15.1:0/1515036002 >> v1:172.21.15.1:6789/0 conn(0x7f5c3013f5c0 legacy=0x7f5c301419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 1 -- 172.21.15.1:0/1515036002 shutdown_connections 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 1 -- 172.21.15.1:0/1515036002 >> 172.21.15.1:0/1515036002 conn(0x7f5c300ac4b0 msgr2=0x7f5c300a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 1 -- 172.21.15.1:0/1515036002 shutdown_connections 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.126+0000 7f5c35020640 1 -- 172.21.15.1:0/1515036002 wait complete. 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 10 monclient: build_initial_monmap 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 10 monclient: monmap: 2024-08-19T09:59:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 1 librados: starting msgr at 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 1 librados: starting objecter 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 1 Processor -- start 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 1 -- start start 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 1 librados: setting wanted keys 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 1 librados: calling monclient init 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.127+0000 7f5c35020640 10 monclient: init 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient: _add_conns ranks=[7,4,5,0,3,6,2,8,1] 2024-08-19T09:59:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient(hunting): picked mon.noname-i con 0x7f5c300a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient(hunting): picked mon.noname-c con 0x7f5c300a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient(hunting): picked mon.noname-h con 0x7f5c3013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3013bad0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c301cf020 con 0x7f5c3013f5c0 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c30139a70 con 0x7f5c300a67c0 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c35020640 10 monclient(hunting): authenticate will time out at 1226.204290s 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c2e59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f5c300a6ba0 0x7f5c301ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:46142/0 (socket says 172.21.15.1:46142) 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.128+0000 7f5c2e59c640 1 -- 172.21.15.1:0/3972437793 learned_addr learned my addr 172.21.15.1:0/3972437793 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c2e59c640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.1:6791/0 conn(0x7f5c300a6ba0 legacy=0x7f5c301ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 client.?.objecter ms_handle_connect 0x7f5c300a6ba0 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 client.?.objecter ms_handle_connect 0x7f5c300a67c0 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c2ed9d640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c300a67c0 legacy=0x7f5c301ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 client.?.objecter ms_handle_connect 0x7f5c3013f5c0 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c2f59e640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6791/0 conn(0x7f5c3013f5c0 legacy=0x7f5c301d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625810570 0 0) 0x7f5c3013bad0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): my global_id is 27616 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c1c004ee0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2445548210 0 0) 0x7f5c30139a70 con 0x7f5c300a67c0 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): my global_id is 144745 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3013bad0 con 0x7f5c300a67c0 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446791953 0 0) 0x7f5c301cf020 con 0x7f5c3013f5c0 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): my global_id is 27275 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c30139a70 con 0x7f5c3013f5c0 2024-08-19T09:59:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2493296912 0 0) 0x7f5c1c004ee0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c301cf020 con 0x7f5c300a6ba0 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c24001d00 con 0x7f5c300a6ba0 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.130+0000 7f5c177fe640 10 monclient(hunting): dump: 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3491384418 0 0) 0x7f5c3013bad0 con 0x7f5c300a67c0 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c1c004ee0 con 0x7f5c300a67c0 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c20003610 con 0x7f5c300a67c0 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): dump: 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 452430477 0 0) 0x7f5c30139a70 con 0x7f5c3013f5c0 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c3013bad0 con 0x7f5c3013f5c0 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c280020b0 con 0x7f5c3013f5c0 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient(hunting): dump: 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.151 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4243023269 0 0) 0x7f5c301cf020 con 0x7f5c300a6ba0 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c300a67c0 legacy=0x7f5c301ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6791/0 conn(0x7f5c3013f5c0 legacy=0x7f5c301d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 monclient: found mon.c 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c30139cb0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 5 monclient: authenticate success, global_id 27616 2024-08-19T09:59:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 10 monclient: _renew_subs 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3013bdb0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 10 client.27616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 10 monclient: _renew_subs 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c3013b7f0 con 0x7f5c300a6ba0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c35020640 1 librados: init done 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c24003a70 con 0x7f5c300a6ba0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c24003f60 con 0x7f5c300a6ba0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:30.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c2d59a640 4 set_mon_vals no callback set 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.131+0000 7f5c177fe640 10 monclient: dump: 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.155 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c35020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c100065c0 con 0x7f5c10002860 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5c24004180 con 0x7f5c300a6ba0 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(365..365 src has 299..365) ==== 5436+0+0 (unknown 3179947854 0 0) 0x7f5c2404e220 con 0x7f5c300a6ba0 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c177fe640 10 client.27616.objecter ms_dispatch 0x7f5c30000b90 osd_map(365..365 src has 299..365) 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c177fe640 3 client.27616.objecter handle_osd_map got epochs [365,365] > 0 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c177fe640 3 client.27616.objecter handle_osd_map decoding full epoch 365 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c177fe640 20 client.27616.objecter dump_active .. 0 homeless 2024-08-19T09:59:30.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.132+0000 7f5c2f59e640 10 monclient: get_auth_request con 0x7f5c1c041a10 auth_method 0 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 client.27616.objecter ms_handle_connect 0x7f5c10002860 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c2ed9d640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c10002860 legacy=0x7f5c10004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 client.27616.objecter resend_mon_ops 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300493767 0 0) 0x7f5c100065c0 con 0x7f5c10002860 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient(hunting): my global_id is 144754 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c301cf020 con 0x7f5c10002860 2024-08-19T09:59:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3202622818 0 0) 0x7f5c301cf020 con 0x7f5c10002860 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c100065c0 con 0x7f5c10002860 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 client.27616.objecter ms_handle_connect 0x7f5c1c041a10 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c20003de0 con 0x7f5c10002860 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 10 monclient: dump: 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2732272346 0 0) 0x7f5c100065c0 con 0x7f5c10002860 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.134+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c1000a160 con 0x7f5c10002860 2024-08-19T09:59:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.135+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5c200040d0 con 0x7f5c10002860 2024-08-19T09:59:30.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.135+0000 7f5c177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:30.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.135+0000 7f5c177fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:30.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.135+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c10002860 legacy=0x7f5c10004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.135+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c10002860 legacy=0x7f5c10004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:30.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.141+0000 7f5c35020640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:30.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.141+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c10009e50 con 0x7f5c10009110 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 client.27616.objecter ms_handle_connect 0x7f5c10009110 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c2ed9d640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c10009110 legacy=0x7f5c100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 client.27616.objecter resend_mon_ops 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3739744814 0 0) 0x7f5c10009e50 con 0x7f5c10009110 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient(hunting): my global_id is 144763 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c100065c0 con 0x7f5c10009110 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 217125647 0 0) 0x7f5c100065c0 con 0x7f5c10009110 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c10009e50 con 0x7f5c10009110 2024-08-19T09:59:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c20005b00 con 0x7f5c10009110 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient: dump: 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:30.163 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1493018266 0 0) 0x7f5c10009e50 con 0x7f5c10009110 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c1000c3f0 con 0x7f5c10009110 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7684 (unknown 0 0 3177596680) 0x7f5c20006e00 con 0x7f5c10009110 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c10009110 legacy=0x7f5c100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.144+0000 7f5c177fe640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6792/0 conn(0x7f5c10009110 legacy=0x7f5c100095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 294523, 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T09:59:30.165 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:30.166 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:30.168 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:30.169 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:30.170 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:30.173 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:30.176 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.179 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.180 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.181 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:30.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:30.184 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:30.185 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 10 librados: watch_flush enter 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 10 librados: watch_flush exit 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 20 client.27616.objecter shutdown clearing up homeless session... 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 10 client.27616.objecter successfully canceled tick 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5c1c041a10 legacy=0x7f5c1c043ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 10 monclient: shutdown 2024-08-19T09:59:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 >> v1:172.21.15.1:6791/0 conn(0x7f5c300a6ba0 legacy=0x7f5c301ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c2f59e640 1 -- 172.21.15.1:0/3972437793 reap_dead start 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c2f59e640 1 -- 172.21.15.1:0/3972437793 reap_dead start 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 shutdown_connections 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 >> 172.21.15.1:0/3972437793 conn(0x7f5c300ac4b0 msgr2=0x7f5c301419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 shutdown_connections 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 -- 172.21.15.1:0/3972437793 wait complete. 2024-08-19T09:59:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:30.145+0000 7f5c35020640 1 librados: shutdown 2024-08-19T09:59:30.188 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T09:59:30.189 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g', 'b', 'e'] 2024-08-19T09:59:30.189 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'f'] 2024-08-19T09:59:30.189 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-08-19T09:59:30.189 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-08-19T09:59:30.189 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T09:59:30.189 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:59:30.206 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T09:59:30.206 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-19T09:59:30.206 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-19T09:59:30.206 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-19T09:59:30.206 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:59:30.240 INFO:tasks.ceph.mon.b:Stopped 2024-08-19T09:59:30.240 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-08-19T09:59:30.240 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-08-19T09:59:30.240 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T09:59:30.240 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T09:59:30.291 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T09:59:30.292 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-08-19T09:59:30.292 INFO:tasks.ceph.mon.a:Sent signal 19 2024-08-19T09:59:30.292 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-08-19T09:59:30.292 INFO:tasks.ceph.mon.f:Sent signal 19 2024-08-19T09:59:30.293 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T09:59:45.293 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-08-19T09:59:45.294 INFO:tasks.ceph.mon.a:Sent signal 18 2024-08-19T09:59:45.294 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-08-19T09:59:45.294 INFO:tasks.ceph.mon.f:Sent signal 18 2024-08-19T09:59:45.294 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-08-19T09:59:45.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4095798196}) 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159618325}) 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2373983497}) 2024-08-19T09:59:45.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=773284026}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3619129250}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3969429087}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1770911210}) 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T09:59:45.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5979}) 2024-08-19T09:59:45.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T09:59:45.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1959971958}) 2024-08-19T09:59:45.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 10 monclient: get_monmap_and_config 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 10 monclient: build_initial_monmap 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 10 monclient: monmap: 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:45.405789+0000 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:45.405789+0000 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:45.406 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:45.407 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:45.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 1 Processor -- start 2024-08-19T09:59:45.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 1 -- start start 2024-08-19T09:59:45.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 10 monclient: init 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.404+0000 7f75211b8640 10 monclient: _add_conns ranks=[2,3,0,4,6,1,8,5,7] 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 10 monclient(hunting): picked mon.noname-b con 0x7f751c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 10 monclient(hunting): picked mon.noname-g con 0x7f751c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 10 monclient(hunting): picked mon.noname-a con 0x7f751c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c0a3b40 con 0x7f751c13f5c0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c0a8130 con 0x7f751c0a67c0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c06ede0 con 0x7f751c0a6ba0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f751ad9d640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f751c0a67c0 legacy=0x7f751c0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f75211b8640 10 monclient(hunting): authenticate will time out at 1241.482109s 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f751a59c640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f751c0a6ba0 legacy=0x7f751c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f751b59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 0x7f751c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53026/0 (socket says 172.21.15.1:53026) 2024-08-19T09:59:45.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.405+0000 7f751b59e640 1 -- 172.21.15.1:0/3935358814 learned_addr learned my addr 172.21.15.1:0/3935358814 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:45.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.408+0000 7f751b59e640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 legacy=0x7f751c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:45.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.408+0000 7f751b59e640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 legacy=0x7f751c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-08-19T09:59:45.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.408+0000 7f751b59e640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 legacy=0x7f751c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T09:59:45.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.408+0000 7f751b59e640 1 --1- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 0x7f751c1419b0 :-1 s=OPENED pgs=402 cs=1 l=1).handle_message read tag failed 2024-08-19T09:59:45.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.408+0000 7f751b59e640 1 --1- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 0x7f751c1419b0 :-1 s=OPENED pgs=402 cs=1 l=1).fault on lossy channel, failing 2024-08-19T09:59:45.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.408+0000 7f7519d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6789/0 2024-08-19T09:59:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.606+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6790/0 conn(0x7f751c0a67c0 legacy=0x7f751c0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:59:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:45.606+0000 7f751a59c640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6790/0 conn(0x7f751c0a6ba0 legacy=0x7f751c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:46.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:46.007+0000 7f751a59c640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6790/0 conn(0x7f751c0a6ba0 legacy=0x7f751c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:46.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:46.008+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6790/0 conn(0x7f751c0a67c0 legacy=0x7f751c0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:59:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:46.808+0000 7f751a59c640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6790/0 conn(0x7f751c0a6ba0 legacy=0x7f751c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:46.809+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6790/0 conn(0x7f751c0a67c0 legacy=0x7f751c0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:59:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): tick 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 monclient(hunting): continuing hunt 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6790/0 conn(0x7f751c0a6ba0 legacy=0x7f751c0a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6790/0 conn(0x7f751c0a67c0 legacy=0x7f751c0a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6789/0 conn(0x7f751c13f5c0 legacy=0x7f751c1419b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient: _add_conns ranks=[8,5,6,4,1,7] 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): picked mon.noname-e con 0x7f75080036c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): picked mon.noname-h con 0x7f7508007350 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): picked mon.noname-d con 0x7f750800b0e0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 -- 172.21.15.1:0/3935358814 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c0a92a0 con 0x7f7508007350 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6793/0 conn(0x7f75080036c0 legacy=0x7f7508005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 -- 172.21.15.1:0/3935358814 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c0a8130 con 0x7f750800b0e0 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 1 -- 172.21.15.1:0/3935358814 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c06ede0 con 0x7f75080036c0 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f7518d99640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f751a59c640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6791/0 conn(0x7f7508007350 legacy=0x7f7508009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f751a59c640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6791/0 conn(0x7f7508007350 legacy=0x7f7508009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.406+0000 7f751a59c640 1 --1- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6791/0 conn(0x7f7508007350 0x7f7508009810 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-08-19T09:59:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.408+0000 7f751b59e640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6792/0 conn(0x7f750800b0e0 legacy=0x7f750800d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:48.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.607+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6793/0 conn(0x7f75080036c0 legacy=0x7f7508005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:48.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:48.609+0000 7f751a59c640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6791/0 conn(0x7f7508007350 legacy=0x7f7508009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:49.008+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6793/0 conn(0x7f75080036c0 legacy=0x7f7508005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:49.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:49.809+0000 7f751ad9d640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6793/0 conn(0x7f75080036c0 legacy=0x7f7508005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408834507 0 0) 0x7f751c0a92a0 con 0x7f7508007350 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): my global_id is 27284 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7504003520 con 0x7f7508007350 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056605912 0 0) 0x7f751c0a8130 con 0x7f750800b0e0 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient(hunting): my global_id is 174282 2024-08-19T09:59:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f751c0a92a0 con 0x7f750800b0e0 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1332503802 0 0) 0x7f7504003520 con 0x7f7508007350 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6793/0 conn(0x7f75080036c0 legacy=0x7f7508005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.1:6792/0 conn(0x7f750800b0e0 legacy=0x7f750800d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient: _finish_hunting 0 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 monclient: found mon.noname-h 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7508011f80 con 0x7f7508007350 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient: _finish_auth 0 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient: _check_auth_tickets 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f75100044e0 con 0x7f7508007350 2024-08-19T09:59:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.299+0000 7f7519d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 10 monclient: dump: 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f751b59e640 1 -- 172.21.15.1:0/3935358814 reap_dead start 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f75211b8640 5 monclient: authenticate success, global_id 27284 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f75211b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7510004780 con 0x7f7508007350 2024-08-19T09:59:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 1 -- 172.21.15.1:0/3935358814 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7510004cb0 con 0x7f7508007350 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f7519d9b640 10 monclient: dump: 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f75211b8640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.300+0000 7f75211b8640 4 set_mon_vals no callback set 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 10 monclient: shutdown 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 1 -- 172.21.15.1:0/3935358814 >> v1:172.21.15.82:6791/0 conn(0x7f7508007350 legacy=0x7f7508009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 1 -- 172.21.15.1:0/3935358814 shutdown_connections 2024-08-19T09:59:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 1 -- 172.21.15.1:0/3935358814 >> 172.21.15.1:0/3935358814 conn(0x7f751c0ac4b0 msgr2=0x7f751c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 1 -- 172.21.15.1:0/3935358814 shutdown_connections 2024-08-19T09:59:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.301+0000 7f75211b8640 1 -- 172.21.15.1:0/3935358814 wait complete. 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.303+0000 7f75211b8640 10 monclient: build_initial_monmap 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.303+0000 7f75211b8640 10 monclient: monmap: 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.303+0000 7f75211b8640 1 librados: starting msgr at 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.303+0000 7f75211b8640 1 librados: starting objecter 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.304+0000 7f75211b8640 1 Processor -- start 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.304+0000 7f75211b8640 1 -- start start 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.304+0000 7f75211b8640 1 librados: setting wanted keys 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.304+0000 7f75211b8640 1 librados: calling monclient init 2024-08-19T09:59:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.304+0000 7f75211b8640 10 monclient: init 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient: _add_conns ranks=[2,1,6,0,7,8,4,3,5] 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient(hunting): picked mon.noname-b con 0x7f7508007350 addr v1:172.21.15.1:6790/0 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient(hunting): picked mon.noname-f con 0x7f751c1cdb30 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f751ad9d640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7f7508007350 legacy=0x7f751c13e960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6790/0 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient(hunting): picked mon.noname-d con 0x7f751c1d4570 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c127fa0 con 0x7f751c1cdb30 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f751b59e640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f751c1d4570 0x7f751c1d6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:40172/0 (socket says 172.21.15.1:40172) 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f751b59e640 1 -- 172.21.15.1:0/727391104 learned_addr learned my addr 172.21.15.1:0/727391104 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c0aa2c0 con 0x7f7508007350 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f751c1cdfa0 con 0x7f751c1d4570 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.305+0000 7f75211b8640 10 monclient(hunting): authenticate will time out at 1247.381988s 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.308+0000 7f751b59e640 1 -- 172.21.15.1:0/727391104 >> v1:172.21.15.1:6792/0 conn(0x7f751c1d4570 legacy=0x7f751c1d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.308+0000 7f75037fe640 10 client.?.objecter ms_handle_connect 0x7f751c1d4570 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.308+0000 7f75037fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.309+0000 7f751a59c640 1 -- 172.21.15.1:0/727391104 >> v1:172.21.15.82:6789/0 conn(0x7f751c1cdb30 legacy=0x7f751c1d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.309+0000 7f75037fe640 10 client.?.objecter ms_handle_connect 0x7f751c1cdb30 2024-08-19T09:59:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.309+0000 7f75037fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 386430991 0 0) 0x7f751c127fa0 con 0x7f751c1cdb30 2024-08-19T09:59:51.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): my global_id is 174241 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7508005e80 con 0x7f751c1cdb30 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119106330 0 0) 0x7f751c1cdfa0 con 0x7f751c1d4570 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): my global_id is 174291 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f751c127fa0 con 0x7f751c1d4570 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 361354958 0 0) 0x7f7508005e80 con 0x7f751c1cdb30 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f751c1cdfa0 con 0x7f751c1cdb30 2024-08-19T09:59:51.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7510004c70 con 0x7f751c1cdb30 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): dump: 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 40711901 0 0) 0x7f751c127fa0 con 0x7f751c1d4570 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7508005e80 con 0x7f751c1d4570 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f75140014e0 con 0x7f751c1d4570 2024-08-19T09:59:51.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 10 monclient(hunting): dump: 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.324 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4174071540 0 0) 0x7f751c1cdfa0 con 0x7f751c1cdb30 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.316+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 >> v1:172.21.15.1:6792/0 conn(0x7f751c1d4570 legacy=0x7f751c1d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 >> v1:172.21.15.1:6790/0 conn(0x7f7508007350 legacy=0x7f751c13e960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 1 monclient: found mon.f 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751c12a300 con 0x7f751c1cdb30 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 5 monclient: authenticate success, global_id 174241 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 10 monclient: _renew_subs 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f751c129a60 con 0x7f751c1cdb30 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75100050c0 con 0x7f751c1cdb30 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 10 client.174241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f75100055b0 con 0x7f751c1cdb30 2024-08-19T09:59:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 10 monclient: _renew_subs 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f751c13f070 con 0x7f751c1cdb30 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75211b8640 1 librados: init done 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f751959a640 4 set_mon_vals no callback set 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.317+0000 7f75037fe640 10 monclient: dump: 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7510011a10 con 0x7f751c1cdb30 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(366..366 src has 299..366) ==== 5449+0+0 (unknown 1073799134 0 0) 0x7f751004e140 con 0x7f751c1cdb30 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75037fe640 10 client.174241.objecter ms_dispatch 0x7f751c000b90 osd_map(366..366 src has 299..366) 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75037fe640 3 client.174241.objecter handle_osd_map got epochs [366,366] > 0 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75037fe640 3 client.174241.objecter handle_osd_map decoding full epoch 366 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75037fe640 20 client.174241.objecter dump_active .. 0 homeless 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f751ad9d640 10 monclient: get_auth_request con 0x7f75080425a0 auth_method 0 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75211b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75211b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75211b8640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.318+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f751c12a120 con 0x7f751c1cdb30 2024-08-19T09:59:51.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.321+0000 7f75037fe640 10 client.174241.objecter ms_handle_connect 0x7f75080425a0 2024-08-19T09:59:51.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.324+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f751c12a120 con 0x7f751c1cdb30 2024-08-19T09:59:51.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.324+0000 7f75037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.324+0000 7f75037fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.581+0000 7f75211b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T09:59:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.581+0000 7f75211b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T09:59:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.581+0000 7f75211b8640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.581+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f751c06ee70 con 0x7f751c1cdb30 2024-08-19T09:59:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.582+0000 7f75037fe640 1 -- 172.21.15.1:0/727391104 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2684 (unknown 254746657 0 2623146271) 0x7f7510017820 con 0x7f751c1cdb30 2024-08-19T09:59:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.582+0000 7f75037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T09:59:51.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.582+0000 7f75037fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:51.584 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":94,"quorum":[0,1,4,5,6,7],"quorum_names":["a","f","c","h","d","i"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 10 librados: watch_flush enter 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 10 librados: watch_flush exit 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 20 client.174241.objecter shutdown clearing up homeless session... 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 10 client.174241.objecter successfully canceled tick 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 >> v1:172.21.15.82:6812/2549104415 conn(0x7f75080425a0 legacy=0x7f7508044a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 10 monclient: shutdown 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 >> v1:172.21.15.82:6789/0 conn(0x7f751c1cdb30 legacy=0x7f751c1d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 shutdown_connections 2024-08-19T09:59:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.584+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 >> 172.21.15.1:0/727391104 conn(0x7f751c0ac4b0 msgr2=0x7f751c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.585+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 shutdown_connections 2024-08-19T09:59:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.585+0000 7f75211b8640 1 -- 172.21.15.1:0/727391104 wait complete. 2024-08-19T09:59:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.585+0000 7f75211b8640 1 librados: shutdown 2024-08-19T09:59:51.597 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-08-19T09:59:51.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon a 2024-08-19T09:59:51.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.695+0000 7fb44fb43640 10 monclient: get_monmap_and_config 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.695+0000 7fb44fb43640 10 monclient: build_initial_monmap 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.695+0000 7fb44fb43640 10 monclient: monmap: 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:51.696655+0000 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:51.696655+0000 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:51.697 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.696+0000 7fb44fb43640 1 Processor -- start 2024-08-19T09:59:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.696+0000 7fb44fb43640 1 -- start start 2024-08-19T09:59:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.696+0000 7fb44fb43640 10 monclient: init 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.696+0000 7fb44fb43640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.696+0000 7fb44fb43640 10 monclient: _add_conns ranks=[1,3,5,8,2,0,4,7,6] 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 10 monclient(hunting): picked mon.noname-f con 0x7fb4480a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 10 monclient(hunting): picked mon.noname-g con 0x7fb4480a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 10 monclient(hunting): picked mon.noname-h con 0x7fb44813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a3b40 con 0x7fb4480a67c0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a8130 con 0x7fb4480a6ba0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44806ede0 con 0x7fb44813f5c0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44fb43640 10 monclient(hunting): authenticate will time out at 1247.773082s 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44d0de640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fb4480a6ba0 legacy=0x7fb4480a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44d8df640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb4480a67c0 0x7fb4480a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:50248/0 (socket says 172.21.15.1:50248) 2024-08-19T09:59:51.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.697+0000 7fb44d8df640 1 -- 172.21.15.1:0/3672140404 learned_addr learned my addr 172.21.15.1:0/3672140404 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44d8df640 1 -- 172.21.15.1:0/3672140404 >> v1:172.21.15.82:6789/0 conn(0x7fb4480a67c0 legacy=0x7fb4480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44e0e0640 1 -- 172.21.15.1:0/3672140404 >> v1:172.21.15.82:6791/0 conn(0x7fb44813f5c0 legacy=0x7fb4481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630038473 0 0) 0x7fb4480a3b40 con 0x7fb4480a67c0 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): my global_id is 174250 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb434003520 con 0x7fb4480a67c0 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1442272756 0 0) 0x7fb44806ede0 con 0x7fb44813f5c0 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient(hunting): my global_id is 27293 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4480a3b40 con 0x7fb44813f5c0 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2189885098 0 0) 0x7fb434003520 con 0x7fb4480a67c0 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 >> v1:172.21.15.82:6791/0 conn(0x7fb44813f5c0 legacy=0x7fb4481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 >> v1:172.21.15.82:6790/0 conn(0x7fb4480a6ba0 legacy=0x7fb4480a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: _finish_hunting 0 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 monclient: found mon.noname-f 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4480a92a0 con 0x7fb4480a67c0 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: _finish_auth 0 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: _check_auth_tickets 2024-08-19T09:59:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb444001ba0 con 0x7fb4480a67c0 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44fb43640 5 monclient: authenticate success, global_id 174250 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44fb43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44c8dd640 10 monclient: dump: 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.699+0000 7fb44fb43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4440048a0 con 0x7fb4480a67c0 2024-08-19T09:59:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44c8dd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44c8dd640 1 -- 172.21.15.1:0/3672140404 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb444004ed0 con 0x7fb4480a67c0 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 4 set_mon_vals no callback set 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44c8dd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44c8dd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44c8dd640 10 monclient: dump: 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 10 monclient: shutdown 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 1 -- 172.21.15.1:0/3672140404 >> v1:172.21.15.82:6789/0 conn(0x7fb4480a67c0 legacy=0x7fb4480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 1 -- 172.21.15.1:0/3672140404 shutdown_connections 2024-08-19T09:59:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 1 -- 172.21.15.1:0/3672140404 >> 172.21.15.1:0/3672140404 conn(0x7fb4480ac4b0 msgr2=0x7fb4480a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 1 -- 172.21.15.1:0/3672140404 shutdown_connections 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.700+0000 7fb44fb43640 1 -- 172.21.15.1:0/3672140404 wait complete. 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 10 monclient: build_initial_monmap 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 10 monclient: monmap: 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 1 librados: starting msgr at 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 1 librados: starting objecter 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 1 Processor -- start 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 1 -- start start 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 1 librados: setting wanted keys 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 1 librados: calling monclient init 2024-08-19T09:59:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.701+0000 7fb44fb43640 10 monclient: init 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient: _add_conns ranks=[6,4,0,3,7,1,5,2,8] 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient(hunting): picked mon.noname-d con 0x7fb4480a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient(hunting): picked mon.noname-c con 0x7fb4480a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient(hunting): picked mon.noname-a con 0x7fb44813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a9fe0 con 0x7fb44813f5c0 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a55b0 con 0x7fb4480a6ba0 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4480a1720 con 0x7fb4480a67c0 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44fb43640 10 monclient(hunting): authenticate will time out at 1247.778285s 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44d0de640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb4480a6ba0 0x7fb4480a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50724/0 (socket says 172.21.15.1:50724) 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44d0de640 1 -- 172.21.15.1:0/2786613546 learned_addr learned my addr 172.21.15.1:0/2786613546 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.702+0000 7fb44d8df640 1 --1- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6792/0 conn(0x7fb4480a67c0 0x7fb44809ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:40180/0 (socket says 172.21.15.1:40180) 2024-08-19T09:59:51.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb44d8df640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6792/0 conn(0x7fb4480a67c0 legacy=0x7fb44809ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 client.?.objecter ms_handle_connect 0x7fb4480a67c0 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 client.?.objecter ms_handle_connect 0x7fb44813f5c0 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb44e0e0640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb44813f5c0 legacy=0x7fb4481d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 client.?.objecter ms_handle_connect 0x7fb4480a6ba0 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb44d0de640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6791/0 conn(0x7fb4480a6ba0 legacy=0x7fb4480a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2731426628 0 0) 0x7fb4480a9fe0 con 0x7fb44813f5c0 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): my global_id is 174285 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb42c004eb0 con 0x7fb44813f5c0 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083868826 0 0) 0x7fb4480a1720 con 0x7fb4480a67c0 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): my global_id is 174300 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4480a9fe0 con 0x7fb4480a67c0 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286944059 0 0) 0x7fb4480a55b0 con 0x7fb4480a6ba0 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): my global_id is 27625 2024-08-19T09:59:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4480a1720 con 0x7fb4480a6ba0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 16286509 0 0) 0x7fb4480a9fe0 con 0x7fb4480a67c0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4480a55b0 con 0x7fb4480a67c0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 231210640 0 0) 0x7fb42c004eb0 con 0x7fb44813f5c0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4480a9fe0 con 0x7fb44813f5c0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb444001b60 con 0x7fb4480a67c0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.704+0000 7fb43e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 351037842 0 0) 0x7fb4480a1720 con 0x7fb4480a6ba0 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb42c004eb0 con 0x7fb4480a6ba0 2024-08-19T09:59:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb440001f80 con 0x7fb44813f5c0 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb438004530 con 0x7fb4480a6ba0 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient(hunting): dump: 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1919990692 0 0) 0x7fb4480a55b0 con 0x7fb4480a67c0 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6791/0 conn(0x7fb4480a6ba0 legacy=0x7fb4480a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb44813f5c0 legacy=0x7fb4481d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 monclient: found mon.d 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44813a880 con 0x7fb4480a67c0 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 5 monclient: authenticate success, global_id 174300 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 10 monclient: _renew_subs 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4480a2280 con 0x7fb4480a67c0 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 10 client.174300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:51.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 10 monclient: _renew_subs 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb448139c90 con 0x7fb4480a67c0 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb44fb43640 1 librados: init done 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4440048a0 con 0x7fb4480a67c0 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb444001580 con 0x7fb4480a67c0 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43ffff640 4 set_mon_vals no callback set 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.705+0000 7fb43e7fc640 10 monclient: dump: 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.730 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb444011a40 con 0x7fb4480a67c0 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb44fb43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb4300065c0 con 0x7fb430002860 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(366..366 src has 299..366) ==== 5449+0+0 (unknown 1073799134 0 0) 0x7fb44404e200 con 0x7fb4480a67c0 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb43e7fc640 10 client.174300.objecter ms_dispatch 0x7fb448000b90 osd_map(366..366 src has 299..366) 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb43e7fc640 3 client.174300.objecter handle_osd_map got epochs [366,366] > 0 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb43e7fc640 3 client.174300.objecter handle_osd_map decoding full epoch 366 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb43e7fc640 20 client.174300.objecter dump_active .. 0 homeless 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.706+0000 7fb44e0e0640 10 monclient: get_auth_request con 0x7fb42c041870 auth_method 0 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb44d0de640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb430002860 legacy=0x7fb430004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 client.174300.objecter ms_handle_connect 0x7fb430002860 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 client.174300.objecter resend_mon_ops 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2680311009 0 0) 0x7fb4300065c0 con 0x7fb430002860 2024-08-19T09:59:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient(hunting): my global_id is 174294 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4480a55b0 con 0x7fb430002860 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3523461181 0 0) 0x7fb4480a55b0 con 0x7fb430002860 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4300065c0 con 0x7fb430002860 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb438004d40 con 0x7fb430002860 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 10 monclient: dump: 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1262288143 0 0) 0x7fb4300065c0 con 0x7fb430002860 2024-08-19T09:59:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.708+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb43000a160 con 0x7fb430002860 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.709+0000 7fb43e7fc640 10 client.174300.objecter ms_handle_connect 0x7fb42c041870 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.709+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb4380062e0 con 0x7fb430002860 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.709+0000 7fb43e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.709+0000 7fb43e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.709+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb430002860 legacy=0x7fb430004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.709+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb430002860 legacy=0x7fb430004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.715+0000 7fb44fb43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.715+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb430009e70 con 0x7fb430009130 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 10 client.174300.objecter ms_handle_connect 0x7fb430009130 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb44d0de640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb430009130 legacy=0x7fb4300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 10 client.174300.objecter resend_mon_ops 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936358850 0 0) 0x7fb430009e70 con 0x7fb430009130 2024-08-19T09:59:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 10 monclient(hunting): my global_id is 174303 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.717+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4300065c0 con 0x7fb430009130 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2078876298 0 0) 0x7fb4300065c0 con 0x7fb430009130 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb430009e70 con 0x7fb430009130 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb4380052f0 con 0x7fb430009130 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 10 monclient: dump: 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3129127939 0 0) 0x7fb430009e70 con 0x7fb430009130 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb43000c3f0 con 0x7fb430009130 2024-08-19T09:59:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7652 (unknown 0 0 2458995904) 0x7fb4380065b0 con 0x7fb430009130 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb430009130 legacy=0x7fb4300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb43e7fc640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6789/0 conn(0x7fb430009130 legacy=0x7fb4300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:51.737 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 41759, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:51.738 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:51.739 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:51.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:51.741 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:51.742 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:51.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:51.746 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:51.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:51.749 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:51.753 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.755 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.756 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:51.757 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 10 librados: watch_flush enter 2024-08-19T09:59:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 10 librados: watch_flush exit 2024-08-19T09:59:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 20 client.174300.objecter shutdown clearing up homeless session... 2024-08-19T09:59:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 10 client.174300.objecter successfully canceled tick 2024-08-19T09:59:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb42c041870 legacy=0x7fb42c043d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 10 monclient: shutdown 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 >> v1:172.21.15.1:6792/0 conn(0x7fb4480a67c0 legacy=0x7fb44809ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44e0e0640 1 -- 172.21.15.1:0/2786613546 reap_dead start 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44e0e0640 1 -- 172.21.15.1:0/2786613546 reap_dead start 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 shutdown_connections 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.718+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 >> 172.21.15.1:0/2786613546 conn(0x7fb4480ac4b0 msgr2=0x7fb4480ab450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.719+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 shutdown_connections 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.719+0000 7fb44fb43640 1 -- 172.21.15.1:0/2786613546 wait complete. 2024-08-19T09:59:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.719+0000 7fb44fb43640 1 librados: shutdown 2024-08-19T09:59:51.760 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:59:51.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon c 2024-08-19T09:59:51.761 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T09:59:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.860+0000 7f5f90728640 10 monclient: get_monmap_and_config 2024-08-19T09:59:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.860+0000 7f5f90728640 10 monclient: build_initial_monmap 2024-08-19T09:59:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.860+0000 7f5f90728640 10 monclient: monmap: 2024-08-19T09:59:51.862 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:51.861419+0000 2024-08-19T09:59:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:51.861419+0000 2024-08-19T09:59:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.863 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:51.864 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:51.865 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 1 Processor -- start 2024-08-19T09:59:51.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 1 -- start start 2024-08-19T09:59:51.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient: init 2024-08-19T09:59:51.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:51.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient: _add_conns ranks=[0,4,8,7,2,3,5,1,6] 2024-08-19T09:59:51.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient(hunting): picked mon.noname-a con 0x7f5f880a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:51.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient(hunting): picked mon.noname-c con 0x7f5f880a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:51.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient(hunting): picked mon.noname-e con 0x7f5f8813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:51.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f880a3b40 con 0x7f5f880a67c0 2024-08-19T09:59:51.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f880a8130 con 0x7f5f880a6ba0 2024-08-19T09:59:51.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f8806ede0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f90728640 10 monclient(hunting): authenticate will time out at 1247.937773s 2024-08-19T09:59:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f8ecc5640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f5f8813f5c0 legacy=0x7f5f881419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f8e4c4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5f880a67c0 0x7f5f880a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56796/0 (socket says 172.21.15.1:56796) 2024-08-19T09:59:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.861+0000 7f5f8e4c4640 1 -- 172.21.15.1:0/1938023227 learned_addr learned my addr 172.21.15.1:0/1938023227 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8dcc3640 1 -- 172.21.15.1:0/1938023227 >> v1:172.21.15.1:6791/0 conn(0x7f5f880a6ba0 legacy=0x7f5f880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8e4c4640 1 -- 172.21.15.1:0/1938023227 >> v1:172.21.15.1:6789/0 conn(0x7f5f880a67c0 legacy=0x7f5f880a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141445121 0 0) 0x7f5f880a8130 con 0x7f5f880a6ba0 2024-08-19T09:59:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): my global_id is 27634 2024-08-19T09:59:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f74003520 con 0x7f5f880a6ba0 2024-08-19T09:59:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3152246949 0 0) 0x7f5f880a3b40 con 0x7f5f880a67c0 2024-08-19T09:59:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient(hunting): my global_id is 174312 2024-08-19T09:59:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f880a8130 con 0x7f5f880a67c0 2024-08-19T09:59:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3659253520 0 0) 0x7f5f74003520 con 0x7f5f880a6ba0 2024-08-19T09:59:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 >> v1:172.21.15.1:6793/0 conn(0x7f5f8813f5c0 legacy=0x7f5f881419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:51.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 >> v1:172.21.15.1:6789/0 conn(0x7f5f880a67c0 legacy=0x7f5f880a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: _finish_hunting 0 2024-08-19T09:59:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 monclient: found mon.noname-c 2024-08-19T09:59:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T09:59:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f880a92a0 con 0x7f5f880a6ba0 2024-08-19T09:59:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: _finish_auth 0 2024-08-19T09:59:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: _check_auth_tickets 2024-08-19T09:59:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f78001b30 con 0x7f5f880a6ba0 2024-08-19T09:59:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f90728640 5 monclient: authenticate success, global_id 27634 2024-08-19T09:59:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f90728640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: dump: 2024-08-19T09:59:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.906 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.906 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.906 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.906 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.906 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.907 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.907 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.907 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.907 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f90728640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:51.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f78003610 con 0x7f5f880a6ba0 2024-08-19T09:59:51.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:51.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 1 -- 172.21.15.1:0/1938023227 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f78003b80 con 0x7f5f880a6ba0 2024-08-19T09:59:51.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f90728640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:51.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f90728640 4 set_mon_vals no callback set 2024-08-19T09:59:51.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:51.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.864+0000 7f5f8d4c2640 10 monclient: dump: 2024-08-19T09:59:51.909 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.909 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.910 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.910 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.910 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.910 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.910 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.911 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.911 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.911 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.911 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.911 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.911 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.912 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.912 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.912 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 10 monclient: shutdown 2024-08-19T09:59:51.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:51.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 1 -- 172.21.15.1:0/1938023227 >> v1:172.21.15.1:6791/0 conn(0x7f5f880a6ba0 legacy=0x7f5f880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 1 -- 172.21.15.1:0/1938023227 shutdown_connections 2024-08-19T09:59:51.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 1 -- 172.21.15.1:0/1938023227 >> 172.21.15.1:0/1938023227 conn(0x7f5f880ac4b0 msgr2=0x7f5f880a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:51.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 1 -- 172.21.15.1:0/1938023227 shutdown_connections 2024-08-19T09:59:51.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 1 -- 172.21.15.1:0/1938023227 wait complete. 2024-08-19T09:59:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 10 monclient: build_initial_monmap 2024-08-19T09:59:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.865+0000 7f5f90728640 10 monclient: monmap: 2024-08-19T09:59:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:51.917 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 librados: starting msgr at 2024-08-19T09:59:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 librados: starting objecter 2024-08-19T09:59:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 Processor -- start 2024-08-19T09:59:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 -- start start 2024-08-19T09:59:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 librados: setting wanted keys 2024-08-19T09:59:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 librados: calling monclient init 2024-08-19T09:59:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient: init 2024-08-19T09:59:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient: _add_conns ranks=[4,3,0,5,7,1,2,6,8] 2024-08-19T09:59:51.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient(hunting): picked mon.noname-c con 0x7f5f880a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:51.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient(hunting): picked mon.noname-g con 0x7f5f880a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:51.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient(hunting): picked mon.noname-a con 0x7f5f8813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:51.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f8813d6d0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f8e4c4640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f5f880a67c0 0x7f5f8813eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50740/0 (socket says 172.21.15.1:50740) 2024-08-19T09:59:51.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f8e4c4640 1 -- 172.21.15.1:0/1339756747 learned_addr learned my addr 172.21.15.1:0/1339756747 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:51.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f8813b370 con 0x7f5f880a6ba0 2024-08-19T09:59:51.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f8dcc3640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.82:6790/0 conn(0x7f5f880a6ba0 legacy=0x7f5f881d2b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:51.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f8813d3f0 con 0x7f5f880a67c0 2024-08-19T09:59:51.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:51.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:59:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.866+0000 7f5f90728640 10 monclient(hunting): authenticate will time out at 1247.942842s 2024-08-19T09:59:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.868+0000 7f5f8e4c4640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f880a67c0 legacy=0x7f5f8813eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.868+0000 7f5f7effd640 10 client.?.objecter ms_handle_connect 0x7f5f880a67c0 2024-08-19T09:59:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.868+0000 7f5f7effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.868+0000 7f5f7effd640 10 client.?.objecter ms_handle_connect 0x7f5f8813f5c0 2024-08-19T09:59:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.868+0000 7f5f7effd640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.868+0000 7f5f8ecc5640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6789/0 conn(0x7f5f8813f5c0 legacy=0x7f5f881d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197358983 0 0) 0x7f5f8813d3f0 con 0x7f5f880a67c0 2024-08-19T09:59:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): my global_id is 27643 2024-08-19T09:59:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f6c004ee0 con 0x7f5f880a67c0 2024-08-19T09:59:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1131827054 0 0) 0x7f5f8813d6d0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): my global_id is 174321 2024-08-19T09:59:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f8813d3f0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3622482153 0 0) 0x7f5f8813d3f0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f8813d6d0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f800047d0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): dump: 2024-08-19T09:59:51.926 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.926 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.926 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.926 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.926 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.927 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.927 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.927 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.927 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.927 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.927 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.928 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.928 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.928 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.928 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.928 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:51.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3362154689 0 0) 0x7f5f6c004ee0 con 0x7f5f880a67c0 2024-08-19T09:59:51.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f8813d3f0 con 0x7f5f880a67c0 2024-08-19T09:59:51.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f840047e0 con 0x7f5f880a67c0 2024-08-19T09:59:51.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:51.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:51.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient(hunting): dump: 2024-08-19T09:59:51.930 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.930 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.930 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.930 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.931 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.931 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.931 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.931 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.931 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.932 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.932 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.932 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.932 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.932 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.932 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.933 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2904015986 0 0) 0x7f5f8813d6d0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f880a67c0 legacy=0x7f5f8813eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.82:6790/0 conn(0x7f5f880a6ba0 legacy=0x7f5f881d2b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:51.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: _finish_hunting 0 2024-08-19T09:59:51.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 monclient: found mon.a 2024-08-19T09:59:51.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:51.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:51.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8813c1a0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: _finish_auth 0 2024-08-19T09:59:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: _check_auth_tickets 2024-08-19T09:59:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 5 monclient: authenticate success, global_id 174321 2024-08-19T09:59:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 10 monclient: _renew_subs 2024-08-19T09:59:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8813bed0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 10 client.174321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 10 monclient: _renew_subs 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f8813f210 con 0x7f5f8813f5c0 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f90728640 1 librados: init done 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f80004d50 con 0x7f5f8813f5c0 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f80005240 con 0x7f5f8813f5c0 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f8ccc1640 4 set_mon_vals no callback set 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.869+0000 7f5f7effd640 10 monclient: dump: 2024-08-19T09:59:51.936 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.937 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f90728640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f700065c0 con 0x7f5f70002860 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5f800053e0 con 0x7f5f8813f5c0 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(366..366 src has 299..366) ==== 5449+0+0 (unknown 1073799134 0 0) 0x7f5f8004e410 con 0x7f5f8813f5c0 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f7effd640 10 client.174321.objecter ms_dispatch 0x7f5f88000b90 osd_map(366..366 src has 299..366) 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f7effd640 3 client.174321.objecter handle_osd_map got epochs [366,366] > 0 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f7effd640 3 client.174321.objecter handle_osd_map decoding full epoch 366 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.870+0000 7f5f7effd640 20 client.174321.objecter dump_active .. 0 homeless 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.871+0000 7f5f8dcc3640 10 monclient: get_auth_request con 0x7f5f6c0414b0 auth_method 0 2024-08-19T09:59:51.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 10 client.174321.objecter ms_handle_connect 0x7f5f70002860 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 10 client.174321.objecter resend_mon_ops 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f8e4c4640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f70002860 legacy=0x7f5f70004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330641894 0 0) 0x7f5f700065c0 con 0x7f5f70002860 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 10 monclient(hunting): my global_id is 27652 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f8813d6d0 con 0x7f5f70002860 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4075167494 0 0) 0x7f5f8813d6d0 con 0x7f5f70002860 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f700065c0 con 0x7f5f70002860 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.872+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f84004f40 con 0x7f5f70002860 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 10 monclient: dump: 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3510831449 0 0) 0x7f5f700065c0 con 0x7f5f70002860 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f7000a160 con 0x7f5f70002860 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 10 client.174321.objecter ms_handle_connect 0x7f5f6c0414b0 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5f840064a0 con 0x7f5f70002860 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f70002860 legacy=0x7f5f70004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.873+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f70002860 legacy=0x7f5f70004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.879+0000 7f5f90728640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.879+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f7000c4f0 con 0x7f5f70009110 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 client.174321.objecter ms_handle_connect 0x7f5f70009110 2024-08-19T09:59:51.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 client.174321.objecter resend_mon_ops 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f8e4c4640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f70009110 legacy=0x7f5f70009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3798645415 0 0) 0x7f5f7000c4f0 con 0x7f5f70009110 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient(hunting): my global_id is 27661 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f700065c0 con 0x7f5f70009110 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1261178194 0 0) 0x7f5f700065c0 con 0x7f5f70009110 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f7000c4f0 con 0x7f5f70009110 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5f840054b0 con 0x7f5f70009110 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient: dump: 2024-08-19T09:59:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 444377016 0 0) 0x7f5f7000c4f0 con 0x7f5f70009110 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f70009ce0 con 0x7f5f70009110 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7651 (unknown 0 0 663238293) 0x7f5f840055e0 con 0x7f5f70009110 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f70009110 legacy=0x7f5f70009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.882+0000 7f5f7effd640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6791/0 conn(0x7f5f70009110 legacy=0x7f5f70009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 598154, 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:51.945 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:51.946 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:51.947 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:51.948 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:51.949 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:51.950 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:51.951 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.952 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.956 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:51.957 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.958 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:51.958 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:51.958 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.958 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.958 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.961 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:51.962 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.963 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:51.964 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.965 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.966 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.966 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:51.966 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:51.966 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:51.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:51.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:51.970 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:51.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:51.971 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:51.971 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 10 librados: watch_flush enter 2024-08-19T09:59:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 10 librados: watch_flush exit 2024-08-19T09:59:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 20 client.174321.objecter shutdown clearing up homeless session... 2024-08-19T09:59:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 10 client.174321.objecter successfully canceled tick 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5f6c0414b0 legacy=0x7f5f6c043970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 10 monclient: shutdown 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 >> v1:172.21.15.1:6789/0 conn(0x7f5f8813f5c0 legacy=0x7f5f881d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f8ecc5640 1 -- 172.21.15.1:0/1339756747 reap_dead start 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f8ecc5640 1 -- 172.21.15.1:0/1339756747 reap_dead start 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 shutdown_connections 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 >> 172.21.15.1:0/1339756747 conn(0x7f5f880ac4b0 msgr2=0x7f5f881419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 shutdown_connections 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 -- 172.21.15.1:0/1339756747 wait complete. 2024-08-19T09:59:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:51.883+0000 7f5f90728640 1 librados: shutdown 2024-08-19T09:59:51.975 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:59:51.975 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon d 2024-08-19T09:59:51.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T09:59:52.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.074+0000 7f4fa6b5a640 10 monclient: get_monmap_and_config 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.074+0000 7f4fa6b5a640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.074+0000 7f4fa6b5a640 10 monclient: monmap: 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:52.075794+0000 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:52.075794+0000 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.077 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.075+0000 7f4fa6b5a640 1 Processor -- start 2024-08-19T09:59:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.075+0000 7f4fa6b5a640 1 -- start start 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.075+0000 7f4fa6b5a640 10 monclient: init 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient: _add_conns ranks=[3,6,7,4,8,1,5,0,2] 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient(hunting): picked mon.noname-g con 0x7f4fa00a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient(hunting): picked mon.noname-d con 0x7f4fa00a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient(hunting): picked mon.noname-i con 0x7f4fa013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa00a3b40 con 0x7f4fa00a67c0 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa00a8130 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa006ede0 con 0x7f4fa013f5c0 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa6b5a640 10 monclient(hunting): authenticate will time out at 1248.152215s 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4f9ffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f4fa00a6ba0 0x7f4fa00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:40184/0 (socket says 172.21.15.1:40184) 2024-08-19T09:59:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4fa48f6640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f4fa00a67c0 legacy=0x7f4fa00a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.076+0000 7f4f9ffff640 1 -- 172.21.15.1:0/722511389 learned_addr learned my addr 172.21.15.1:0/722511389 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9ffff640 1 -- 172.21.15.1:0/722511389 >> v1:172.21.15.1:6792/0 conn(0x7f4fa00a6ba0 legacy=0x7f4fa00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1915818423 0 0) 0x7f4fa00a8130 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient(hunting): my global_id is 174309 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4fa50f7640 1 -- 172.21.15.1:0/722511389 >> v1:172.21.15.82:6792/0 conn(0x7f4fa013f5c0 legacy=0x7f4fa01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f8c003520 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1291317259 0 0) 0x7f4f8c003520 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 >> v1:172.21.15.82:6792/0 conn(0x7f4fa013f5c0 legacy=0x7f4fa01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 >> v1:172.21.15.82:6790/0 conn(0x7f4fa00a67c0 legacy=0x7f4fa00a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 monclient: found mon.noname-d 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa00a92a0 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f90001e60 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4f9f7fe640 10 monclient: dump: 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4fa6b5a640 5 monclient: authenticate success, global_id 174309 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.078+0000 7f4fa6b5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f90004c20 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4f9f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4f9f7fe640 1 -- 172.21.15.1:0/722511389 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f90005190 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 4 set_mon_vals no callback set 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4f9f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4f9f7fe640 10 monclient: dump: 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.090 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 10 monclient: shutdown 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/722511389 >> v1:172.21.15.1:6792/0 conn(0x7f4fa00a6ba0 legacy=0x7f4fa00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/722511389 shutdown_connections 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/722511389 >> 172.21.15.1:0/722511389 conn(0x7f4fa00ac4b0 msgr2=0x7f4fa00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/722511389 shutdown_connections 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.079+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/722511389 wait complete. 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 10 monclient: monmap: 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 1 librados: starting msgr at 2024-08-19T09:59:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 1 librados: starting objecter 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 1 Processor -- start 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 1 -- start start 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 1 librados: setting wanted keys 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 1 librados: calling monclient init 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.080+0000 7f4fa6b5a640 10 monclient: init 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient: _add_conns ranks=[1,3,5,6,2,0,7,8,4] 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient(hunting): picked mon.noname-f con 0x7f4fa00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient(hunting): picked mon.noname-g con 0x7f4fa00a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient(hunting): picked mon.noname-h con 0x7f4fa013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa013b550 con 0x7f4fa00a67c0 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa013c910 con 0x7f4fa00a6ba0 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa013a7b0 con 0x7f4fa013f5c0 2024-08-19T09:59:52.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa6b5a640 10 monclient(hunting): authenticate will time out at 1248.157253s 2024-08-19T09:59:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4f9ffff640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f4fa00a6ba0 legacy=0x7f4fa013a0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa48f6640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f4fa00a67c0 0x7f4fa0139990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:50262/0 (socket says 172.21.15.1:50262) 2024-08-19T09:59:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.081+0000 7f4fa48f6640 1 -- 172.21.15.1:0/2880407059 learned_addr learned my addr 172.21.15.1:0/2880407059 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.083+0000 7f4fa48f6640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.82:6789/0 conn(0x7f4fa00a67c0 legacy=0x7f4fa0139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.083+0000 7f4f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f4fa00a67c0 2024-08-19T09:59:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.083+0000 7f4f9d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.083+0000 7f4fa50f7640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.82:6791/0 conn(0x7f4fa013f5c0 legacy=0x7f4fa01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.083+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f4fa013f5c0 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641264784 0 0) 0x7f4fa013b550 con 0x7f4fa00a67c0 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): my global_id is 174259 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f80004eb0 con 0x7f4fa00a67c0 2024-08-19T09:59:52.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4192581325 0 0) 0x7f4fa013a7b0 con 0x7f4fa013f5c0 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 10 monclient(hunting): my global_id is 27302 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.085+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fa013b550 con 0x7f4fa013f5c0 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3673552676 0 0) 0x7f4f80004eb0 con 0x7f4fa00a67c0 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4fa013a7b0 con 0x7f4fa00a67c0 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f980046e0 con 0x7f4fa00a67c0 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T09:59:52.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): dump: 2024-08-19T09:59:52.106 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.107 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1982242160 0 0) 0x7f4fa013b550 con 0x7f4fa013f5c0 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f80004eb0 con 0x7f4fa013f5c0 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f88001c80 con 0x7f4fa013f5c0 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient(hunting): dump: 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3088845655 0 0) 0x7f4fa013a7b0 con 0x7f4fa00a67c0 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.82:6791/0 conn(0x7f4fa013f5c0 legacy=0x7f4fa01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.82:6790/0 conn(0x7f4fa00a6ba0 legacy=0x7f4fa013a0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 monclient: found mon.f 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa01429d0 con 0x7f4fa00a67c0 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 5 monclient: authenticate success, global_id 174259 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 10 monclient: _renew_subs 2024-08-19T09:59:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa013b290 con 0x7f4fa00a67c0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 10 client.174259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 10 monclient: _renew_subs 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4fa013c630 con 0x7f4fa00a67c0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4fa6b5a640 1 librados: init done 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f98004900 con 0x7f4fa00a67c0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f98004e30 con 0x7f4fa00a67c0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9effd640 4 set_mon_vals no callback set 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.086+0000 7f4f9d7fa640 10 monclient: dump: 2024-08-19T09:59:52.111 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.112 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4fa6b5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4f840065c0 con 0x7f4f84002860 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4f98005100 con 0x7f4fa00a67c0 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(366..366 src has 299..366) ==== 5449+0+0 (unknown 1073799134 0 0) 0x7f4f9804e340 con 0x7f4fa00a67c0 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4f9d7fa640 10 client.174259.objecter ms_dispatch 0x7f4fa0000b90 osd_map(366..366 src has 299..366) 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4f9d7fa640 3 client.174259.objecter handle_osd_map got epochs [366,366] > 0 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4f9d7fa640 3 client.174259.objecter handle_osd_map decoding full epoch 366 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4f9d7fa640 20 client.174259.objecter dump_active .. 0 homeless 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.087+0000 7f4fa50f7640 10 monclient: get_auth_request con 0x7f4f800413f0 auth_method 0 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9ffff640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.1:6792/0 conn(0x7f4f84002860 legacy=0x7f4f84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 client.174259.objecter ms_handle_connect 0x7f4f84002860 2024-08-19T09:59:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 client.174259.objecter resend_mon_ops 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 91203553 0 0) 0x7f4f840065c0 con 0x7f4f84002860 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient(hunting): my global_id is 174318 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fa013a7b0 con 0x7f4f84002860 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 595463370 0 0) 0x7f4fa013a7b0 con 0x7f4f84002860 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f840065c0 con 0x7f4f84002860 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f900016a0 con 0x7f4f84002860 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.089+0000 7f4f9d7fa640 10 monclient: dump: 2024-08-19T09:59:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.115 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2694165044 0 0) 0x7f4f840065c0 con 0x7f4f84002860 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f8400a160 con 0x7f4f84002860 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4f90004d90 con 0x7f4f84002860 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:52.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.1:6792/0 conn(0x7f4f84002860 legacy=0x7f4f84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.1:6792/0 conn(0x7f4f84002860 legacy=0x7f4f84004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.090+0000 7f4f9d7fa640 10 client.174259.objecter ms_handle_connect 0x7f4f800413f0 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.096+0000 7f4fa6b5a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.096+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4f84009e50 con 0x7f4f84009110 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.098+0000 7f4f9d7fa640 10 client.174259.objecter ms_handle_connect 0x7f4f84009110 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.098+0000 7f4f9d7fa640 10 client.174259.objecter resend_mon_ops 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.098+0000 7f4f9ffff640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.1:6792/0 conn(0x7f4f84009110 legacy=0x7f4f840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820707056 0 0) 0x7f4f84009e50 con 0x7f4f84009110 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient(hunting): my global_id is 174327 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f840065c0 con 0x7f4f84009110 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 679206459 0 0) 0x7f4f840065c0 con 0x7f4f84009110 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f84009e50 con 0x7f4f84009110 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4f90005810 con 0x7f4f84009110 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient: dump: 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.118 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3766264804 0 0) 0x7f4f84009e50 con 0x7f4f84009110 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f8400c3f0 con 0x7f4f84009110 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7483 (unknown 0 0 849631851) 0x7f4f90005a30 con 0x7f4f84009110 2024-08-19T09:59:52.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.1:6792/0 conn(0x7f4f84009110 legacy=0x7f4f840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4f9d7fa640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.1:6792/0 conn(0x7f4f84009110 legacy=0x7f4f840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:52.120 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 42133, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:52.121 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.122 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.123 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:52.124 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.125 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:52.126 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:52.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.133 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:52.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.141 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.141 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:52.141 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4fa6b5a640 10 librados: watch_flush enter 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4fa6b5a640 10 librados: watch_flush exit 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4fa6b5a640 20 client.174259.objecter shutdown clearing up homeless session... 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.099+0000 7f4fa6b5a640 10 client.174259.objecter successfully canceled tick 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4f800413f0 legacy=0x7f4f800438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 10 monclient: shutdown 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 >> v1:172.21.15.82:6789/0 conn(0x7f4fa00a67c0 legacy=0x7f4fa0139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa50f7640 1 -- 172.21.15.1:0/2880407059 reap_dead start 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa50f7640 1 -- 172.21.15.1:0/2880407059 reap_dead start 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 shutdown_connections 2024-08-19T09:59:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 >> 172.21.15.1:0/2880407059 conn(0x7f4fa00ac4b0 msgr2=0x7f4fa00ab170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 shutdown_connections 2024-08-19T09:59:52.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 -- 172.21.15.1:0/2880407059 wait complete. 2024-08-19T09:59:52.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.100+0000 7f4fa6b5a640 1 librados: shutdown 2024-08-19T09:59:52.144 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:59:52.144 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-08-19T09:59:52.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T09:59:52.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.242+0000 7f13da341640 10 monclient: get_monmap_and_config 2024-08-19T09:59:52.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.242+0000 7f13da341640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.242+0000 7f13da341640 10 monclient: monmap: 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:52.243547+0000 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:52.243547+0000 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.245 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.245 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 1 Processor -- start 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 1 -- start start 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient: init 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient: _add_conns ranks=[7,0,5,8,1,4,6,2,3] 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient(hunting): picked mon.noname-i con 0x7f13d40a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient(hunting): picked mon.noname-a con 0x7f13d40a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient(hunting): picked mon.noname-h con 0x7f13d413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d40a3b40 con 0x7f13d40a6ba0 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d40a8130 con 0x7f13d413f5c0 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d406ede0 con 0x7f13d40a67c0 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.243+0000 7f13da341640 10 monclient(hunting): authenticate will time out at 1248.319988s 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.244+0000 7f13d37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f13d40a6ba0 0x7f13d40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56800/0 (socket says 172.21.15.1:56800) 2024-08-19T09:59:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.244+0000 7f13d37fe640 1 -- 172.21.15.1:0/1816169554 learned_addr learned my addr 172.21.15.1:0/1816169554 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d37fe640 1 -- 172.21.15.1:0/1816169554 >> v1:172.21.15.1:6789/0 conn(0x7f13d40a6ba0 legacy=0x7f13d40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2274911685 0 0) 0x7f13d40a3b40 con 0x7f13d40a6ba0 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient(hunting): my global_id is 174330 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13bc003520 con 0x7f13d40a6ba0 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d3fff640 1 -- 172.21.15.1:0/1816169554 >> v1:172.21.15.82:6792/0 conn(0x7f13d40a67c0 legacy=0x7f13d40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d88de640 1 -- 172.21.15.1:0/1816169554 >> v1:172.21.15.82:6791/0 conn(0x7f13d413f5c0 legacy=0x7f13d41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2738564913 0 0) 0x7f13bc003520 con 0x7f13d40a6ba0 2024-08-19T09:59:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 >> v1:172.21.15.82:6792/0 conn(0x7f13d40a67c0 legacy=0x7f13d40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 >> v1:172.21.15.82:6791/0 conn(0x7f13d413f5c0 legacy=0x7f13d41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 monclient: found mon.noname-a 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d40a92a0 con 0x7f13d40a6ba0 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13c8001ae0 con 0x7f13d40a6ba0 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13da341640 5 monclient: authenticate success, global_id 174330 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13da341640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: dump: 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13da341640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c80048a0 con 0x7f13d40a6ba0 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 1 -- 172.21.15.1:0/1816169554 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13c8004e50 con 0x7f13d40a6ba0 2024-08-19T09:59:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13d2ffd640 10 monclient: dump: 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13da341640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.246+0000 7f13da341640 4 set_mon_vals no callback set 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 10 monclient: shutdown 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 1 -- 172.21.15.1:0/1816169554 >> v1:172.21.15.1:6789/0 conn(0x7f13d40a6ba0 legacy=0x7f13d40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 1 -- 172.21.15.1:0/1816169554 shutdown_connections 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 1 -- 172.21.15.1:0/1816169554 >> 172.21.15.1:0/1816169554 conn(0x7f13d40ac4b0 msgr2=0x7f13d40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 1 -- 172.21.15.1:0/1816169554 shutdown_connections 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 1 -- 172.21.15.1:0/1816169554 wait complete. 2024-08-19T09:59:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.247+0000 7f13da341640 10 monclient: monmap: 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 librados: starting msgr at 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 librados: starting objecter 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 Processor -- start 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 -- start start 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 librados: setting wanted keys 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 librados: calling monclient init 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient: init 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient: _add_conns ranks=[4,3,0,1,2,6,5,8,7] 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient(hunting): picked mon.noname-c con 0x7f13d40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:52.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient(hunting): picked mon.noname-g con 0x7f13d40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient(hunting): picked mon.noname-a con 0x7f13d413f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d40a9fe0 con 0x7f13d413f5c0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d413c030 con 0x7f13d40a6ba0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d413df40 con 0x7f13d40a67c0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13da341640 10 monclient(hunting): authenticate will time out at 1248.324880s 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13d88de640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f13d413f5c0 0x7f13d41d6670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56810/0 (socket says 172.21.15.1:56810) 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13d88de640 1 -- 172.21.15.1:0/1025096739 learned_addr learned my addr 172.21.15.1:0/1025096739 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.248+0000 7f13d37fe640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6790/0 conn(0x7f13d40a6ba0 legacy=0x7f13d413d460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d3fff640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.1:6791/0 conn(0x7f13d40a67c0 legacy=0x7f13d4139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 client.?.objecter ms_handle_connect 0x7f13d40a67c0 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d88de640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.1:6789/0 conn(0x7f13d413f5c0 legacy=0x7f13d41d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 client.?.objecter ms_handle_connect 0x7f13d413f5c0 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539029401 0 0) 0x7f13d413df40 con 0x7f13d40a67c0 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 27670 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c0004eb0 con 0x7f13d40a67c0 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1605806488 0 0) 0x7f13d40a9fe0 con 0x7f13d413f5c0 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 174339 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d413df40 con 0x7f13d413f5c0 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1773683381 0 0) 0x7f13c0004eb0 con 0x7f13d40a67c0 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13d40a9fe0 con 0x7f13d40a67c0 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13c4001d20 con 0x7f13d40a67c0 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): dump: 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.265 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2137007002 0 0) 0x7f13d413df40 con 0x7f13d413f5c0 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13c0004eb0 con 0x7f13d413f5c0 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13cc002050 con 0x7f13d413f5c0 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient(hunting): dump: 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1904764568 0 0) 0x7f13d40a9fe0 con 0x7f13d40a67c0 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6790/0 conn(0x7f13d40a6ba0 legacy=0x7f13d413d460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.1:6789/0 conn(0x7f13d413f5c0 legacy=0x7f13d41d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 monclient: found mon.c 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d413b790 con 0x7f13d40a67c0 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 5 monclient: authenticate success, global_id 27670 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 10 monclient: _renew_subs 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13d413a830 con 0x7f13d40a67c0 2024-08-19T09:59:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 10 client.27670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 10 monclient: _renew_subs 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f13d4139910 con 0x7f13d40a67c0 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13da341640 1 librados: init done 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c4003a90 con 0x7f13d40a67c0 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13c4003fc0 con 0x7f13d40a67c0 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d27fc640 4 set_mon_vals no callback set 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.251+0000 7f13d0ff9640 10 monclient: dump: 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.271 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.252+0000 7f13da341640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.252+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f13b40065c0 con 0x7f13b4002860 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.252+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f13c4004160 con 0x7f13d40a67c0 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.253+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(366..366 src has 299..366) ==== 5449+0+0 (unknown 1073799134 0 0) 0x7f13c404e200 con 0x7f13d40a67c0 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.253+0000 7f13d0ff9640 10 client.27670.objecter ms_dispatch 0x7f13d4000b90 osd_map(366..366 src has 299..366) 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.253+0000 7f13d0ff9640 3 client.27670.objecter handle_osd_map got epochs [366,366] > 0 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.253+0000 7f13d0ff9640 3 client.27670.objecter handle_osd_map decoding full epoch 366 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.253+0000 7f13d0ff9640 20 client.27670.objecter dump_active .. 0 homeless 2024-08-19T09:59:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.253+0000 7f13d88de640 10 monclient: get_auth_request con 0x7f13c00414f0 auth_method 0 2024-08-19T09:59:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 client.27670.objecter ms_handle_connect 0x7f13b4002860 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d37fe640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6789/0 conn(0x7f13b4002860 legacy=0x7f13b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 client.27670.objecter resend_mon_ops 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3017797102 0 0) 0x7f13b40065c0 con 0x7f13b4002860 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 174268 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d40a9fe0 con 0x7f13b4002860 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 client.27670.objecter ms_handle_connect 0x7f13c00414f0 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2614436356 0 0) 0x7f13d40a9fe0 con 0x7f13b4002860 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13b40065c0 con 0x7f13b4002860 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13c8004e50 con 0x7f13b4002860 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 10 monclient: dump: 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.277 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 892313716 0 0) 0x7f13b40065c0 con 0x7f13b4002860 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.255+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13b400a160 con 0x7f13b4002860 2024-08-19T09:59:52.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.256+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f13c8004e50 con 0x7f13b4002860 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.256+0000 7f13d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.256+0000 7f13d0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.256+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6789/0 conn(0x7f13b4002860 legacy=0x7f13b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.256+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6789/0 conn(0x7f13b4002860 legacy=0x7f13b4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.262+0000 7f13da341640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.262+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f13b4009e50 con 0x7f13b4009110 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.264+0000 7f13d37fe640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6789/0 conn(0x7f13b4009110 legacy=0x7f13b40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.264+0000 7f13d0ff9640 10 client.27670.objecter ms_handle_connect 0x7f13b4009110 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.264+0000 7f13d0ff9640 10 client.27670.objecter resend_mon_ops 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164410568 0 0) 0x7f13b4009e50 con 0x7f13b4009110 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 174277 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13b40065c0 con 0x7f13b4009110 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2890227173 0 0) 0x7f13b40065c0 con 0x7f13b4009110 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13b4009e50 con 0x7f13b4009110 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f13c8005450 con 0x7f13b4009110 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient: dump: 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 471615160 0 0) 0x7f13b4009e50 con 0x7f13b4009110 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13b400c3f0 con 0x7f13b4009110 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7817 (unknown 0 0 1308133503) 0x7f13c80085c0 con 0x7f13b4009110 2024-08-19T09:59:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6789/0 conn(0x7f13b4009110 legacy=0x7f13b40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.265+0000 7f13d0ff9640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6789/0 conn(0x7f13b4009110 legacy=0x7f13b40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-08-19T09:59:52.282 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 42324, 2024-08-19T09:59:52.283 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:52.283 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:52.283 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:52.283 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:52.283 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:52.283 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.284 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.285 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:52.286 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.287 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.288 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.289 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:52.290 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.291 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.292 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:52.293 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.294 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:52.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:52.300 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.301 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.302 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.303 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:52.304 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 10 librados: watch_flush enter 2024-08-19T09:59:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 10 librados: watch_flush exit 2024-08-19T09:59:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 20 client.27670.objecter shutdown clearing up homeless session... 2024-08-19T09:59:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 10 client.27670.objecter successfully canceled tick 2024-08-19T09:59:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.82:6812/2549104415 conn(0x7f13c00414f0 legacy=0x7f13c00439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 10 monclient: shutdown 2024-08-19T09:59:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 >> v1:172.21.15.1:6791/0 conn(0x7f13d40a67c0 legacy=0x7f13d4139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13d88de640 1 -- 172.21.15.1:0/1025096739 reap_dead start 2024-08-19T09:59:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13d88de640 1 -- 172.21.15.1:0/1025096739 reap_dead start 2024-08-19T09:59:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 shutdown_connections 2024-08-19T09:59:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 >> 172.21.15.1:0/1025096739 conn(0x7f13d40ac4b0 msgr2=0x7f13d4141cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 shutdown_connections 2024-08-19T09:59:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 -- 172.21.15.1:0/1025096739 wait complete. 2024-08-19T09:59:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.266+0000 7f13da341640 1 librados: shutdown 2024-08-19T09:59:52.312 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:59:52.312 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon h 2024-08-19T09:59:52.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T09:59:52.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.411+0000 7fddf13e2640 10 monclient: get_monmap_and_config 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.411+0000 7fddf13e2640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.411+0000 7fddf13e2640 10 monclient: monmap: 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:52.412845+0000 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:52.412845+0000 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.413 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.414 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.412+0000 7fddf13e2640 1 Processor -- start 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.412+0000 7fddf13e2640 1 -- start start 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.412+0000 7fddf13e2640 10 monclient: init 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient: _add_conns ranks=[7,3,1,4,8,5,6,0,2] 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient(hunting): picked mon.noname-i con 0x7fddec0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient(hunting): picked mon.noname-g con 0x7fddec0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:52.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient(hunting): picked mon.noname-f con 0x7fddec13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:52.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec0a3b40 con 0x7fddec13f5c0 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec0a8130 con 0x7fddec0a6ba0 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec06ede0 con 0x7fddec0a67c0 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddf13e2640 10 monclient(hunting): authenticate will time out at 1248.489158s 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddea7fc640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fddec0a6ba0 legacy=0x7fddec0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddeb7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fddec13f5c0 0x7fddec1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:50282/0 (socket says 172.21.15.1:50282) 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddeaffd640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7fddec0a67c0 0x7fddec0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:39416/0 (socket says 172.21.15.1:39416) 2024-08-19T09:59:52.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.413+0000 7fddeb7fe640 1 -- 172.21.15.1:0/2528767440 learned_addr learned my addr 172.21.15.1:0/2528767440 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fddeb7fe640 1 -- 172.21.15.1:0/2528767440 >> v1:172.21.15.82:6789/0 conn(0x7fddec13f5c0 legacy=0x7fddec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fddeaffd640 1 -- 172.21.15.1:0/2528767440 >> v1:172.21.15.82:6792/0 conn(0x7fddec0a67c0 legacy=0x7fddec0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810232330 0 0) 0x7fddec06ede0 con 0x7fddec0a67c0 2024-08-19T09:59:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): my global_id is 144790 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddd8003520 con 0x7fddec0a67c0 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592231754 0 0) 0x7fddec0a3b40 con 0x7fddec13f5c0 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 10 monclient(hunting): my global_id is 174286 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.415+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddec06ede0 con 0x7fddec13f5c0 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3452455184 0 0) 0x7fddd8003520 con 0x7fddec0a67c0 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 >> v1:172.21.15.82:6790/0 conn(0x7fddec0a6ba0 legacy=0x7fddec0a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 >> v1:172.21.15.82:6789/0 conn(0x7fddec13f5c0 legacy=0x7fddec1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 monclient: found mon.noname-i 2024-08-19T09:59:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec0a92a0 con 0x7fddec0a67c0 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 5 monclient: authenticate success, global_id 144790 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdddc001e50 con 0x7fddec0a67c0 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: dump: 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdddc004b60 con 0x7fddec0a67c0 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 4 set_mon_vals no callback set 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 1 -- 172.21.15.1:0/2528767440 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdddc0050d0 con 0x7fddec0a67c0 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fdde9ffb640 10 monclient: dump: 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 10 monclient: shutdown 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 1 -- 172.21.15.1:0/2528767440 >> v1:172.21.15.82:6792/0 conn(0x7fddec0a67c0 legacy=0x7fddec0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 1 -- 172.21.15.1:0/2528767440 shutdown_connections 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 1 -- 172.21.15.1:0/2528767440 >> 172.21.15.1:0/2528767440 conn(0x7fddec0ac4b0 msgr2=0x7fddec0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 1 -- 172.21.15.1:0/2528767440 shutdown_connections 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.416+0000 7fddf13e2640 1 -- 172.21.15.1:0/2528767440 wait complete. 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 10 monclient: monmap: 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.430 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 1 librados: starting msgr at 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 1 librados: starting objecter 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 1 Processor -- start 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 1 -- start start 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 1 librados: setting wanted keys 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 1 librados: calling monclient init 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.417+0000 7fddf13e2640 10 monclient: init 2024-08-19T09:59:52.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient: _add_conns ranks=[3,4,8,1,6,5,0,2,7] 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient(hunting): picked mon.noname-g con 0x7fddec0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient(hunting): picked mon.noname-c con 0x7fddec0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient(hunting): picked mon.noname-e con 0x7fddec13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec1429d0 con 0x7fddec0a67c0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec13a850 con 0x7fddec0a6ba0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec139fd0 con 0x7fddec13f5c0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddf13e2640 10 monclient(hunting): authenticate will time out at 1248.494438s 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddeb7fe640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7fddec13f5c0 legacy=0x7fddec1d62a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddeaffd640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fddec0a67c0 legacy=0x7fddec09ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddea7fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fddec0a6ba0 0x7fddec0a0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50770/0 (socket says 172.21.15.1:50770) 2024-08-19T09:59:52.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.418+0000 7fddea7fc640 1 -- 172.21.15.1:0/2652331431 learned_addr learned my addr 172.21.15.1:0/2652331431 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fddea7fc640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.1:6791/0 conn(0x7fddec0a6ba0 legacy=0x7fddec0a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 10 client.?.objecter ms_handle_connect 0x7fddec0a6ba0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1038996107 0 0) 0x7fddec13a850 con 0x7fddec0a6ba0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 10 monclient(hunting): my global_id is 27679 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.420+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddcc004eb0 con 0x7fddec0a6ba0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1347728516 0 0) 0x7fddcc004eb0 con 0x7fddec0a6ba0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddec13a850 con 0x7fddec0a6ba0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fddd4004730 con 0x7fddec0a6ba0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient(hunting): dump: 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.434 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1893393531 0 0) 0x7fddec13a850 con 0x7fddec0a6ba0 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.1:6793/0 conn(0x7fddec13f5c0 legacy=0x7fddec1d62a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6790/0 conn(0x7fddec0a67c0 legacy=0x7fddec09ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 monclient: found mon.c 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec0a1780 con 0x7fddec0a6ba0 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 5 monclient: authenticate success, global_id 27679 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 10 monclient: _renew_subs 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fddec13a570 con 0x7fddec0a6ba0 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 10 client.27679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 10 monclient: _renew_subs 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fddec0a0da0 con 0x7fddec0a6ba0 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fddf13e2640 1 librados: init done 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fddd4004990 con 0x7fddec0a6ba0 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fddd4004ec0 con 0x7fddec0a6ba0 2024-08-19T09:59:52.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde97fa640 4 set_mon_vals no callback set 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.421+0000 7fdde37fe640 10 monclient: dump: 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.437 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fddf13e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fddd00065c0 con 0x7fddd0002860 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fddd40051d0 con 0x7fddec0a6ba0 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(367..367 src has 299..367) ==== 5449+0+0 (unknown 1296131761 0 0) 0x7fddd404e380 con 0x7fddec0a6ba0 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fdde37fe640 10 client.27679.objecter ms_dispatch 0x7fddec000b90 osd_map(367..367 src has 299..367) 2024-08-19T09:59:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fdde37fe640 3 client.27679.objecter handle_osd_map got epochs [367,367] > 0 2024-08-19T09:59:52.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fdde37fe640 3 client.27679.objecter handle_osd_map decoding full epoch 367 2024-08-19T09:59:52.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fdde37fe640 20 client.27679.objecter dump_active .. 0 homeless 2024-08-19T09:59:52.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.422+0000 7fddeb7fe640 10 monclient: get_auth_request con 0x7fddcc040da0 auth_method 0 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fddeaffd640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6791/0 conn(0x7fddd0002860 legacy=0x7fddd0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 client.27679.objecter ms_handle_connect 0x7fddd0002860 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 client.27679.objecter resend_mon_ops 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568753183 0 0) 0x7fddd00065c0 con 0x7fddd0002860 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 monclient(hunting): my global_id is 27320 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddec13a850 con 0x7fddd0002860 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 10 client.27679.objecter ms_handle_connect 0x7fddcc040da0 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 617744700 0 0) 0x7fddec13a850 con 0x7fddd0002860 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.424+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddd00065c0 con 0x7fddd0002860 2024-08-19T09:59:52.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdddc001c50 con 0x7fddd0002860 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 10 monclient: dump: 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 776223885 0 0) 0x7fddd00065c0 con 0x7fddd0002860 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fddd000a160 con 0x7fddd0002860 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdddc004b60 con 0x7fddd0002860 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6791/0 conn(0x7fddd0002860 legacy=0x7fddd0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6791/0 conn(0x7fddd0002860 legacy=0x7fddd0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.431+0000 7fddf13e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.431+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fddd0009e70 con 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fddeaffd640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6791/0 conn(0x7fddd0009130 legacy=0x7fddd00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 10 client.27679.objecter ms_handle_connect 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 10 client.27679.objecter resend_mon_ops 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181659749 0 0) 0x7fddd0009e70 con 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 10 monclient(hunting): my global_id is 27329 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.434+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddd00065c0 con 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 844965935 0 0) 0x7fddd00065c0 con 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddd0009e70 con 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdddc0056d0 con 0x7fddd0009130 2024-08-19T09:59:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 10 monclient: dump: 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3039450670 0 0) 0x7fddd0009e70 con 0x7fddd0009130 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fddd000c410 con 0x7fddd0009130 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7651 (unknown 0 0 3747251396) 0x7fdddc008840 con 0x7fddd0009130 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6791/0 conn(0x7fddd0009130 legacy=0x7fddd00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.435+0000 7fdde37fe640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6791/0 conn(0x7fddd0009130 legacy=0x7fddd00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 598701, 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:52.467 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 10 librados: watch_flush enter 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 10 librados: watch_flush exit 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 20 client.27679.objecter shutdown clearing up homeless session... 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 10 client.27679.objecter successfully canceled tick 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.82:6812/2549104415 conn(0x7fddcc040da0 legacy=0x7fddcc043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 10 monclient: shutdown 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 >> v1:172.21.15.1:6791/0 conn(0x7fddec0a6ba0 legacy=0x7fddec0a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddeb7fe640 1 -- 172.21.15.1:0/2652331431 reap_dead start 2024-08-19T09:59:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddeb7fe640 1 -- 172.21.15.1:0/2652331431 reap_dead start 2024-08-19T09:59:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 shutdown_connections 2024-08-19T09:59:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 >> 172.21.15.1:0/2652331431 conn(0x7fddec0ac4b0 msgr2=0x7fddec0a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 shutdown_connections 2024-08-19T09:59:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 -- 172.21.15.1:0/2652331431 wait complete. 2024-08-19T09:59:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.436+0000 7fddf13e2640 1 librados: shutdown 2024-08-19T09:59:52.470 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:59:52.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon i 2024-08-19T09:59:52.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T09:59:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.569+0000 7f7153d24640 10 monclient: get_monmap_and_config 2024-08-19T09:59:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.569+0000 7f7153d24640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.569+0000 7f7153d24640 10 monclient: monmap: 2024-08-19T09:59:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:59:52.570427+0000 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:59:52.570427+0000 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.572 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 1 Processor -- start 2024-08-19T09:59:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 1 -- start start 2024-08-19T09:59:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient: init 2024-08-19T09:59:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient: _add_conns ranks=[1,6,3,4,8,2,7,0,5] 2024-08-19T09:59:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient(hunting): picked mon.noname-f con 0x7f714c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient(hunting): picked mon.noname-d con 0x7f714c0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient(hunting): picked mon.noname-g con 0x7f714c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714c0a3b40 con 0x7f714c0a67c0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714c0a8130 con 0x7f714c13f5c0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714c06ede0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f7153d24640 10 monclient(hunting): authenticate will time out at 1248.646923s 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f71512bf640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f714c0a6ba0 0x7f714c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:40216/0 (socket says 172.21.15.1:40216) 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f71522c1640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f714c13f5c0 legacy=0x7f714c1419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.570+0000 7f71512bf640 1 -- 172.21.15.1:0/1860382320 learned_addr learned my addr 172.21.15.1:0/1860382320 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.571+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 DispatchQueue::entry inject delay of 0.002000 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f71512bf640 1 -- 172.21.15.1:0/1860382320 >> v1:172.21.15.1:6792/0 conn(0x7f714c0a6ba0 legacy=0x7f714c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727563356 0 0) 0x7f714c06ede0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient(hunting): my global_id is 174336 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7138003520 con 0x7f714c0a6ba0 2024-08-19T09:59:52.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7151ac0640 1 -- 172.21.15.1:0/1860382320 >> v1:172.21.15.82:6789/0 conn(0x7f714c0a67c0 legacy=0x7f714c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3619825793 0 0) 0x7f7138003520 con 0x7f714c0a6ba0 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 >> v1:172.21.15.82:6790/0 conn(0x7f714c13f5c0 legacy=0x7f714c1419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 >> v1:172.21.15.82:6789/0 conn(0x7f714c0a67c0 legacy=0x7f714c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 monclient: found mon.noname-d 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714c0a92a0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7153d24640 5 monclient: authenticate success, global_id 174336 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f713c001ad0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7153d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: dump: 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7153d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f713c0048a0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 1 -- 172.21.15.1:0/1860382320 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f713c004e10 con 0x7f714c0a6ba0 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7153d24640 10 monclient: get_monmap_and_config success 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7153d24640 4 set_mon_vals no callback set 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T09:59:52.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.573+0000 7f7150abe640 10 monclient: dump: 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.582 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 10 monclient: shutdown 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 -- 172.21.15.1:0/1860382320 >> v1:172.21.15.1:6792/0 conn(0x7f714c0a6ba0 legacy=0x7f714c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 -- 172.21.15.1:0/1860382320 shutdown_connections 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 -- 172.21.15.1:0/1860382320 >> 172.21.15.1:0/1860382320 conn(0x7f714c0ac4b0 msgr2=0x7f714c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 -- 172.21.15.1:0/1860382320 shutdown_connections 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 -- 172.21.15.1:0/1860382320 wait complete. 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 10 monclient: build_initial_monmap 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 10 monclient: monmap: 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T09:59:52.583 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 librados: starting msgr at 2024-08-19T09:59:52.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.574+0000 7f7153d24640 1 librados: starting objecter 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 Processor -- start 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 -- start start 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 librados: setting wanted keys 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 librados: calling monclient init 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient: init 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient: _reopen_session rank -1 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient: _add_conns ranks=[8,4,1,5,7,0,2,6,3] 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient(hunting): picked mon.noname-e con 0x7f714c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient(hunting): picked mon.noname-c con 0x7f714c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient(hunting): picked mon.noname-f con 0x7f714c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714c0a15d0 con 0x7f714c13f5c0 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714c13a990 con 0x7f714c0a6ba0 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714c13a0f0 con 0x7f714c0a67c0 2024-08-19T09:59:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient(hunting): _renew_subs 2024-08-19T09:59:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7151ac0640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f714c0a67c0 legacy=0x7f714c09fde0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T09:59:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f7153d24640 10 monclient(hunting): authenticate will time out at 1248.651623s 2024-08-19T09:59:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f71512bf640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f714c0a6ba0 0x7f714c0a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50774/0 (socket says 172.21.15.1:50774) 2024-08-19T09:59:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.575+0000 7f71512bf640 1 -- 172.21.15.1:0/2191090783 learned_addr learned my addr 172.21.15.1:0/2191090783 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T09:59:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.577+0000 7f71512bf640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.1:6791/0 conn(0x7f714c0a6ba0 legacy=0x7f714c0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.577+0000 7f71427fc640 10 client.?.objecter ms_handle_connect 0x7f714c0a6ba0 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.577+0000 7f71427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71522c1640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6789/0 conn(0x7f714c13f5c0 legacy=0x7f714c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 client.?.objecter ms_handle_connect 0x7f714c13f5c0 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176495896 0 0) 0x7f714c0a15d0 con 0x7f714c13f5c0 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): my global_id is 174304 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7130004eb0 con 0x7f714c13f5c0 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117546245 0 0) 0x7f714c13a990 con 0x7f714c0a6ba0 2024-08-19T09:59:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): my global_id is 27688 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714c0a15d0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 964424116 0 0) 0x7f714c0a15d0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f714c13a990 con 0x7f714c0a6ba0 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f713c004d90 con 0x7f714c0a6ba0 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): dump: 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 214934801 0 0) 0x7f7130004eb0 con 0x7f714c13f5c0 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f714c0a15d0 con 0x7f714c13f5c0 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71440047c0 con 0x7f714c13f5c0 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient(hunting): dump: 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 866327835 0 0) 0x7f714c13a990 con 0x7f714c0a6ba0 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.1:6793/0 conn(0x7f714c0a67c0 legacy=0x7f714c09fde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6789/0 conn(0x7f714c13f5c0 legacy=0x7f714c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: _finish_hunting 0 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 monclient: found mon.c 2024-08-19T09:59:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714c13f220 con 0x7f714c0a6ba0 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: _finish_auth 0 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: _check_auth_tickets 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 5 monclient: authenticate success, global_id 27688 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 10 monclient: _renew_subs 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f714c139d10 con 0x7f714c0a6ba0 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 10 client.27688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 10 monclient: _renew_subs 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T09:59:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f714c0a0de0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7153d24640 1 librados: init done 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f713c001d20 con 0x7f714c0a6ba0 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: handle_config config(0 keys) 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f713c001af0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f71427fc640 10 monclient: dump: 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.597 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.578+0000 7f7143fff640 4 set_mon_vals no callback set 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.579+0000 7f7153d24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.579+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f714c0008d0 con 0x7f714c13fcc0 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.579+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f713c011a00 con 0x7f714c0a6ba0 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.580+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(367..367 src has 299..367) ==== 5449+0+0 (unknown 1296131761 0 0) 0x7f713c04e1a0 con 0x7f714c0a6ba0 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.580+0000 7f71427fc640 10 client.27688.objecter ms_dispatch 0x7f714c000b90 osd_map(367..367 src has 299..367) 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.580+0000 7f71427fc640 3 client.27688.objecter handle_osd_map got epochs [367,367] > 0 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.580+0000 7f71427fc640 3 client.27688.objecter handle_osd_map decoding full epoch 367 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.580+0000 7f71427fc640 20 client.27688.objecter dump_active .. 0 homeless 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.580+0000 7f71522c1640 10 monclient: get_auth_request con 0x7f7130041560 auth_method 0 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 client.27688.objecter ms_handle_connect 0x7f714c13fcc0 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 client.27688.objecter resend_mon_ops 2024-08-19T09:59:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f7151ac0640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6792/0 conn(0x7f714c13fcc0 legacy=0x7f714c1400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665922487 0 0) 0x7f714c0008d0 con 0x7f714c13fcc0 2024-08-19T09:59:52.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient(hunting): my global_id is 144799 2024-08-19T09:59:52.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714c13a990 con 0x7f714c13fcc0 2024-08-19T09:59:52.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 814376889 0 0) 0x7f714c13a990 con 0x7f714c13fcc0 2024-08-19T09:59:52.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f714c0008d0 con 0x7f714c13fcc0 2024-08-19T09:59:52.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7148001be0 con 0x7f714c13fcc0 2024-08-19T09:59:52.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:52.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 monclient: dump: 2024-08-19T09:59:52.602 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.602 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.602 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.602 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.602 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.603 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 10 client.27688.objecter ms_handle_connect 0x7f7130041560 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 779516361 0 0) 0x7f714c0008d0 con 0x7f714c13fcc0 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.582+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f714c142d30 con 0x7f714c13fcc0 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.583+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f71480057c0 con 0x7f714c13fcc0 2024-08-19T09:59:52.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.583+0000 7f71427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T09:59:52.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.583+0000 7f71427fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T09:59:52.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.583+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6792/0 conn(0x7f714c13fcc0 legacy=0x7f714c1400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.583+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6792/0 conn(0x7f714c13fcc0 legacy=0x7f714c1400a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.589+0000 7f7153d24640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T09:59:52.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.589+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f714c140b90 con 0x7f714c1407b0 2024-08-19T09:59:52.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f7151ac0640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6792/0 conn(0x7f714c1407b0 legacy=0x7f714c1db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T09:59:52.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 client.27688.objecter ms_handle_connect 0x7f714c1407b0 2024-08-19T09:59:52.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 client.27688.objecter resend_mon_ops 2024-08-19T09:59:52.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2282290079 0 0) 0x7f714c140b90 con 0x7f714c1407b0 2024-08-19T09:59:52.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient(hunting): _negotiate 2024-08-19T09:59:52.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T09:59:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T09:59:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient(hunting): my global_id is 144808 2024-08-19T09:59:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714c0008d0 con 0x7f714c1407b0 2024-08-19T09:59:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1552492541 0 0) 0x7f714c0008d0 con 0x7f714c1407b0 2024-08-19T09:59:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f714c140b90 con 0x7f714c1407b0 2024-08-19T09:59:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7148003d80 con 0x7f714c1407b0 2024-08-19T09:59:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T09:59:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T09:59:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.591+0000 7f71427fc640 10 monclient: dump: 2024-08-19T09:59:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T09:59:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T09:59:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.611 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T09:59:52.611 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T09:59:52.611 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T09:59:52.611 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T09:59:52.611 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T09:59:52.612 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T09:59:52.612 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T09:59:52.612 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T09:59:52.612 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T09:59:52.612 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T09:59:52.612 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 473057327 0 0) 0x7f714c140b90 con 0x7f714c1407b0 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f714c13a310 con 0x7f714c1407b0 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7651 (unknown 0 0 4133796155) 0x7f7148004020 con 0x7f714c1407b0 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T09:59:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6792/0 conn(0x7f714c1407b0 legacy=0x7f714c1db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71427fc640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6792/0 conn(0x7f714c1407b0 legacy=0x7f714c1db990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T09:59:52.614 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T09:59:52.614 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:52.614 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:52.614 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 316972, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.616 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.617 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T09:59:52.618 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T09:59:52.619 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T09:59:52.620 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.621 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T09:59:52.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T09:59:52.625 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.625 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.625 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.626 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T09:59:52.626 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T09:59:52.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T09:59:52.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T09:59:52.628 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.629 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.629 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.629 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T09:59:52.629 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T09:59:52.630 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T09:59:52.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.631 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T09:59:52.631 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T09:59:52.632 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T09:59:52.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.634 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T09:59:52.634 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.634 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.638 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T09:59:52.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T09:59:52.640 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T09:59:52.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.642 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T09:59:52.643 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T09:59:52.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T09:59:52.644 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T09:59:52.644 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T09:59:52.644 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T09:59:52.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T09:59:52.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T09:59:52.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T09:59:52.645 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T09:59:52.645 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T09:59:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 10 librados: watch_flush enter 2024-08-19T09:59:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 10 librados: watch_flush exit 2024-08-19T09:59:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 20 client.27688.objecter shutdown clearing up homeless session... 2024-08-19T09:59:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 10 client.27688.objecter successfully canceled tick 2024-08-19T09:59:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7130041560 legacy=0x7f7130043a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 10 monclient: shutdown 2024-08-19T09:59:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T09:59:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 >> v1:172.21.15.1:6791/0 conn(0x7f714c0a6ba0 legacy=0x7f714c0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T09:59:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71522c1640 1 -- 172.21.15.1:0/2191090783 reap_dead start 2024-08-19T09:59:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f71522c1640 1 -- 172.21.15.1:0/2191090783 reap_dead start 2024-08-19T09:59:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 shutdown_connections 2024-08-19T09:59:52.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 >> 172.21.15.1:0/2191090783 conn(0x7f714c0ac4b0 msgr2=0x7f714c0a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T09:59:52.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 shutdown_connections 2024-08-19T09:59:52.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.592+0000 7f7153d24640 1 -- 172.21.15.1:0/2191090783 wait complete. 2024-08-19T09:59:52.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T09:59:52.593+0000 7f7153d24640 1 librados: shutdown 2024-08-19T09:59:52.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T09:59:52.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T10:01:03.517 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T10:01:03.515+0000 7f0f16240640 -1 snap_mapper.add_oid found existing snaps mapped on 300:2ad3294e:test-rados-api-smithi082-25997-80::foo:2, removing 2024-08-19T10:01:03.518 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-08-19T10:01:03.515+0000 7fc350075640 -1 snap_mapper.add_oid found existing snaps mapped on 300:2ad3294e:test-rados-api-smithi082-25997-80::foo:2, removing 2024-08-19T10:01:22.653 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-08-19T10:01:22.653 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T10:01:22.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T10:01:22.657 INFO:tasks.ceph.mon.g:Started 2024-08-19T10:01:22.657 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-19T10:01:22.657 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-19T10:01:22.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-19T10:01:22.660 INFO:tasks.ceph.mon.b:Started 2024-08-19T10:01:22.660 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-08-19T10:01:22.660 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T10:01:22.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T10:01:22.663 INFO:tasks.ceph.mon.e:Started 2024-08-19T10:01:22.663 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-08-19T10:01:22.663 INFO:tasks.ceph.mon.a:Sent signal 19 2024-08-19T10:01:22.663 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-08-19T10:01:22.663 INFO:tasks.ceph.mon.f:Sent signal 19 2024-08-19T10:01:22.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T10:01:37.665 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-08-19T10:01:37.666 INFO:tasks.ceph.mon.a:Sent signal 18 2024-08-19T10:01:37.666 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-08-19T10:01:37.666 INFO:tasks.ceph.mon.f:Sent signal 18 2024-08-19T10:01:37.666 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:01:37.667 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:01:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.781+0000 7f7212e7e640 10 monclient: get_monmap_and_config 2024-08-19T10:01:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.781+0000 7f7212e7e640 10 monclient: build_initial_monmap 2024-08-19T10:01:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.781+0000 7f7212e7e640 10 monclient: monmap: 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:37.782393+0000 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:37.782393+0000 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:37.784 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 1 Processor -- start 2024-08-19T10:01:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 1 -- start start 2024-08-19T10:01:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient: init 2024-08-19T10:01:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient: _add_conns ranks=[5,6,3,7,2,4,0,8,1] 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient(hunting): picked mon.noname-h con 0x7f720c0a6720 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient(hunting): picked mon.noname-d con 0x7f720c0a4590 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient(hunting): picked mon.noname-g con 0x7f720c13f730 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a32c0 con 0x7f720c13f730 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a35a0 con 0x7f720c0a6720 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a3880 con 0x7f720c0a4590 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.782+0000 7f7212e7e640 10 monclient(hunting): authenticate will time out at 1353.858927s 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.783+0000 7f720bfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f720c0a4590 0x7f720c0a7210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:42976/0 (socket says 172.21.15.1:42976) 2024-08-19T10:01:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.783+0000 7f720bfff640 1 -- 172.21.15.1:0/3000082125 learned_addr learned my addr 172.21.15.1:0/3000082125 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:37.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.785+0000 7f720bfff640 1 -- 172.21.15.1:0/3000082125 >> v1:172.21.15.1:6792/0 conn(0x7f720c0a4590 legacy=0x7f720c0a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:37.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.785+0000 7f7210c1a640 1 -- 172.21.15.1:0/3000082125 >> v1:172.21.15.82:6791/0 conn(0x7f720c0a6720 legacy=0x7f720c0a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:37.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:37.785+0000 7f721141b640 1 -- 172.21.15.1:0/3000082125 >> v1:172.21.15.82:6790/0 conn(0x7f720c13f730 legacy=0x7f720c141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:38.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125858059 0 0) 0x7f7204001820 con 0x7f720c13f730 2024-08-19T10:01:38.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:38.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:38.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:38.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): my global_id is 184107 2024-08-19T10:01:38.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71fc003520 con 0x7f720c13f730 2024-08-19T10:01:38.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439599075 0 0) 0x7f720c0a35a0 con 0x7f720c0a6720 2024-08-19T10:01:38.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient(hunting): my global_id is 27338 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7204001820 con 0x7f720c0a6720 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1739863422 0 0) 0x7f71fc003520 con 0x7f720c13f730 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 >> v1:172.21.15.1:6792/0 conn(0x7f720c0a4590 legacy=0x7f720c0a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 >> v1:172.21.15.82:6791/0 conn(0x7f720c0a6720 legacy=0x7f720c0a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:01:38.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 monclient: found mon.noname-g 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c0a92b0 con 0x7f720c13f730 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: _finish_auth 0 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7204002080 con 0x7f720c13f730 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:38.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f720b7fe640 10 monclient: dump: 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:38.454 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f7212e7e640 5 monclient: authenticate success, global_id 184107 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.443+0000 7f7212e7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7204003a80 con 0x7f720c13f730 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f720b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f720b7fe640 1 -- 172.21.15.1:0/3000082125 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7204003fb0 con 0x7f720c13f730 2024-08-19T10:01:38.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f720b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f720b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f720b7fe640 10 monclient: dump: 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:38.456 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f7212e7e640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f7212e7e640 4 set_mon_vals no callback set 2024-08-19T10:01:38.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.444+0000 7f7212e7e640 10 monclient: shutdown 2024-08-19T10:01:38.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.445+0000 7f7212e7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:38.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.445+0000 7f7212e7e640 1 -- 172.21.15.1:0/3000082125 >> v1:172.21.15.82:6790/0 conn(0x7f720c13f730 legacy=0x7f720c141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.445+0000 7f7212e7e640 1 -- 172.21.15.1:0/3000082125 shutdown_connections 2024-08-19T10:01:38.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.445+0000 7f7212e7e640 1 -- 172.21.15.1:0/3000082125 >> 172.21.15.1:0/3000082125 conn(0x7f720c0ac410 msgr2=0x7f720c0a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:38.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.445+0000 7f7212e7e640 1 -- 172.21.15.1:0/3000082125 shutdown_connections 2024-08-19T10:01:38.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.445+0000 7f7212e7e640 1 -- 172.21.15.1:0/3000082125 wait complete. 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 10 monclient: build_initial_monmap 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 10 monclient: monmap: 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:38.465 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 1 librados: starting msgr at 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 1 librados: starting objecter 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 1 Processor -- start 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 1 -- start start 2024-08-19T10:01:38.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 1 librados: setting wanted keys 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 1 librados: calling monclient init 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.446+0000 7f7212e7e640 10 monclient: init 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient: _add_conns ranks=[2,3,8,0,6,4,5,7,1] 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient(hunting): picked mon.noname-b con 0x7f720c0a4590 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient(hunting): picked mon.noname-g con 0x7f720c0a6720 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient(hunting): picked mon.noname-e con 0x7f720c13f730 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a15f0 con 0x7f720c0a4590 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a2200 con 0x7f720c0a6720 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a24c0 con 0x7f720c13f730 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7212e7e640 10 monclient(hunting): authenticate will time out at 1354.523891s 2024-08-19T10:01:38.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.447+0000 7f7210c1a640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f720c0a4590 0x7f720c1ce660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42674/0 (socket says 172.21.15.1:42674) 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.448+0000 7f7210c1a640 1 -- 172.21.15.1:0/2921708211 learned_addr learned my addr 172.21.15.1:0/2921708211 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f7210c1a640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.1:6790/0 conn(0x7f720c0a4590 legacy=0x7f720c1ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f72097fa640 10 client.?.objecter ms_handle_connect 0x7f720c0a4590 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f72097fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f72097fa640 10 client.?.objecter ms_handle_connect 0x7f720c13f730 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f72097fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f721141b640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.1:6793/0 conn(0x7f720c13f730 legacy=0x7f720c1d6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f720bfff640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.82:6790/0 conn(0x7f720c0a6720 legacy=0x7f720c1d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f72097fa640 10 client.?.objecter ms_handle_connect 0x7f720c0a6720 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.450+0000 7f72097fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567927928 0 0) 0x7f720c0a15f0 con 0x7f720c0a4590 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:38.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): my global_id is 184106 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71f0004eb0 con 0x7f720c0a4590 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890927945 0 0) 0x7f720c0a24c0 con 0x7f720c13f730 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): my global_id is 184112 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f720c0a15f0 con 0x7f720c13f730 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1051837219 0 0) 0x7f71f0004eb0 con 0x7f720c0a4590 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f720c0a24c0 con 0x7f720c0a4590 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7200003610 con 0x7f720c0a4590 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:01:38.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): dump: 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2788464067 0 0) 0x7f720c0a15f0 con 0x7f720c13f730 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71f0004eb0 con 0x7f720c13f730 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7204003f70 con 0x7f720c13f730 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.454+0000 7f72097fa640 10 monclient(hunting): dump: 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.471 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1371084886 0 0) 0x7f720c0a24c0 con 0x7f720c0a4590 2024-08-19T10:01:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.1:6793/0 conn(0x7f720c13f730 legacy=0x7f720c1d6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.82:6790/0 conn(0x7f720c0a6720 legacy=0x7f720c1d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: _finish_hunting 0 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 monclient: found mon.b 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c0a27a0 con 0x7f720c0a4590 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: _finish_auth 0 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: _check_auth_tickets 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 5 monclient: authenticate success, global_id 184106 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 10 monclient: _renew_subs 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:38.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f720c13e660 con 0x7f720c0a4590 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 10 client.184106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 10 monclient: _renew_subs 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7200003b50 con 0x7f720c0a4590 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f720c0a01f0 con 0x7f720c0a4590 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f7212e7e640 1 librados: init done 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7200004040 con 0x7f720c0a4590 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f72097fa640 10 monclient: dump: 2024-08-19T10:01:38.474 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:38.475 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.455+0000 7f720affd640 4 set_mon_vals no callback set 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f72000041e0 con 0x7f720c0a4590 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f7212e7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f7212e7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f7212e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(413..413 src has 299..413) ==== 5449+0+0 (unknown 3438123259 0 0) 0x7f720004e150 con 0x7f720c0a4590 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f72097fa640 10 client.184106.objecter ms_dispatch 0x7f720c000b90 osd_map(413..413 src has 299..413) 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f72097fa640 3 client.184106.objecter handle_osd_map got epochs [413,413] > 0 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f72097fa640 3 client.184106.objecter handle_osd_map decoding full epoch 413 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720c051b00 con 0x7f720c0a4590 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f72097fa640 20 client.184106.objecter dump_active .. 0 homeless 2024-08-19T10:01:38.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.456+0000 7f720bfff640 10 monclient: get_auth_request con 0x7f71f00414b0 auth_method 0 2024-08-19T10:01:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.459+0000 7f72097fa640 10 client.184106.objecter ms_handle_connect 0x7f71f00414b0 2024-08-19T10:01:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.461+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f7200017490 con 0x7f720c0a4590 2024-08-19T10:01:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.461+0000 7f72097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.461+0000 7f72097fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f7212e7e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f7212e7e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f7212e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f720c06bda0 con 0x7f720c0a4590 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f72097fa640 1 -- 172.21.15.1:0/2921708211 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 598210535) 0x7f7200016d90 con 0x7f720c0a4590 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f72097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:01:38.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.725+0000 7f72097fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:38.727 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":106,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 10 librados: watch_flush enter 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 10 librados: watch_flush exit 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 20 client.184106.objecter shutdown clearing up homeless session... 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 10 client.184106.objecter successfully canceled tick 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.82:6812/2549104415 conn(0x7f71f00414b0 legacy=0x7f71f0043970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 10 monclient: shutdown 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 >> v1:172.21.15.1:6790/0 conn(0x7f720c0a4590 legacy=0x7f720c1ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 shutdown_connections 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 >> 172.21.15.1:0/2921708211 conn(0x7f720c0ac410 msgr2=0x7f720c0a4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 shutdown_connections 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 -- 172.21.15.1:0/2921708211 wait complete. 2024-08-19T10:01:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.727+0000 7f7212e7e640 1 librados: shutdown 2024-08-19T10:01:38.739 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:01:38.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:01:38.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:01:38.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.838+0000 7f7d84333640 10 monclient: get_monmap_and_config 2024-08-19T10:01:38.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.838+0000 7f7d84333640 10 monclient: build_initial_monmap 2024-08-19T10:01:38.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.838+0000 7f7d84333640 10 monclient: monmap: 2024-08-19T10:01:38.839 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:38.839 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:38.839427+0000 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:38.839427+0000 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:38.841 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 1 Processor -- start 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 1 -- start start 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient: init 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient: _add_conns ranks=[3,7,1,4,2,6,5,8,0] 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient(hunting): picked mon.noname-g con 0x7f7d7c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient(hunting): picked mon.noname-i con 0x7f7d7c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient(hunting): picked mon.noname-f con 0x7f7d7c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c0a3c70 con 0x7f7d7c0a46a0 2024-08-19T10:01:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c0a33d0 con 0x7f7d7c0a6830 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c0a36b0 con 0x7f7d7c0694b0 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d84333640 10 monclient(hunting): authenticate will time out at 1354.915748s 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d818ce640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f7d7c0694b0 0x7f7d7c0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:40210/0 (socket says 172.21.15.1:40210) 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d820cf640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f7d7c0a6830 0x7f7d7c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:38110/0 (socket says 172.21.15.1:38110) 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d828d0640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f7d7c0a46a0 0x7f7d7c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:55568/0 (socket says 172.21.15.1:55568) 2024-08-19T10:01:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.839+0000 7f7d818ce640 1 -- 172.21.15.1:0/3191805836 learned_addr learned my addr 172.21.15.1:0/3191805836 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.842+0000 7f7d820cf640 1 -- 172.21.15.1:0/3191805836 >> v1:172.21.15.82:6790/0 conn(0x7f7d7c0a6830 legacy=0x7f7d7c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.842+0000 7f7d828d0640 1 -- 172.21.15.1:0/3191805836 >> v1:172.21.15.82:6789/0 conn(0x7f7d7c0a46a0 legacy=0x7f7d7c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:38.842+0000 7f7d818ce640 1 -- 172.21.15.1:0/3191805836 >> v1:172.21.15.82:6792/0 conn(0x7f7d7c0694b0 legacy=0x7f7d7c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340406368 0 0) 0x7f7d7c0a3c70 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient(hunting): my global_id is 174313 2024-08-19T10:01:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d64003520 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513916292 0 0) 0x7f7d64003520 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 >> v1:172.21.15.82:6792/0 conn(0x7f7d7c0694b0 legacy=0x7f7d7c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 >> v1:172.21.15.82:6790/0 conn(0x7f7d7c0a6830 legacy=0x7f7d7c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 monclient: found mon.noname-f 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c06dbb0 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7d74001ad0 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:01:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d810cd640 10 monclient: dump: 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d84333640 5 monclient: authenticate success, global_id 174313 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.135+0000 7f7d84333640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d74001cf0 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d810cd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d84333640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d84333640 4 set_mon_vals no callback set 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d810cd640 1 -- 172.21.15.1:0/3191805836 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7d74004b60 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d810cd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d810cd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.136+0000 7f7d810cd640 10 monclient: dump: 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.137+0000 7f7d84333640 10 monclient: shutdown 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.137+0000 7f7d84333640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.137+0000 7f7d84333640 1 -- 172.21.15.1:0/3191805836 >> v1:172.21.15.82:6789/0 conn(0x7f7d7c0a46a0 legacy=0x7f7d7c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.137+0000 7f7d84333640 1 -- 172.21.15.1:0/3191805836 shutdown_connections 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.137+0000 7f7d84333640 1 -- 172.21.15.1:0/3191805836 >> 172.21.15.1:0/3191805836 conn(0x7f7d7c0a9020 msgr2=0x7f7d7c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.137+0000 7f7d84333640 1 -- 172.21.15.1:0/3191805836 shutdown_connections 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.138+0000 7f7d84333640 1 -- 172.21.15.1:0/3191805836 wait complete. 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.138+0000 7f7d84333640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.138+0000 7f7d84333640 10 monclient: monmap: 2024-08-19T10:01:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.148 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.138+0000 7f7d84333640 1 librados: starting msgr at 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.138+0000 7f7d84333640 1 librados: starting objecter 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 Processor -- start 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 -- start start 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 librados: setting wanted keys 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 librados: calling monclient init 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient: init 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient: _add_conns ranks=[6,1,7,0,8,5,4,2,3] 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient(hunting): picked mon.noname-d con 0x7f7d7c0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient(hunting): picked mon.noname-f con 0x7f7d7c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient(hunting): picked mon.noname-i con 0x7f7d7c0a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:39.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c142cf0 con 0x7f7d7c0a46a0 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c13b040 con 0x7f7d7c0694b0 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c137a20 con 0x7f7d7c0a6830 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d84333640 10 monclient(hunting): authenticate will time out at 1355.215828s 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d820cf640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f7d7c0694b0 0x7f7d7c136c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:42982/0 (socket says 172.21.15.1:42982) 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.139+0000 7f7d820cf640 1 -- 172.21.15.1:0/3925495786 learned_addr learned my addr 172.21.15.1:0/3925495786 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d820cf640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6792/0 conn(0x7f7d7c0694b0 legacy=0x7f7d7c136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 client.?.objecter ms_handle_connect 0x7f7d7c0694b0 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 client.?.objecter ms_handle_connect 0x7f7d7c0a6830 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d828d0640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.82:6792/0 conn(0x7f7d7c0a6830 legacy=0x7f7d7c1d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193480440 0 0) 0x7f7d7c13b040 con 0x7f7d7c0694b0 2024-08-19T10:01:39.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): my global_id is 174345 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d68004eb0 con 0x7f7d7c0694b0 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 225890766 0 0) 0x7f7d68004eb0 con 0x7f7d7c0694b0 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d7c13b040 con 0x7f7d7c0694b0 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7d6c004b60 con 0x7f7d7c0694b0 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): dump: 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649397477 0 0) 0x7f7d7c137a20 con 0x7f7d7c0a6830 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient(hunting): my global_id is 144817 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d68004eb0 con 0x7f7d7c0a6830 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 235046642 0 0) 0x7f7d7c13b040 con 0x7f7d7c0694b0 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.82:6792/0 conn(0x7f7d7c0a6830 legacy=0x7f7d7c1d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.82:6789/0 conn(0x7f7d7c0a46a0 legacy=0x7f7d7c137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 monclient: found mon.d 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c13a190 con 0x7f7d7c0694b0 2024-08-19T10:01:39.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 5 monclient: authenticate success, global_id 174345 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 10 monclient: _renew_subs 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7c1386c0 con 0x7f7d7c0694b0 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 10 client.174345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 10 monclient: _renew_subs 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d7c1394f0 con 0x7f7d7c0694b0 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d84333640 1 librados: init done 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d6c0050b0 con 0x7f7d7c0694b0 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d808cc640 4 set_mon_vals no callback set 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7d6c005660 con 0x7f7d7c0694b0 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.142+0000 7f7d7affd640 10 monclient: dump: 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.143+0000 7f7d84333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.143+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7d5c0065c0 con 0x7f7d5c002860 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.143+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7d6c005880 con 0x7f7d7c0694b0 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.144+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(414..414 src has 299..414) ==== 5465+0+0 (unknown 3895169888 0 0) 0x7f7d6c04e7b0 con 0x7f7d7c0694b0 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.144+0000 7f7d7affd640 10 client.174345.objecter ms_dispatch 0x7f7d7c000b90 osd_map(414..414 src has 299..414) 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.144+0000 7f7d7affd640 3 client.174345.objecter handle_osd_map got epochs [414,414] > 0 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.144+0000 7f7d7affd640 3 client.174345.objecter handle_osd_map decoding full epoch 414 2024-08-19T10:01:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.144+0000 7f7d7affd640 20 client.174345.objecter dump_active .. 0 homeless 2024-08-19T10:01:39.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.144+0000 7f7d828d0640 10 monclient: get_auth_request con 0x7f7d68040ee0 auth_method 0 2024-08-19T10:01:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.146+0000 7f7d7affd640 10 client.174345.objecter ms_handle_connect 0x7f7d68040ee0 2024-08-19T10:01:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.148+0000 7f7d818ce640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6789/0 conn(0x7f7d5c002860 legacy=0x7f7d5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.148+0000 7f7d7affd640 10 client.174345.objecter ms_handle_connect 0x7f7d5c002860 2024-08-19T10:01:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.148+0000 7f7d7affd640 10 client.174345.objecter resend_mon_ops 2024-08-19T10:01:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133776968 0 0) 0x7f7d5c0065c0 con 0x7f7d5c002860 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient(hunting): my global_id is 174348 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d7c13b040 con 0x7f7d5c002860 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1928860406 0 0) 0x7f7d7c13b040 con 0x7f7d5c002860 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d5c0065c0 con 0x7f7d5c002860 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7d70001b10 con 0x7f7d5c002860 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient: dump: 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1583185119 0 0) 0x7f7d5c0065c0 con 0x7f7d5c002860 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d5c00a160 con 0x7f7d5c002860 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7d700058d0 con 0x7f7d5c002860 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6789/0 conn(0x7f7d5c002860 legacy=0x7f7d5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.149+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6789/0 conn(0x7f7d5c002860 legacy=0x7f7d5c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.156+0000 7f7d84333640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.156+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7d5c009e50 con 0x7f7d5c009110 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d818ce640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6789/0 conn(0x7f7d5c009110 legacy=0x7f7d5c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 client.174345.objecter ms_handle_connect 0x7f7d5c009110 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 client.174345.objecter resend_mon_ops 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3947649052 0 0) 0x7f7d5c009e50 con 0x7f7d5c009110 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient(hunting): my global_id is 174357 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d5c0065c0 con 0x7f7d5c009110 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3607060542 0 0) 0x7f7d5c0065c0 con 0x7f7d5c009110 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d5c009e50 con 0x7f7d5c009110 2024-08-19T10:01:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7d70004d50 con 0x7f7d5c009110 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.158+0000 7f7d7affd640 10 monclient: dump: 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1996277701 0 0) 0x7f7d5c009e50 con 0x7f7d5c009110 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d5c00c3a0 con 0x7f7d5c009110 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7687 (unknown 0 0 2347777591) 0x7f7d70005b00 con 0x7f7d5c009110 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6789/0 conn(0x7f7d5c009110 legacy=0x7f7d5c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d7affd640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6789/0 conn(0x7f7d5c009110 legacy=0x7f7d5c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 149200, 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:39.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.167 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.168 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:39.169 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.171 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:39.174 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.175 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.176 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:39.177 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.180 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.181 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.182 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.183 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.184 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:39.185 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:39.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 10 librados: watch_flush enter 2024-08-19T10:01:39.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 10 librados: watch_flush exit 2024-08-19T10:01:39.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 20 client.174345.objecter shutdown clearing up homeless session... 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 10 client.174345.objecter successfully canceled tick 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7d68040ee0 legacy=0x7f7d680433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 10 monclient: shutdown 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 >> v1:172.21.15.1:6792/0 conn(0x7f7d7c0694b0 legacy=0x7f7d7c136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d828d0640 1 -- 172.21.15.1:0/3925495786 reap_dead start 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d828d0640 1 -- 172.21.15.1:0/3925495786 reap_dead start 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 shutdown_connections 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 >> 172.21.15.1:0/3925495786 conn(0x7f7d7c0a9020 msgr2=0x7f7d7c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 shutdown_connections 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 -- 172.21.15.1:0/3925495786 wait complete. 2024-08-19T10:01:39.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.159+0000 7f7d84333640 1 librados: shutdown 2024-08-19T10:01:39.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:39.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:01:39.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:01:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.289+0000 7f8118928640 10 monclient: get_monmap_and_config 2024-08-19T10:01:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.289+0000 7f8118928640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.289+0000 7f8118928640 10 monclient: monmap: 2024-08-19T10:01:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:39.290105+0000 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:39.290105+0000 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.292 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.289+0000 7f8118928640 1 Processor -- start 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.289+0000 7f8118928640 1 -- start start 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.289+0000 7f8118928640 10 monclient: init 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient: _add_conns ranks=[4,5,7,2,8,6,1,3,0] 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient(hunting): picked mon.noname-c con 0x7f81140a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient(hunting): picked mon.noname-h con 0x7f81140694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient(hunting): picked mon.noname-i con 0x7f81140a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81140a3c70 con 0x7f81140a6830 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81140a33d0 con 0x7f81140694b0 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81140a36b0 con 0x7f81140a46a0 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f8118928640 10 monclient(hunting): authenticate will time out at 1355.366470s 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f811259c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f81140a6830 0x7f81140a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44654/0 (socket says 172.21.15.1:44654) 2024-08-19T10:01:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.290+0000 7f811259c640 1 -- 172.21.15.1:0/304429376 learned_addr learned my addr 172.21.15.1:0/304429376 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f811259c640 1 -- 172.21.15.1:0/304429376 >> v1:172.21.15.1:6791/0 conn(0x7f81140a6830 legacy=0x7f81140a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182874503 0 0) 0x7f81140a3c70 con 0x7f81140a6830 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f8111d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f8111d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f8111d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f8111d9b640 10 monclient(hunting): my global_id is 27697 2024-08-19T10:01:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.292+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8100003520 con 0x7f81140a6830 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8112d9d640 1 -- 172.21.15.1:0/304429376 >> v1:172.21.15.82:6792/0 conn(0x7f81140a46a0 legacy=0x7f8114141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f810bfff640 1 -- 172.21.15.1:0/304429376 >> v1:172.21.15.82:6791/0 conn(0x7f81140694b0 legacy=0x7f81140a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2413724500 0 0) 0x7f8100003520 con 0x7f81140a6830 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 >> v1:172.21.15.82:6792/0 conn(0x7f81140a46a0 legacy=0x7f8114141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 >> v1:172.21.15.82:6791/0 conn(0x7f81140694b0 legacy=0x7f81140a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 monclient: found mon.noname-c 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:01:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811406dbb0 con 0x7f81140a6830 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8104001ae0 con 0x7f81140a6830 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: dump: 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 5 monclient: authenticate success, global_id 27697 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81040048a0 con 0x7f81140a6830 2024-08-19T10:01:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 1 -- 172.21.15.1:0/304429376 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8104004e10 con 0x7f81140a6830 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 4 set_mon_vals no callback set 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8111d9b640 10 monclient: dump: 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.309 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.310 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 10 monclient: shutdown 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 1 -- 172.21.15.1:0/304429376 >> v1:172.21.15.1:6791/0 conn(0x7f81140a6830 legacy=0x7f81140a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 1 -- 172.21.15.1:0/304429376 shutdown_connections 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 1 -- 172.21.15.1:0/304429376 >> 172.21.15.1:0/304429376 conn(0x7f81140a9020 msgr2=0x7f8114068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 1 -- 172.21.15.1:0/304429376 shutdown_connections 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.293+0000 7f8118928640 1 -- 172.21.15.1:0/304429376 wait complete. 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.294+0000 7f8118928640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.294+0000 7f8118928640 10 monclient: monmap: 2024-08-19T10:01:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.294+0000 7f8118928640 1 librados: starting msgr at 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.294+0000 7f8118928640 1 librados: starting objecter 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.294+0000 7f8118928640 1 Processor -- start 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 1 -- start start 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 1 librados: setting wanted keys 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 1 librados: calling monclient init 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient: init 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient: _add_conns ranks=[6,7,1,0,4,8,3,5,2] 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient(hunting): picked mon.noname-d con 0x7f81140694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient(hunting): picked mon.noname-i con 0x7f81140a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient(hunting): picked mon.noname-f con 0x7f81140a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8114137e50 con 0x7f81140a6830 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8114138440 con 0x7f81140694b0 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81140a9f50 con 0x7f81140a46a0 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f8118928640 10 monclient(hunting): authenticate will time out at 1355.371571s 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f811259c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f81140694b0 0x7f81141ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:42990/0 (socket says 172.21.15.1:42990) 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.295+0000 7f811259c640 1 -- 172.21.15.1:0/3063287451 learned_addr learned my addr 172.21.15.1:0/3063287451 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f811259c640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6792/0 conn(0x7f81140694b0 legacy=0x7f81141ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 10 client.?.objecter ms_handle_connect 0x7f81140694b0 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897214425 0 0) 0x7f8114138440 con 0x7f81140694b0 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 10 monclient(hunting): my global_id is 174354 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.297+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f4004eb0 con 0x7f81140694b0 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 client.?.objecter ms_handle_connect 0x7f81140a46a0 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810bfff640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.82:6792/0 conn(0x7f81140a46a0 legacy=0x7f81141ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8112d9d640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.82:6789/0 conn(0x7f81140a6830 legacy=0x7f81141d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 client.?.objecter ms_handle_connect 0x7f81140a6830 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 517479362 0 0) 0x7f80f4004eb0 con 0x7f81140694b0 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8114138440 con 0x7f81140694b0 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8104001d30 con 0x7f81140694b0 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): dump: 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647047187 0 0) 0x7f81140a33d0 con 0x7f81140a46a0 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): my global_id is 144835 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f4004eb0 con 0x7f81140a46a0 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369228109 0 0) 0x7f8114137e50 con 0x7f81140a6830 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient(hunting): my global_id is 174322 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81140a33d0 con 0x7f81140a6830 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1365206750 0 0) 0x7f8114138440 con 0x7f81140694b0 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.82:6792/0 conn(0x7f81140a46a0 legacy=0x7f81141ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.82:6789/0 conn(0x7f81140a6830 legacy=0x7f81141d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 monclient: found mon.d 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81141cf1d0 con 0x7f81140694b0 2024-08-19T10:01:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 5 monclient: authenticate success, global_id 174354 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 10 monclient: _renew_subs 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f811413ae60 con 0x7f81140694b0 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 10 client.174354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 10 monclient: _renew_subs 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f81141426b0 con 0x7f81140694b0 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f8118928640 1 librados: init done 2024-08-19T10:01:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8104005040 con 0x7f81140694b0 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8104005530 con 0x7f81140694b0 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f811159a640 4 set_mon_vals no callback set 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.298+0000 7f810affd640 10 monclient: dump: 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.322 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f8118928640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f80f80065c0 con 0x7f80f8002860 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8104011990 con 0x7f81140694b0 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(414..414 src has 299..414) ==== 5465+0+0 (unknown 3895169888 0 0) 0x7f810404e270 con 0x7f81140694b0 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f810affd640 10 client.174354.objecter ms_dispatch 0x7f8114000b90 osd_map(414..414 src has 299..414) 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f810affd640 3 client.174354.objecter handle_osd_map got epochs [414,414] > 0 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f810affd640 3 client.174354.objecter handle_osd_map decoding full epoch 414 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f810affd640 20 client.174354.objecter dump_active .. 0 homeless 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.299+0000 7f8112d9d640 10 monclient: get_auth_request con 0x7f80f4041370 auth_method 0 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810bfff640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6790/0 conn(0x7f80f8002860 legacy=0x7f80f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 client.174354.objecter ms_handle_connect 0x7f80f8002860 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 client.174354.objecter resend_mon_ops 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577380671 0 0) 0x7f80f80065c0 con 0x7f80f8002860 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient(hunting): my global_id is 184115 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8114138440 con 0x7f80f8002860 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2343628790 0 0) 0x7f8114138440 con 0x7f80f8002860 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80f80065c0 con 0x7f80f8002860 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f80fc001c30 con 0x7f80f8002860 2024-08-19T10:01:39.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.301+0000 7f810affd640 10 monclient: dump: 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3657729974 0 0) 0x7f80f80065c0 con 0x7f80f8002860 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80f800a160 con 0x7f80f8002860 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 10 client.174354.objecter ms_handle_connect 0x7f80f4041370 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f80fc006810 con 0x7f80f8002860 2024-08-19T10:01:39.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:39.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6790/0 conn(0x7f80f8002860 legacy=0x7f80f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.302+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6790/0 conn(0x7f80f8002860 legacy=0x7f80f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.308+0000 7f8118928640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.308+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f80f8009e70 con 0x7f80f8009130 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810bfff640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6790/0 conn(0x7f80f8009130 legacy=0x7f80f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 10 client.174354.objecter ms_handle_connect 0x7f80f8009130 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 10 client.174354.objecter resend_mon_ops 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3073344330 0 0) 0x7f80f8009e70 con 0x7f80f8009130 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 10 monclient(hunting): my global_id is 184124 2024-08-19T10:01:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.310+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f80065c0 con 0x7f80f8009130 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1485170235 0 0) 0x7f80f80065c0 con 0x7f80f8009130 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80f8009e70 con 0x7f80f8009130 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f80fc003d40 con 0x7f80f8009130 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 10 monclient: dump: 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.330 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2906546861 0 0) 0x7f80f8009e70 con 0x7f80f8009130 2024-08-19T10:01:39.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f80f800c3f0 con 0x7f80f8009130 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3208471908) 0x7f80fc004110 con 0x7f80f8009130 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6790/0 conn(0x7f80f8009130 legacy=0x7f80f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f810affd640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6790/0 conn(0x7f80f8009130 legacy=0x7f80f80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:39.332 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16559, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:39.333 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.335 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.336 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.338 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:39.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.342 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.345 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.347 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.349 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:39.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:39.355 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:39.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 10 librados: watch_flush enter 2024-08-19T10:01:39.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 10 librados: watch_flush exit 2024-08-19T10:01:39.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 20 client.174354.objecter shutdown clearing up homeless session... 2024-08-19T10:01:39.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 10 client.174354.objecter successfully canceled tick 2024-08-19T10:01:39.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.82:6812/2549104415 conn(0x7f80f4041370 legacy=0x7f80f4043830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 10 monclient: shutdown 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 >> v1:172.21.15.1:6792/0 conn(0x7f81140694b0 legacy=0x7f81141ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.311+0000 7f8112d9d640 1 -- 172.21.15.1:0/3063287451 reap_dead start 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.312+0000 7f8112d9d640 1 -- 172.21.15.1:0/3063287451 reap_dead start 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.312+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 shutdown_connections 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.312+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 >> 172.21.15.1:0/3063287451 conn(0x7f81140a9020 msgr2=0x7f81140a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.312+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 shutdown_connections 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.312+0000 7f8118928640 1 -- 172.21.15.1:0/3063287451 wait complete. 2024-08-19T10:01:39.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.312+0000 7f8118928640 1 librados: shutdown 2024-08-19T10:01:39.359 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:39.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:01:39.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:01:39.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.458+0000 7fe87a73a640 10 monclient: get_monmap_and_config 2024-08-19T10:01:39.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.458+0000 7fe87a73a640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.458+0000 7fe87a73a640 10 monclient: monmap: 2024-08-19T10:01:39.459 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:39.459198+0000 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:39.459198+0000 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.460 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.461 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.461 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.461 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.458+0000 7fe87a73a640 1 Processor -- start 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 1 -- start start 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient: init 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient: _add_conns ranks=[5,3,4,6,8,7,1,2,0] 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient(hunting): picked mon.noname-h con 0x7fe8740a3b50 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient(hunting): picked mon.noname-g con 0x7fe8740a4640 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient(hunting): picked mon.noname-c con 0x7fe87409cc80 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87409d0b0 con 0x7fe8740a4640 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87409c250 con 0x7fe87409cc80 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87409c4f0 con 0x7fe8740a3b50 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe87a73a640 10 monclient(hunting): authenticate will time out at 1355.535524s 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe878cd7640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fe87409cc80 0x7fe874141e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44662/0 (socket says 172.21.15.1:44662) 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.459+0000 7fe878cd7640 1 -- 172.21.15.1:0/3991709191 learned_addr learned my addr 172.21.15.1:0/3991709191 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe878cd7640 1 -- 172.21.15.1:0/3991709191 >> v1:172.21.15.1:6791/0 conn(0x7fe87409cc80 legacy=0x7fe874141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe873fff640 1 -- 172.21.15.1:0/3991709191 >> v1:172.21.15.82:6791/0 conn(0x7fe8740a3b50 legacy=0x7fe8740a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617806860 0 0) 0x7fe87409c250 con 0x7fe87409cc80 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe872ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe872ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe872ffd640 10 monclient(hunting): my global_id is 27706 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe8737fe640 1 -- 172.21.15.1:0/3991709191 >> v1:172.21.15.82:6790/0 conn(0x7fe8740a4640 legacy=0x7fe87413e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.461+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe85c003520 con 0x7fe87409cc80 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3846917553 0 0) 0x7fe85c003520 con 0x7fe87409cc80 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 >> v1:172.21.15.82:6791/0 conn(0x7fe8740a3b50 legacy=0x7fe8740a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 >> v1:172.21.15.82:6790/0 conn(0x7fe8740a4640 legacy=0x7fe87413e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 monclient: found mon.noname-c 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87409c7d0 con 0x7fe87409cc80 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe86c001e60 con 0x7fe87409cc80 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 5 monclient: authenticate success, global_id 27706 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: dump: 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.472 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe86c004c20 con 0x7fe87409cc80 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 1 -- 172.21.15.1:0/3991709191 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe86c005190 con 0x7fe87409cc80 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe872ffd640 10 monclient: dump: 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.474 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 4 set_mon_vals no callback set 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 10 monclient: shutdown 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 1 -- 172.21.15.1:0/3991709191 >> v1:172.21.15.1:6791/0 conn(0x7fe87409cc80 legacy=0x7fe874141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 1 -- 172.21.15.1:0/3991709191 shutdown_connections 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 1 -- 172.21.15.1:0/3991709191 >> 172.21.15.1:0/3991709191 conn(0x7fe8740a4c40 msgr2=0x7fe8740a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 1 -- 172.21.15.1:0/3991709191 shutdown_connections 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.462+0000 7fe87a73a640 1 -- 172.21.15.1:0/3991709191 wait complete. 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.463+0000 7fe87a73a640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.463+0000 7fe87a73a640 10 monclient: monmap: 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.463+0000 7fe87a73a640 1 librados: starting msgr at 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.463+0000 7fe87a73a640 1 librados: starting objecter 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.463+0000 7fe87a73a640 1 Processor -- start 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 1 -- start start 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 1 librados: setting wanted keys 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 1 librados: calling monclient init 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient: init 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient: _add_conns ranks=[8,0,4,5,3,2,1,7,6] 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient(hunting): picked mon.noname-e con 0x7fe87409cc80 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:39.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient(hunting): picked mon.noname-a con 0x7fe8740a3b50 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient(hunting): picked mon.noname-c con 0x7fe8740a4640 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87409b9c0 con 0x7fe8740a3b50 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87409dcf0 con 0x7fe8740a4640 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe874097c20 con 0x7fe87409cc80 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe87a73a640 10 monclient(hunting): authenticate will time out at 1355.540529s 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe8737fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe8740a3b50 0x7fe8741d35a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47848/0 (socket says 172.21.15.1:47848) 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe8737fe640 1 -- 172.21.15.1:0/2234135891 learned_addr learned my addr 172.21.15.1:0/2234135891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.464+0000 7fe878cd7640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fe8740a4640 0x7fe8741d6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44678/0 (socket says 172.21.15.1:44678) 2024-08-19T10:01:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe8737fe640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6789/0 conn(0x7fe8740a3b50 legacy=0x7fe8741d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe878cd7640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe8740a4640 legacy=0x7fe8741d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 client.?.objecter ms_handle_connect 0x7fe8740a3b50 2024-08-19T10:01:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 client.?.objecter ms_handle_connect 0x7fe8740a4640 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 client.?.objecter ms_handle_connect 0x7fe87409cc80 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe873fff640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6793/0 conn(0x7fe87409cc80 legacy=0x7fe8741ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575529809 0 0) 0x7fe87409b9c0 con 0x7fe8740a3b50 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): my global_id is 174366 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe860004eb0 con 0x7fe8740a3b50 2024-08-19T10:01:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547769174 0 0) 0x7fe874097c20 con 0x7fe87409cc80 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 10 monclient(hunting): my global_id is 184121 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.466+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87409b9c0 con 0x7fe87409cc80 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2517684374 0 0) 0x7fe860004eb0 con 0x7fe8740a3b50 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe874097c20 con 0x7fe8740a3b50 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3624947990 0 0) 0x7fe87409b9c0 con 0x7fe87409cc80 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe860004eb0 con 0x7fe87409cc80 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe868001d40 con 0x7fe8740a3b50 2024-08-19T10:01:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): dump: 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe864003610 con 0x7fe87409cc80 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient(hunting): dump: 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 495551484 0 0) 0x7fe874097c20 con 0x7fe8740a3b50 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe873fff640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6793/0 conn(0x7fe87409cc80 legacy=0x7fe8741ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe873fff640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6793/0 conn(0x7fe87409cc80 legacy=0x7fe8741ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe873fff640 1 --1- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6793/0 conn(0x7fe87409cc80 0x7fe8741ce630 :-1 s=OPENED pgs=3481 cs=1 l=1).handle_message read tag failed 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6793/0 conn(0x7fe87409cc80 legacy=0x7fe8741ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe873fff640 1 --1- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6793/0 conn(0x7fe87409cc80 0x7fe8741ce630 :-1 s=OPENED pgs=3481 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:01:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe8740a4640 legacy=0x7fe8741d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 monclient: found mon.a 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe874096660 con 0x7fe8740a3b50 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: ms_handle_reset stray mon v1:172.21.15.1:6793/0 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 5 monclient: authenticate success, global_id 174366 2024-08-19T10:01:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 10 monclient: _renew_subs 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe874098180 con 0x7fe8740a3b50 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 10 client.174366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 10 monclient: _renew_subs 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe87409bec0 con 0x7fe8740a3b50 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe87a73a640 1 librados: init done 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe868003b00 con 0x7fe8740a3b50 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe868003ff0 con 0x7fe8740a3b50 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe8727fc640 4 set_mon_vals no callback set 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.467+0000 7fe870ff9640 10 monclient: dump: 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe8680041c0 con 0x7fe8740a3b50 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe87a73a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe8540065c0 con 0x7fe854002860 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(414..414 src has 299..414) ==== 5465+0+0 (unknown 3895169888 0 0) 0x7fe86804e170 con 0x7fe8740a3b50 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe870ff9640 10 client.174366.objecter ms_dispatch 0x7fe874000b90 osd_map(414..414 src has 299..414) 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe870ff9640 3 client.174366.objecter handle_osd_map got epochs [414,414] > 0 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe870ff9640 3 client.174366.objecter handle_osd_map decoding full epoch 414 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe870ff9640 20 client.174366.objecter dump_active .. 0 homeless 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.468+0000 7fe878cd7640 10 monclient: get_auth_request con 0x7fe860041560 auth_method 0 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe873fff640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe854002860 legacy=0x7fe854004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 10 client.174366.objecter ms_handle_connect 0x7fe854002860 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 10 client.174366.objecter resend_mon_ops 2024-08-19T10:01:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1785158713 0 0) 0x7fe8540065c0 con 0x7fe854002860 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 10 monclient(hunting): my global_id is 27715 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.470+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe874097c20 con 0x7fe854002860 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3112410272 0 0) 0x7fe874097c20 con 0x7fe854002860 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8540065c0 con 0x7fe854002860 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe864003dd0 con 0x7fe854002860 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 10 monclient: dump: 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4011646799 0 0) 0x7fe8540065c0 con 0x7fe854002860 2024-08-19T10:01:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe85400a160 con 0x7fe854002860 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 10 client.174366.objecter ms_handle_connect 0x7fe860041560 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe864004070 con 0x7fe854002860 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe854002860 legacy=0x7fe854004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.471+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe854002860 legacy=0x7fe854004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.477+0000 7fe87a73a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.477+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe854009e50 con 0x7fe854009110 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.479+0000 7fe873fff640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe854009110 legacy=0x7fe8540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.479+0000 7fe870ff9640 10 client.174366.objecter ms_handle_connect 0x7fe854009110 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.479+0000 7fe870ff9640 10 client.174366.objecter resend_mon_ops 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2986781398 0 0) 0x7fe854009e50 con 0x7fe854009110 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient(hunting): my global_id is 27724 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8540065c0 con 0x7fe854009110 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2822850623 0 0) 0x7fe8540065c0 con 0x7fe854009110 2024-08-19T10:01:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe854009e50 con 0x7fe854009110 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe864005b00 con 0x7fe854009110 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient: dump: 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 220642621 0 0) 0x7fe854009e50 con 0x7fe854009110 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe85400c3a0 con 0x7fe854009110 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 4268688844) 0x7fe864006fb0 con 0x7fe854009110 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe854009110 legacy=0x7fe8540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe870ff9640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6791/0 conn(0x7fe854009110 legacy=0x7fe8540095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.506 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 705752, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:39.530 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:39.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe87a73a640 10 librados: watch_flush enter 2024-08-19T10:01:39.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe87a73a640 10 librados: watch_flush exit 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe87a73a640 20 client.174366.objecter shutdown clearing up homeless session... 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe87a73a640 10 client.174366.objecter successfully canceled tick 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.480+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe860041560 legacy=0x7fe860043a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 10 monclient: shutdown 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 >> v1:172.21.15.1:6789/0 conn(0x7fe8740a3b50 legacy=0x7fe8741d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe878cd7640 1 -- 172.21.15.1:0/2234135891 reap_dead start 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe878cd7640 1 -- 172.21.15.1:0/2234135891 reap_dead start 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 shutdown_connections 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 >> 172.21.15.1:0/2234135891 conn(0x7fe8740a4c40 msgr2=0x7fe87408efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 shutdown_connections 2024-08-19T10:01:39.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 1 -- 172.21.15.1:0/2234135891 wait complete. 2024-08-19T10:01:39.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.481+0000 7fe87a73a640 1 librados: shutdown 2024-08-19T10:01:39.534 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:39.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:01:39.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:01:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.633+0000 7f1c45ffd640 10 monclient: get_monmap_and_config 2024-08-19T10:01:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.633+0000 7f1c45ffd640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.633+0000 7f1c45ffd640 10 monclient: monmap: 2024-08-19T10:01:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:39.634240+0000 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:39.634240+0000 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.636 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 1 Processor -- start 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 1 -- start start 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient: init 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient: _add_conns ranks=[0,2,4,5,6,8,7,3,1] 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f1c400a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f1c400a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c400a3b40 con 0x7f1c400a67c0 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c400a8130 con 0x7f1c400a6ba0 2024-08-19T10:01:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4006ede0 con 0x7f1c4013f5c0 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c45ffd640 10 monclient(hunting): authenticate will time out at 1355.710595s 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c3effd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f1c400a6ba0 0x7f1c400a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42712/0 (socket says 172.21.15.1:42712) 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c3ffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f1c4013f5c0 0x7f1c401419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44706/0 (socket says 172.21.15.1:44706) 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.634+0000 7f1c3effd640 1 -- 172.21.15.1:0/2371701640 learned_addr learned my addr 172.21.15.1:0/2371701640 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.636+0000 7f1c3effd640 1 -- 172.21.15.1:0/2371701640 >> v1:172.21.15.1:6790/0 conn(0x7f1c400a6ba0 legacy=0x7f1c400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.636+0000 7f1c3ffff640 1 -- 172.21.15.1:0/2371701640 >> v1:172.21.15.1:6791/0 conn(0x7f1c4013f5c0 legacy=0x7f1c401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.636+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/2371701640 >> v1:172.21.15.1:6789/0 conn(0x7f1c400a67c0 legacy=0x7f1c400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1381066306 0 0) 0x7f1c4006ede0 con 0x7f1c4013f5c0 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): my global_id is 27733 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c28003520 con 0x7f1c4013f5c0 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84316679 0 0) 0x7f1c400a8130 con 0x7f1c400a6ba0 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): my global_id is 184133 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4006ede0 con 0x7f1c400a6ba0 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511488053 0 0) 0x7f1c400a3b40 con 0x7f1c400a67c0 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient(hunting): my global_id is 174375 2024-08-19T10:01:39.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c400a8130 con 0x7f1c400a67c0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 546573334 0 0) 0x7f1c28003520 con 0x7f1c4013f5c0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 >> v1:172.21.15.1:6790/0 conn(0x7f1c400a6ba0 legacy=0x7f1c400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 >> v1:172.21.15.1:6789/0 conn(0x7f1c400a67c0 legacy=0x7f1c400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 monclient: found mon.noname-c 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c400a92a0 con 0x7f1c4013f5c0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c38001d80 con 0x7f1c4013f5c0 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 5 monclient: authenticate success, global_id 27733 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: dump: 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c30001a50 con 0x7f1c400a67c0 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c38002090 con 0x7f1c4013f5c0 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 1 -- 172.21.15.1:0/2371701640 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c38004ea0 con 0x7f1c4013f5c0 2024-08-19T10:01:39.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 4 set_mon_vals no callback set 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c3e7fc640 10 monclient: dump: 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.652 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 10 monclient: shutdown 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.637+0000 7f1c45ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2371701640 >> v1:172.21.15.1:6791/0 conn(0x7f1c4013f5c0 legacy=0x7f1c401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2371701640 shutdown_connections 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2371701640 >> 172.21.15.1:0/2371701640 conn(0x7f1c400ac4b0 msgr2=0x7f1c400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2371701640 shutdown_connections 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2371701640 wait complete. 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 10 monclient: monmap: 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 librados: starting msgr at 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 librados: starting objecter 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.638+0000 7f1c45ffd640 1 Processor -- start 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 1 -- start start 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 1 librados: setting wanted keys 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 1 librados: calling monclient init 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient: init 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient: _add_conns ranks=[7,0,6,2,4,1,5,3,8] 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient(hunting): picked mon.noname-i con 0x7f1c400a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f1c400a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient(hunting): picked mon.noname-d con 0x7f1c4013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4013a800 con 0x7f1c400a6ba0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c400a2860 con 0x7f1c4013f5c0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c400a2200 con 0x7f1c400a67c0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c45ffd640 10 monclient(hunting): authenticate will time out at 1355.715733s 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c3effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1c400a6ba0 0x7f1c400a02a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47870/0 (socket says 172.21.15.1:47870) 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.639+0000 7f1c3effd640 1 -- 172.21.15.1:0/2763028670 learned_addr learned my addr 172.21.15.1:0/2763028670 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.641+0000 7f1c3effd640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6789/0 conn(0x7f1c400a6ba0 legacy=0x7f1c400a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.641+0000 7f1c3ffff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c4013f5c0 legacy=0x7f1c400a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.641+0000 7f1c27fff640 10 client.?.objecter ms_handle_connect 0x7f1c400a6ba0 2024-08-19T10:01:39.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.641+0000 7f1c27fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.641+0000 7f1c27fff640 10 client.?.objecter ms_handle_connect 0x7f1c4013f5c0 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.641+0000 7f1c27fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1514146802 0 0) 0x7f1c4013a800 con 0x7f1c400a6ba0 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): my global_id is 174384 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c2c004eb0 con 0x7f1c400a6ba0 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 981940299 0 0) 0x7f1c400a2860 con 0x7f1c4013f5c0 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): my global_id is 174363 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4013a800 con 0x7f1c4013f5c0 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.82:6792/0 conn(0x7f1c400a67c0 legacy=0x7f1c4009fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 client.?.objecter ms_handle_connect 0x7f1c400a67c0 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 850846173 0 0) 0x7f1c4013a800 con 0x7f1c4013f5c0 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c400a2860 con 0x7f1c4013f5c0 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c38004ea0 con 0x7f1c4013f5c0 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): dump: 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.658 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:39.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 263309878 0 0) 0x7f1c2c004eb0 con 0x7f1c400a6ba0 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c4013a800 con 0x7f1c400a6ba0 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c340047e0 con 0x7f1c400a6ba0 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): dump: 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3789562106 0 0) 0x7f1c400a2200 con 0x7f1c400a67c0 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient(hunting): my global_id is 144844 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c2c004eb0 con 0x7f1c400a67c0 2024-08-19T10:01:39.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 548463504 0 0) 0x7f1c400a2860 con 0x7f1c4013f5c0 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.82:6792/0 conn(0x7f1c400a67c0 legacy=0x7f1c4009fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6789/0 conn(0x7f1c400a6ba0 legacy=0x7f1c400a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 monclient: found mon.d 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c40139b50 con 0x7f1c4013f5c0 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 5 monclient: authenticate success, global_id 174363 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 10 monclient: _renew_subs 2024-08-19T10:01:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c400a1600 con 0x7f1c4013f5c0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 10 client.174363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 10 monclient: _renew_subs 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c40139f60 con 0x7f1c4013f5c0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c45ffd640 1 librados: init done 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c38002090 con 0x7f1c4013f5c0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c38001d80 con 0x7f1c4013f5c0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c3dffb640 4 set_mon_vals no callback set 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.642+0000 7f1c27fff640 10 monclient: dump: 2024-08-19T10:01:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.664 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c45ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1c380118f0 con 0x7f1c4013f5c0 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1c200065c0 con 0x7f1c20002860 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(414..414 src has 299..414) ==== 5465+0+0 (unknown 3895169888 0 0) 0x7f1c3804e320 con 0x7f1c4013f5c0 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c27fff640 10 client.174363.objecter ms_dispatch 0x7f1c40000b90 osd_map(414..414 src has 299..414) 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c27fff640 3 client.174363.objecter handle_osd_map got epochs [414,414] > 0 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c27fff640 3 client.174363.objecter handle_osd_map decoding full epoch 414 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.643+0000 7f1c27fff640 20 client.174363.objecter dump_active .. 0 homeless 2024-08-19T10:01:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.644+0000 7f1c3effd640 10 monclient: get_auth_request con 0x7f1c2c041620 auth_method 0 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c20002860 legacy=0x7f1c20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 10 client.174363.objecter ms_handle_connect 0x7f1c20002860 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 10 client.174363.objecter resend_mon_ops 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651916783 0 0) 0x7f1c200065c0 con 0x7f1c20002860 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 10 monclient(hunting): my global_id is 174372 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.645+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c400a2860 con 0x7f1c20002860 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3876964067 0 0) 0x7f1c400a2860 con 0x7f1c20002860 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c200065c0 con 0x7f1c20002860 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c30001ac0 con 0x7f1c20002860 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 10 monclient: dump: 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 498347867 0 0) 0x7f1c200065c0 con 0x7f1c20002860 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c2000a160 con 0x7f1c20002860 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1c300057c0 con 0x7f1c20002860 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c20002860 legacy=0x7f1c20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c20002860 legacy=0x7f1c20004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.646+0000 7f1c27fff640 10 client.174363.objecter ms_handle_connect 0x7f1c2c041620 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.652+0000 7f1c45ffd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.652+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1c20009e50 con 0x7f1c20009110 2024-08-19T10:01:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c20009110 legacy=0x7f1c200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 client.174363.objecter ms_handle_connect 0x7f1c20009110 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 client.174363.objecter resend_mon_ops 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146113067 0 0) 0x7f1c20009e50 con 0x7f1c20009110 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient(hunting): my global_id is 174381 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c200065c0 con 0x7f1c20009110 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 810472976 0 0) 0x7f1c200065c0 con 0x7f1c20009110 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c20009e50 con 0x7f1c20009110 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1c30003de0 con 0x7f1c20009110 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient: dump: 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2579208119 0 0) 0x7f1c20009e50 con 0x7f1c20009110 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c2000c3f0 con 0x7f1c20009110 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2890259234) 0x7f1c30004150 con 0x7f1c20009110 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c20009110 legacy=0x7f1c200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.655+0000 7f1c27fff640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c20009110 legacy=0x7f1c200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:39.673 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 149689, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:39.678 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.679 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:39.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:39.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:39.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.687 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:39.690 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.692 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:39.693 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 10 librados: watch_flush enter 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 10 librados: watch_flush exit 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 20 client.174363.objecter shutdown clearing up homeless session... 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 10 client.174363.objecter successfully canceled tick 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1c2c041620 legacy=0x7f1c2c043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 10 monclient: shutdown 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 >> v1:172.21.15.1:6792/0 conn(0x7f1c4013f5c0 legacy=0x7f1c400a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c3ffff640 1 -- 172.21.15.1:0/2763028670 reap_dead start 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c3ffff640 1 -- 172.21.15.1:0/2763028670 reap_dead start 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 shutdown_connections 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 >> 172.21.15.1:0/2763028670 conn(0x7f1c400ac4b0 msgr2=0x7f1c400a4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 shutdown_connections 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 -- 172.21.15.1:0/2763028670 wait complete. 2024-08-19T10:01:39.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.656+0000 7f1c45ffd640 1 librados: shutdown 2024-08-19T10:01:39.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:39.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:01:39.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:01:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.797+0000 7fcc3a333640 10 monclient: get_monmap_and_config 2024-08-19T10:01:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.797+0000 7fcc3a333640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.797+0000 7fcc3a333640 10 monclient: monmap: 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:39.798162+0000 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:39.798162+0000 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.800 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.797+0000 7fcc3a333640 1 Processor -- start 2024-08-19T10:01:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 1 -- start start 2024-08-19T10:01:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient: init 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient: _add_conns ranks=[7,6,8,1,3,5,4,2,0] 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient(hunting): picked mon.noname-i con 0x7fcc340a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient(hunting): picked mon.noname-d con 0x7fcc340a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient(hunting): picked mon.noname-e con 0x7fcc3413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a3b40 con 0x7fcc340a6ba0 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a8130 con 0x7fcc340a67c0 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3406ede0 con 0x7fcc3413f5c0 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc3a333640 10 monclient(hunting): authenticate will time out at 1355.874507s 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc337fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fcc340a6ba0 0x7fcc340a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43022/0 (socket says 172.21.15.1:43022) 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc337fe640 1 -- 172.21.15.1:0/2058790821 learned_addr learned my addr 172.21.15.1:0/2058790821 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.798+0000 7fcc388d0640 1 --1- 172.21.15.1:0/2058790821 >> v1:172.21.15.1:6793/0 conn(0x7fcc3413f5c0 0x7fcc341419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42694/0 (socket says 172.21.15.1:42694) 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc337fe640 1 -- 172.21.15.1:0/2058790821 >> v1:172.21.15.1:6792/0 conn(0x7fcc340a6ba0 legacy=0x7fcc340a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc388d0640 1 -- 172.21.15.1:0/2058790821 >> v1:172.21.15.1:6793/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc33fff640 1 -- 172.21.15.1:0/2058790821 >> v1:172.21.15.82:6792/0 conn(0x7fcc340a67c0 legacy=0x7fcc340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988615072 0 0) 0x7fcc340a3b40 con 0x7fcc340a6ba0 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc32ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc32ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.800+0000 7fcc32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): my global_id is 174390 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc1c003520 con 0x7fcc340a6ba0 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162672380 0 0) 0x7fcc3406ede0 con 0x7fcc3413f5c0 2024-08-19T10:01:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): my global_id is 184130 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc340a3b40 con 0x7fcc3413f5c0 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094821152 0 0) 0x7fcc340a8130 con 0x7fcc340a67c0 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient(hunting): my global_id is 144853 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc3406ede0 con 0x7fcc340a67c0 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1682489669 0 0) 0x7fcc1c003520 con 0x7fcc340a6ba0 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 >> v1:172.21.15.1:6793/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 >> v1:172.21.15.82:6792/0 conn(0x7fcc340a67c0 legacy=0x7fcc340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 monclient: found mon.noname-d 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340a92a0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc28001ae0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: dump: 2024-08-19T10:01:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 5 monclient: authenticate success, global_id 174390 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc280048a0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 1 -- 172.21.15.1:0/2058790821 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc28004e10 con 0x7fcc340a6ba0 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 4 set_mon_vals no callback set 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc32ffd640 10 monclient: dump: 2024-08-19T10:01:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.822 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 10 monclient: shutdown 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 1 -- 172.21.15.1:0/2058790821 >> v1:172.21.15.1:6792/0 conn(0x7fcc340a6ba0 legacy=0x7fcc340a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 1 -- 172.21.15.1:0/2058790821 shutdown_connections 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.801+0000 7fcc3a333640 1 -- 172.21.15.1:0/2058790821 >> 172.21.15.1:0/2058790821 conn(0x7fcc340ac4b0 msgr2=0x7fcc340a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 1 -- 172.21.15.1:0/2058790821 shutdown_connections 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 1 -- 172.21.15.1:0/2058790821 wait complete. 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 10 monclient: monmap: 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.824 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 1 librados: starting msgr at 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 1 librados: starting objecter 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.802+0000 7fcc3a333640 1 Processor -- start 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 1 -- start start 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 1 librados: setting wanted keys 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 1 librados: calling monclient init 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient: init 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient: _add_conns ranks=[5,6,8,1,3,0,2,4,7] 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient(hunting): picked mon.noname-h con 0x7fcc340a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient(hunting): picked mon.noname-d con 0x7fcc340a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient(hunting): picked mon.noname-e con 0x7fcc3413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:39.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3413bdd0 con 0x7fcc340a67c0 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3413af60 con 0x7fcc340a6ba0 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc34139a70 con 0x7fcc3413f5c0 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc3a333640 10 monclient(hunting): authenticate will time out at 1355.879532s 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc337fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fcc340a6ba0 0x7fcc3413c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43026/0 (socket says 172.21.15.1:43026) 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc337fe640 1 -- 172.21.15.1:0/788268276 learned_addr learned my addr 172.21.15.1:0/788268276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.803+0000 7fcc388d0640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcc3413f5c0 0x7fcc341d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42708/0 (socket says 172.21.15.1:42708) 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc388d0640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 client.?.objecter ms_handle_connect 0x7fcc3413f5c0 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc337fe640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6792/0 conn(0x7fcc340a6ba0 legacy=0x7fcc3413c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 client.?.objecter ms_handle_connect 0x7fcc340a6ba0 2024-08-19T10:01:39.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2730742159 0 0) 0x7fcc34139a70 con 0x7fcc3413f5c0 2024-08-19T10:01:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): my global_id is 184139 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc20004eb0 con 0x7fcc3413f5c0 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3451197476 0 0) 0x7fcc3413af60 con 0x7fcc340a6ba0 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 10 monclient(hunting): my global_id is 174399 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.805+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc34139a70 con 0x7fcc340a6ba0 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc33fff640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.82:6791/0 conn(0x7fcc340a67c0 legacy=0x7fcc3413c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 client.?.objecter ms_handle_connect 0x7fcc340a67c0 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2774583217 0 0) 0x7fcc20004eb0 con 0x7fcc3413f5c0 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc3413af60 con 0x7fcc3413f5c0 2024-08-19T10:01:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc2c002020 con 0x7fcc3413f5c0 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): dump: 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1753278167 0 0) 0x7fcc34139a70 con 0x7fcc340a6ba0 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc20004eb0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc28004d90 con 0x7fcc340a6ba0 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): dump: 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.831 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888689646 0 0) 0x7fcc3413bdd0 con 0x7fcc340a67c0 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient(hunting): my global_id is 27356 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc34139a70 con 0x7fcc340a67c0 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1217840750 0 0) 0x7fcc20004eb0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc3413f5c0 legacy=0x7fcc341d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.82:6791/0 conn(0x7fcc340a67c0 legacy=0x7fcc3413c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 monclient: found mon.d 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3413b530 con 0x7fcc340a6ba0 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 5 monclient: authenticate success, global_id 174399 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 10 monclient: _renew_subs 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3413e640 con 0x7fcc340a6ba0 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 10 client.174399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:39.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 10 monclient: _renew_subs 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc3413d010 con 0x7fcc340a6ba0 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc3a333640 1 librados: init done 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc28001d30 con 0x7fcc340a6ba0 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc280048a0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc327fc640 4 set_mon_vals no callback set 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.806+0000 7fcc30ff9640 10 monclient: dump: 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc3a333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcc140065c0 con 0x7fcc14002860 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcc28011a00 con 0x7fcc340a6ba0 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(414..414 src has 299..414) ==== 5465+0+0 (unknown 3895169888 0 0) 0x7fcc2804e1d0 con 0x7fcc340a6ba0 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc30ff9640 10 client.174399.objecter ms_dispatch 0x7fcc34000b90 osd_map(414..414 src has 299..414) 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc30ff9640 3 client.174399.objecter handle_osd_map got epochs [414,414] > 0 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc30ff9640 3 client.174399.objecter handle_osd_map decoding full epoch 414 2024-08-19T10:01:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc30ff9640 20 client.174399.objecter dump_active .. 0 homeless 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.807+0000 7fcc388d0640 10 monclient: get_auth_request con 0x7fcc20041790 auth_method 0 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc33fff640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc14002860 legacy=0x7fcc14004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 client.174399.objecter ms_handle_connect 0x7fcc14002860 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 client.174399.objecter resend_mon_ops 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648720584 0 0) 0x7fcc140065c0 con 0x7fcc14002860 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient(hunting): my global_id is 184148 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc20004eb0 con 0x7fcc14002860 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2016922850 0 0) 0x7fcc20004eb0 con 0x7fcc14002860 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc140065c0 con 0x7fcc14002860 2024-08-19T10:01:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc24001c20 con 0x7fcc14002860 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 10 monclient: dump: 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1174630038 0 0) 0x7fcc140065c0 con 0x7fcc14002860 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.809+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc1400a160 con 0x7fcc14002860 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.810+0000 7fcc30ff9640 10 client.174399.objecter ms_handle_connect 0x7fcc20041790 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.810+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcc240057c0 con 0x7fcc14002860 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.810+0000 7fcc30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.810+0000 7fcc30ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.810+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc14002860 legacy=0x7fcc14004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.810+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc14002860 legacy=0x7fcc14004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.816+0000 7fcc3a333640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:39.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.816+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcc14009e70 con 0x7fcc14009130 2024-08-19T10:01:39.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc33fff640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc14009130 legacy=0x7fcc140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 10 client.174399.objecter ms_handle_connect 0x7fcc14009130 2024-08-19T10:01:39.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 10 client.174399.objecter resend_mon_ops 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256541980 0 0) 0x7fcc14009e70 con 0x7fcc14009130 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 10 monclient(hunting): my global_id is 184157 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc140065c0 con 0x7fcc14009130 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.818+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1014254669 0 0) 0x7fcc140065c0 con 0x7fcc14009130 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc14009e70 con 0x7fcc14009130 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcc24003cc0 con 0x7fcc14009130 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 10 monclient: dump: 2024-08-19T10:01:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2040303467 0 0) 0x7fcc14009e70 con 0x7fcc14009130 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcc1400c3f0 con 0x7fcc14009130 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 1739417924) 0x7fcc24005a10 con 0x7fcc14009130 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc14009130 legacy=0x7fcc140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc30ff9640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6793/0 conn(0x7fcc14009130 legacy=0x7fcc140095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:39.844 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17067, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:39.845 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.846 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.847 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:39.848 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:39.849 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.850 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.851 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.853 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:39.854 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.859 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.862 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:39.864 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:39.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:39.866 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:39.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 10 librados: watch_flush enter 2024-08-19T10:01:39.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 10 librados: watch_flush exit 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 20 client.174399.objecter shutdown clearing up homeless session... 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 10 client.174399.objecter successfully canceled tick 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcc20041790 legacy=0x7fcc20043c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 10 monclient: shutdown 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 >> v1:172.21.15.1:6792/0 conn(0x7fcc340a6ba0 legacy=0x7fcc3413c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc388d0640 1 -- 172.21.15.1:0/788268276 reap_dead start 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc388d0640 1 -- 172.21.15.1:0/788268276 reap_dead start 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 shutdown_connections 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 >> 172.21.15.1:0/788268276 conn(0x7fcc340ac4b0 msgr2=0x7fcc34141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 shutdown_connections 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.819+0000 7fcc3a333640 1 -- 172.21.15.1:0/788268276 wait complete. 2024-08-19T10:01:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.820+0000 7fcc3a333640 1 librados: shutdown 2024-08-19T10:01:39.870 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:39.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:01:39.870 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:01:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.969+0000 7f474de60640 10 monclient: get_monmap_and_config 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.969+0000 7f474de60640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.969+0000 7f474de60640 10 monclient: monmap: 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:39.970065+0000 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:39.970065+0000 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.972 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.969+0000 7f474de60640 1 Processor -- start 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.969+0000 7f474de60640 1 -- start start 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.969+0000 7f474de60640 10 monclient: init 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient: _add_conns ranks=[3,1,6,8,5,4,2,0,7] 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient(hunting): picked mon.noname-g con 0x7f47480a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient(hunting): picked mon.noname-f con 0x7f47480694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient(hunting): picked mon.noname-d con 0x7f47480a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47480a3c70 con 0x7f47480694b0 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47480a33d0 con 0x7f47480a6830 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47480a36b0 con 0x7f47480a46a0 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f474de60640 10 monclient(hunting): authenticate will time out at 1356.046412s 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f4747fff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f47480a46a0 0x7f4748141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43036/0 (socket says 172.21.15.1:43036) 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.970+0000 7f4747fff640 1 -- 172.21.15.1:0/1841151351 learned_addr learned my addr 172.21.15.1:0/1841151351 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.972+0000 7f47477fe640 1 -- 172.21.15.1:0/1841151351 >> v1:172.21.15.82:6790/0 conn(0x7f47480a6830 legacy=0x7f47480a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.972+0000 7f4747fff640 1 -- 172.21.15.1:0/1841151351 >> v1:172.21.15.1:6792/0 conn(0x7f47480a46a0 legacy=0x7f4748141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.972+0000 7f4746ffd640 1 -- 172.21.15.1:0/1841151351 >> v1:172.21.15.82:6789/0 conn(0x7f47480694b0 legacy=0x7f47480a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3987181433 0 0) 0x7f47480a33d0 con 0x7f47480a6830 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): my global_id is 184134 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4730003520 con 0x7f47480a6830 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119494856 0 0) 0x7f47480a36b0 con 0x7f47480a46a0 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): my global_id is 174408 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47480a33d0 con 0x7f47480a46a0 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3078396759 0 0) 0x7f47480a3c70 con 0x7f47480694b0 2024-08-19T10:01:39.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient(hunting): my global_id is 174331 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47480a36b0 con 0x7f47480694b0 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2202789262 0 0) 0x7f4730003520 con 0x7f47480a6830 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 >> v1:172.21.15.1:6792/0 conn(0x7f47480a46a0 legacy=0x7f4748141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 >> v1:172.21.15.82:6789/0 conn(0x7f47480694b0 legacy=0x7f47480a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 monclient: found mon.noname-g 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474806dbb0 con 0x7f47480a6830 2024-08-19T10:01:39.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4738001b30 con 0x7f47480a6830 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f474de60640 5 monclient: authenticate success, global_id 184134 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f474de60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: dump: 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.986 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f474de60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4738003610 con 0x7f47480a6830 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:39.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 1 -- 172.21.15.1:0/1841151351 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4738003b80 con 0x7f47480a6830 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f47467fc640 10 monclient: dump: 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.988 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f474de60640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.973+0000 7f474de60640 4 set_mon_vals no callback set 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 10 monclient: shutdown 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 -- 172.21.15.1:0/1841151351 >> v1:172.21.15.82:6790/0 conn(0x7f47480a6830 legacy=0x7f47480a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 -- 172.21.15.1:0/1841151351 shutdown_connections 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 -- 172.21.15.1:0/1841151351 >> 172.21.15.1:0/1841151351 conn(0x7f47480a9020 msgr2=0x7f4748068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:39.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 -- 172.21.15.1:0/1841151351 shutdown_connections 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 -- 172.21.15.1:0/1841151351 wait complete. 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 10 monclient: build_initial_monmap 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 10 monclient: monmap: 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:39.990 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 librados: starting msgr at 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.974+0000 7f474de60640 1 librados: starting objecter 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 Processor -- start 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 -- start start 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 librados: setting wanted keys 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 librados: calling monclient init 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient: init 2024-08-19T10:01:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient: _add_conns ranks=[7,1,0,2,3,5,6,4,8] 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient(hunting): picked mon.noname-i con 0x7f47480694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient(hunting): picked mon.noname-f con 0x7f47480a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient(hunting): picked mon.noname-a con 0x7f47480a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4748138500 con 0x7f47480a6830 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47480689e0 con 0x7f47480a46a0 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f474813a580 con 0x7f47480694b0 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f474de60640 10 monclient(hunting): authenticate will time out at 1356.051779s 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f4747fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f47480a6830 0x7f47481d5ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47880/0 (socket says 172.21.15.1:47880) 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.975+0000 7f4747fff640 1 -- 172.21.15.1:0/90093403 learned_addr learned my addr 172.21.15.1:0/90093403 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f4747fff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.1:6789/0 conn(0x7f47480a6830 legacy=0x7f47481d5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 client.?.objecter ms_handle_connect 0x7f47480a6830 2024-08-19T10:01:39.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227225253 0 0) 0x7f4748138500 con 0x7f47480a6830 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): my global_id is 174393 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f4746ffd640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f47480a46a0 legacy=0x7f4748141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4734004eb0 con 0x7f47480a6830 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 client.?.objecter ms_handle_connect 0x7f47480a46a0 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 client.?.objecter ms_handle_connect 0x7f47480694b0 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f47477fe640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6792/0 conn(0x7f47480694b0 legacy=0x7f47481cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298235058 0 0) 0x7f473c001350 con 0x7f47480a46a0 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): my global_id is 174340 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4748138500 con 0x7f47480a46a0 2024-08-19T10:01:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582934692 0 0) 0x7f474813a580 con 0x7f47480694b0 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): my global_id is 144862 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f473c001350 con 0x7f47480694b0 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3063439866 0 0) 0x7f4734004eb0 con 0x7f47480a6830 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f474813a580 con 0x7f47480a6830 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4740004b60 con 0x7f47480a6830 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): dump: 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:39.996 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2754403552 0 0) 0x7f473c001350 con 0x7f47480694b0 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4734004eb0 con 0x7f47480694b0 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2877552606 0 0) 0x7f474813a580 con 0x7f47480a6830 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6792/0 conn(0x7f47480694b0 legacy=0x7f47481cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f47480a46a0 legacy=0x7f4748141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 monclient: found mon.a 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474813a860 con 0x7f47480a6830 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: _finish_auth 0 2024-08-19T10:01:39.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:01:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 5 monclient: authenticate success, global_id 174393 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 10 monclient: _renew_subs 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47480a9fc0 con 0x7f47480a6830 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 10 client.174393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 10 monclient: _renew_subs 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4748142500 con 0x7f47480a6830 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f474de60640 1 librados: init done 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4740004fb0 con 0x7f47480a6830 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47400054e0 con 0x7f47480a6830 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f4745ffb640 4 set_mon_vals no callback set 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.978+0000 7f472ffff640 10 monclient: dump: 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:40.001 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f474de60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f47280065c0 con 0x7f4728002860 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4740005680 con 0x7f47480a6830 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(414..414 src has 299..414) ==== 5465+0+0 (unknown 3895169888 0 0) 0x7f474004e570 con 0x7f47480a6830 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f472ffff640 10 client.174393.objecter ms_dispatch 0x7f4748000b90 osd_map(414..414 src has 299..414) 2024-08-19T10:01:40.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f472ffff640 3 client.174393.objecter handle_osd_map got epochs [414,414] > 0 2024-08-19T10:01:40.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.979+0000 7f472ffff640 3 client.174393.objecter handle_osd_map decoding full epoch 414 2024-08-19T10:01:40.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.980+0000 7f472ffff640 20 client.174393.objecter dump_active .. 0 homeless 2024-08-19T10:01:40.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.980+0000 7f4746ffd640 10 monclient: get_auth_request con 0x7f47340410c0 auth_method 0 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 client.174393.objecter ms_handle_connect 0x7f4728002860 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f47477fe640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f4728002860 legacy=0x7f4728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 client.174393.objecter resend_mon_ops 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2198512119 0 0) 0x7f47280065c0 con 0x7f4728002860 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient(hunting): my global_id is 174349 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f474813a580 con 0x7f4728002860 2024-08-19T10:01:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 client.174393.objecter ms_handle_connect 0x7f47340410c0 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1446844952 0 0) 0x7f474813a580 con 0x7f4728002860 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47280065c0 con 0x7f4728002860 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47380015d0 con 0x7f4728002860 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 10 monclient: dump: 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1283228148 0 0) 0x7f47280065c0 con 0x7f4728002860 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.982+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f472800a160 con 0x7f4728002860 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.983+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4738003610 con 0x7f4728002860 2024-08-19T10:01:40.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.983+0000 7f472ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.983+0000 7f472ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.983+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f4728002860 legacy=0x7f4728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.983+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f4728002860 legacy=0x7f4728004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.989+0000 7f474de60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.989+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4728009e70 con 0x7f4728009130 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 client.174393.objecter ms_handle_connect 0x7f4728009130 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f47477fe640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f4728009130 legacy=0x7f47280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 client.174393.objecter resend_mon_ops 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433141970 0 0) 0x7f4728009e70 con 0x7f4728009130 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:40.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient(hunting): my global_id is 174358 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47280065c0 con 0x7f4728009130 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1662310775 0 0) 0x7f47280065c0 con 0x7f4728009130 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4728009e70 con 0x7f4728009130 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4738005b00 con 0x7f4728009130 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 10 monclient: dump: 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:40.010 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:40.011 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3472871815 0 0) 0x7f4728009e70 con 0x7f4728009130 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.992+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f472800c410 con 0x7f4728009130 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7852 (unknown 0 0 2580589490) 0x7f4738007ec0 con 0x7f4728009130 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f472ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f472ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:40.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f4728009130 legacy=0x7f47280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f472ffff640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6789/0 conn(0x7f4728009130 legacy=0x7f47280095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:40.013 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:40.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:40.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:40.013 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 150050, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:40.014 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:40.015 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:40.016 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:40.018 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:40.018 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:40.018 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:40.018 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:40.018 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:40.018 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:40.019 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:40.021 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:40.026 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:40.027 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:40.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.032 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:40.033 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:40.034 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:40.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:40.036 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:40.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:40.038 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:40.038 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 10 librados: watch_flush enter 2024-08-19T10:01:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 10 librados: watch_flush exit 2024-08-19T10:01:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 20 client.174393.objecter shutdown clearing up homeless session... 2024-08-19T10:01:40.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 10 client.174393.objecter successfully canceled tick 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.82:6812/2549104415 conn(0x7f47340410c0 legacy=0x7f4734043580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 10 monclient: shutdown 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 >> v1:172.21.15.1:6789/0 conn(0x7f47480a6830 legacy=0x7f47481d5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f4747fff640 1 -- 172.21.15.1:0/90093403 reap_dead start 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 shutdown_connections 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 >> 172.21.15.1:0/90093403 conn(0x7f47480a9020 msgr2=0x7f47480a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:40.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 shutdown_connections 2024-08-19T10:01:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 -- 172.21.15.1:0/90093403 wait complete. 2024-08-19T10:01:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:39.993+0000 7f474de60640 1 librados: shutdown 2024-08-19T10:01:40.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:40.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:01:40.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:01:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.143+0000 7f5688112640 10 monclient: get_monmap_and_config 2024-08-19T10:01:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.143+0000 7f5688112640 10 monclient: build_initial_monmap 2024-08-19T10:01:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.143+0000 7f5688112640 10 monclient: monmap: 2024-08-19T10:01:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:40.144044+0000 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:40.144044+0000 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:40.146 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.143+0000 7f5688112640 1 Processor -- start 2024-08-19T10:01:40.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.143+0000 7f5688112640 1 -- start start 2024-08-19T10:01:40.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.143+0000 7f5688112640 10 monclient: init 2024-08-19T10:01:40.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:40.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient: _add_conns ranks=[7,2,3,8,5,4,1,0,6] 2024-08-19T10:01:40.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient(hunting): picked mon.noname-i con 0x7f56800a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient(hunting): picked mon.noname-b con 0x7f56800694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient(hunting): picked mon.noname-g con 0x7f56800a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800a3c70 con 0x7f56800694b0 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800a33d0 con 0x7f56800a46a0 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800a36b0 con 0x7f56800a6830 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f5688112640 10 monclient(hunting): authenticate will time out at 1356.220403s 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f56856ad640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f56800694b0 0x7f56800a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42728/0 (socket says 172.21.15.1:42728) 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.144+0000 7f56856ad640 1 -- 172.21.15.1:0/1415519081 learned_addr learned my addr 172.21.15.1:0/1415519081 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.146+0000 7f56856ad640 1 -- 172.21.15.1:0/1415519081 >> v1:172.21.15.1:6790/0 conn(0x7f56800694b0 legacy=0x7f56800a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.146+0000 7f56866af640 1 -- 172.21.15.1:0/1415519081 >> v1:172.21.15.82:6790/0 conn(0x7f56800a46a0 legacy=0x7f5680141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.146+0000 7f5685eae640 1 -- 172.21.15.1:0/1415519081 >> v1:172.21.15.82:6792/0 conn(0x7f56800a6830 legacy=0x7f56800a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210642878 0 0) 0x7f56800a36b0 con 0x7f56800a6830 2024-08-19T10:01:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient(hunting): _negotiate 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient(hunting): my global_id is 144871 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5668003520 con 0x7f56800a6830 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 901382691 0 0) 0x7f5668003520 con 0x7f56800a6830 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 >> v1:172.21.15.82:6790/0 conn(0x7f56800a46a0 legacy=0x7f5680141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 >> v1:172.21.15.1:6790/0 conn(0x7f56800694b0 legacy=0x7f56800a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: _finish_hunting 0 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 monclient: found mon.noname-i 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568006dbb0 con 0x7f56800a6830 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: _finish_auth 0 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: _check_auth_tickets 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5678001b30 con 0x7f56800a6830 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5688112640 5 monclient: authenticate success, global_id 144871 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5688112640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:01:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: dump: 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5688112640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5678003610 con 0x7f56800a6830 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 1 -- 172.21.15.1:0/1415519081 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5678003b80 con 0x7f56800a6830 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5688112640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5688112640 4 set_mon_vals no callback set 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.147+0000 7f5684eac640 10 monclient: dump: 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 10 monclient: shutdown 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 -- 172.21.15.1:0/1415519081 >> v1:172.21.15.82:6792/0 conn(0x7f56800a6830 legacy=0x7f56800a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 -- 172.21.15.1:0/1415519081 shutdown_connections 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 -- 172.21.15.1:0/1415519081 >> 172.21.15.1:0/1415519081 conn(0x7f56800a9020 msgr2=0x7f5680068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 -- 172.21.15.1:0/1415519081 shutdown_connections 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 -- 172.21.15.1:0/1415519081 wait complete. 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 10 monclient: build_initial_monmap 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 10 monclient: monmap: 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 librados: starting msgr at 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.148+0000 7f5688112640 1 librados: starting objecter 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 Processor -- start 2024-08-19T10:01:40.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 -- start start 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 librados: setting wanted keys 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 librados: calling monclient init 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient: init 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient: _add_conns ranks=[6,8,1,7,3,4,2,0,5] 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient(hunting): picked mon.noname-d con 0x7f56800694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient(hunting): picked mon.noname-e con 0x7f56800a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient(hunting): picked mon.noname-f con 0x7f56800a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568013ae30 con 0x7f56800a6830 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56801cf1d0 con 0x7f56800694b0 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800a9f30 con 0x7f56800a46a0 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5688112640 10 monclient(hunting): authenticate will time out at 1356.225688s 2024-08-19T10:01:40.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5685eae640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f56800694b0 0x7f56801ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43048/0 (socket says 172.21.15.1:43048) 2024-08-19T10:01:40.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f5685eae640 1 -- 172.21.15.1:0/3637295769 learned_addr learned my addr 172.21.15.1:0/3637295769 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:40.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.149+0000 7f56856ad640 1 --1- 172.21.15.1:0/3637295769 >> v1:172.21.15.1:6793/0 conn(0x7f56800a46a0 0x7f56801ceac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42734/0 (socket says 172.21.15.1:42734) 2024-08-19T10:01:40.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.151+0000 7f56856ad640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.1:6793/0 conn(0x7f56800a46a0 legacy=0x7f56801ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.151+0000 7f5685eae640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.1:6792/0 conn(0x7f56800694b0 legacy=0x7f56801ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.151+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f56800a46a0 2024-08-19T10:01:40.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.151+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:40.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.151+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f56800694b0 2024-08-19T10:01:40.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.151+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:40.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.152+0000 7f56866af640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6789/0 conn(0x7f56800a6830 legacy=0x7f56801d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:40.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.152+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f56800a6830 2024-08-19T10:01:40.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.152+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726752072 0 0) 0x7f568013ae30 con 0x7f56800a6830 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): my global_id is 174367 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f566c004eb0 con 0x7f56800a6830 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466919716 0 0) 0x7f56800a9f30 con 0x7f56800a46a0 2024-08-19T10:01:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 10 monclient(hunting): my global_id is 184166 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.677+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568013ae30 con 0x7f56800a46a0 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695073102 0 0) 0x7f56801cf1d0 con 0x7f56800694b0 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): my global_id is 174417 2024-08-19T10:01:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f566c0053a0 con 0x7f56800694b0 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 524971198 0 0) 0x7f566c0053a0 con 0x7f56800694b0 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56801cf1d0 con 0x7f56800694b0 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5678003b00 con 0x7f56800694b0 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): dump: 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:40.678+0000 7f567e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1259312891 0 0) 0x7f56801cf1d0 con 0x7f56800694b0 2024-08-19T10:01:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.1:6793/0 conn(0x7f56800a46a0 legacy=0x7f56801ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6789/0 conn(0x7f56800a6830 legacy=0x7f56801d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 1 monclient: found mon.d 2024-08-19T10:01:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5680142c60 con 0x7f56800694b0 2024-08-19T10:01:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f567e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 5 monclient: authenticate success, global_id 174417 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 10 monclient: _renew_subs 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56801386e0 con 0x7f56800694b0 2024-08-19T10:01:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 10 client.174417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:41.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 10 monclient: _renew_subs 2024-08-19T10:01:41.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f5680139610 con 0x7f56800694b0 2024-08-19T10:01:41.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.076+0000 7f5688112640 1 librados: init done 2024-08-19T10:01:41.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5678001b50 con 0x7f56800694b0 2024-08-19T10:01:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5678004290 con 0x7f56800694b0 2024-08-19T10:01:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567ffff640 4 set_mon_vals no callback set 2024-08-19T10:01:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.077+0000 7f567e7fc640 10 monclient: dump: 2024-08-19T10:01:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.093 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.093 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.078+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5678011a80 con 0x7f56800694b0 2024-08-19T10:01:41.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.078+0000 7f5688112640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.078+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f56600065c0 con 0x7f5660002860 2024-08-19T10:01:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.079+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(415..415 src has 299..415) ==== 5481+0+0 (unknown 518798970 0 0) 0x7f567804e190 con 0x7f56800694b0 2024-08-19T10:01:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.079+0000 7f567e7fc640 10 client.174417.objecter ms_dispatch 0x7f5680000b90 osd_map(415..415 src has 299..415) 2024-08-19T10:01:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.079+0000 7f567e7fc640 3 client.174417.objecter handle_osd_map got epochs [415,415] > 0 2024-08-19T10:01:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.079+0000 7f567e7fc640 3 client.174417.objecter handle_osd_map decoding full epoch 415 2024-08-19T10:01:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.079+0000 7f567e7fc640 20 client.174417.objecter dump_active .. 0 homeless 2024-08-19T10:01:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.079+0000 7f56866af640 10 monclient: get_auth_request con 0x7f566c041310 auth_method 0 2024-08-19T10:01:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f56856ad640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6790/0 conn(0x7f5660002860 legacy=0x7f5660004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 10 client.174417.objecter ms_handle_connect 0x7f5660002860 2024-08-19T10:01:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 10 client.174417.objecter resend_mon_ops 2024-08-19T10:01:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104193187 0 0) 0x7f56600065c0 con 0x7f5660002860 2024-08-19T10:01:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 10 monclient(hunting): my global_id is 184143 2024-08-19T10:01:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.080+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56801cf1d0 con 0x7f5660002860 2024-08-19T10:01:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3305145271 0 0) 0x7f56801cf1d0 con 0x7f5660002860 2024-08-19T10:01:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56600065c0 con 0x7f5660002860 2024-08-19T10:01:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5674001f10 con 0x7f5660002860 2024-08-19T10:01:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 10 monclient: dump: 2024-08-19T10:01:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.100 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 516055057 0 0) 0x7f56600065c0 con 0x7f5660002860 2024-08-19T10:01:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f566000a160 con 0x7f5660002860 2024-08-19T10:01:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 10 client.174417.objecter ms_handle_connect 0x7f566c041310 2024-08-19T10:01:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5674004b60 con 0x7f5660002860 2024-08-19T10:01:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6790/0 conn(0x7f5660002860 legacy=0x7f5660004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.081+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6790/0 conn(0x7f5660002860 legacy=0x7f5660004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:41.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.088+0000 7f5688112640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.088+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5660009e70 con 0x7f5660009130 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f56856ad640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6790/0 conn(0x7f5660009130 legacy=0x7f56600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 client.174417.objecter ms_handle_connect 0x7f5660009130 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 client.174417.objecter resend_mon_ops 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 21667971 0 0) 0x7f5660009e70 con 0x7f5660009130 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient(hunting): my global_id is 184152 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56600065c0 con 0x7f5660009130 2024-08-19T10:01:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1902853599 0 0) 0x7f56600065c0 con 0x7f5660009130 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5660009e70 con 0x7f5660009130 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5674005350 con 0x7f5660009130 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 10 monclient: dump: 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3009310417 0 0) 0x7f5660009e70 con 0x7f5660009130 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.091+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f566000c3f0 con 0x7f5660009130 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3544814258) 0x7f56740055d0 con 0x7f5660009130 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f567e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f567e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6790/0 conn(0x7f5660009130 legacy=0x7f56600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f567e7fc640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6790/0 conn(0x7f5660009130 legacy=0x7f56600095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 18345, 2024-08-19T10:01:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:41.113 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.114 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.116 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.118 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:41.119 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:41.133 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 10 librados: watch_flush enter 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 10 librados: watch_flush exit 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 20 client.174417.objecter shutdown clearing up homeless session... 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 10 client.174417.objecter successfully canceled tick 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.82:6812/2549104415 conn(0x7f566c041310 legacy=0x7f566c0437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 10 monclient: shutdown 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 >> v1:172.21.15.1:6792/0 conn(0x7f56800694b0 legacy=0x7f56801ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f56866af640 1 -- 172.21.15.1:0/3637295769 reap_dead start 2024-08-19T10:01:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f56866af640 1 -- 172.21.15.1:0/3637295769 reap_dead start 2024-08-19T10:01:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 shutdown_connections 2024-08-19T10:01:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 >> 172.21.15.1:0/3637295769 conn(0x7f56800a9020 msgr2=0x7f56800a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 shutdown_connections 2024-08-19T10:01:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.092+0000 7f5688112640 1 -- 172.21.15.1:0/3637295769 wait complete. 2024-08-19T10:01:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.093+0000 7f5688112640 1 librados: shutdown 2024-08-19T10:01:41.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:41.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:01:41.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.242+0000 7f69fac5f640 10 monclient: get_monmap_and_config 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.242+0000 7f69fac5f640 10 monclient: build_initial_monmap 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.242+0000 7f69fac5f640 10 monclient: monmap: 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:41.243719+0000 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:41.243719+0000 2024-08-19T10:01:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:41.245 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.243+0000 7f69fac5f640 1 Processor -- start 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.243+0000 7f69fac5f640 1 -- start start 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.243+0000 7f69fac5f640 10 monclient: init 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient: _add_conns ranks=[1,4,7,0,5,3,8,6,2] 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient(hunting): picked mon.noname-f con 0x7f69f40a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient(hunting): picked mon.noname-c con 0x7f69f40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient(hunting): picked mon.noname-i con 0x7f69f413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f40a3b40 con 0x7f69f40a67c0 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f40a8130 con 0x7f69f40a6ba0 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f406ede0 con 0x7f69f413f5c0 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69fac5f640 10 monclient(hunting): authenticate will time out at 1357.320118s 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69f3fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f69f40a6ba0 0x7f69f40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44716/0 (socket says 172.21.15.1:44716) 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.244+0000 7f69f3fff640 1 -- 172.21.15.1:0/2160694159 learned_addr learned my addr 172.21.15.1:0/2160694159 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f3fff640 1 -- 172.21.15.1:0/2160694159 >> v1:172.21.15.1:6791/0 conn(0x7f69f40a6ba0 legacy=0x7f69f40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f89fb640 1 -- 172.21.15.1:0/2160694159 >> v1:172.21.15.82:6789/0 conn(0x7f69f40a67c0 legacy=0x7f69f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878060795 0 0) 0x7f69f40a8130 con 0x7f69f40a6ba0 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f91fc640 1 -- 172.21.15.1:0/2160694159 >> v1:172.21.15.82:6792/0 conn(0x7f69f413f5c0 legacy=0x7f69f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): my global_id is 27742 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69dc003520 con 0x7f69f40a6ba0 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 787910973 0 0) 0x7f69f406ede0 con 0x7f69f413f5c0 2024-08-19T10:01:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): my global_id is 144880 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f40a8130 con 0x7f69f413f5c0 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1697178934 0 0) 0x7f69f40a3b40 con 0x7f69f40a67c0 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient(hunting): my global_id is 174376 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f406ede0 con 0x7f69f40a67c0 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3485540794 0 0) 0x7f69dc003520 con 0x7f69f40a6ba0 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 >> v1:172.21.15.82:6792/0 conn(0x7f69f413f5c0 legacy=0x7f69f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 >> v1:172.21.15.82:6789/0 conn(0x7f69f40a67c0 legacy=0x7f69f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 monclient: found mon.noname-c 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f40a92a0 con 0x7f69f40a6ba0 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69e8001ae0 con 0x7f69f40a6ba0 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69f37fe640 10 monclient: dump: 2024-08-19T10:01:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69fac5f640 5 monclient: authenticate success, global_id 27742 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.246+0000 7f69fac5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69e80048a0 con 0x7f69f40a6ba0 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69f37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69f37fe640 1 -- 172.21.15.1:0/2160694159 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69e8004e10 con 0x7f69f40a6ba0 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69f37fe640 10 monclient: dump: 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.260 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:41.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 4 set_mon_vals no callback set 2024-08-19T10:01:41.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 10 monclient: shutdown 2024-08-19T10:01:41.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 1 -- 172.21.15.1:0/2160694159 >> v1:172.21.15.1:6791/0 conn(0x7f69f40a6ba0 legacy=0x7f69f40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 1 -- 172.21.15.1:0/2160694159 shutdown_connections 2024-08-19T10:01:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 1 -- 172.21.15.1:0/2160694159 >> 172.21.15.1:0/2160694159 conn(0x7f69f40ac4b0 msgr2=0x7f69f40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 1 -- 172.21.15.1:0/2160694159 shutdown_connections 2024-08-19T10:01:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.247+0000 7f69fac5f640 1 -- 172.21.15.1:0/2160694159 wait complete. 2024-08-19T10:01:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 10 monclient: build_initial_monmap 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 10 monclient: monmap: 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 1 librados: starting msgr at 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 1 librados: starting objecter 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 1 Processor -- start 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 1 -- start start 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 1 librados: setting wanted keys 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 1 librados: calling monclient init 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.248+0000 7f69fac5f640 10 monclient: init 2024-08-19T10:01:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient: _add_conns ranks=[0,3,1,5,4,7,8,2,6] 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient(hunting): picked mon.noname-a con 0x7f69f40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient(hunting): picked mon.noname-g con 0x7f69f40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient(hunting): picked mon.noname-f con 0x7f69f413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f413ae40 con 0x7f69f40a67c0 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f413b410 con 0x7f69f413f5c0 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f4139950 con 0x7f69f40a6ba0 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69fac5f640 10 monclient(hunting): authenticate will time out at 1357.325210s 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69f89fb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f69f40a67c0 0x7f69f413c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47882/0 (socket says 172.21.15.1:47882) 2024-08-19T10:01:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.249+0000 7f69f89fb640 1 -- 172.21.15.1:0/4100202102 learned_addr learned my addr 172.21.15.1:0/4100202102 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f89fb640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.1:6789/0 conn(0x7f69f40a67c0 legacy=0x7f69f413c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 client.?.objecter ms_handle_connect 0x7f69f40a67c0 2024-08-19T10:01:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f3fff640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6790/0 conn(0x7f69f40a6ba0 legacy=0x7f69f413c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 client.?.objecter ms_handle_connect 0x7f69f40a6ba0 2024-08-19T10:01:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 client.?.objecter ms_handle_connect 0x7f69f413f5c0 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f91fc640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6789/0 conn(0x7f69f413f5c0 legacy=0x7f69f41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 608215583 0 0) 0x7f69f413ae40 con 0x7f69f40a67c0 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): my global_id is 174402 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69e0004ee0 con 0x7f69f40a67c0 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091591055 0 0) 0x7f69f4139950 con 0x7f69f40a6ba0 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): my global_id is 184161 2024-08-19T10:01:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f413ae40 con 0x7f69f40a6ba0 2024-08-19T10:01:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1797485233 0 0) 0x7f69f40a8130 con 0x7f69f413f5c0 2024-08-19T10:01:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): my global_id is 174385 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f4139950 con 0x7f69f413f5c0 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3423690710 0 0) 0x7f69e0004ee0 con 0x7f69f40a67c0 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69f40a8130 con 0x7f69f40a67c0 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69e4003610 con 0x7f69f40a67c0 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): dump: 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.251+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2981585671 0 0) 0x7f69f4139950 con 0x7f69f413f5c0 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69e0004ee0 con 0x7f69f413f5c0 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69ec002040 con 0x7f69f413f5c0 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient(hunting): dump: 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1031726931 0 0) 0x7f69f40a8130 con 0x7f69f40a67c0 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6790/0 conn(0x7f69f40a6ba0 legacy=0x7f69f413c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6789/0 conn(0x7f69f413f5c0 legacy=0x7f69f41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 monclient: found mon.a 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f413bcb0 con 0x7f69f40a67c0 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 5 monclient: authenticate success, global_id 174402 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 10 monclient: _renew_subs 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69f413e520 con 0x7f69f40a67c0 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 10 client.174402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 10 monclient: _renew_subs 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f69f413cfd0 con 0x7f69f40a67c0 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 1 librados: init done 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69e4003b90 con 0x7f69f40a67c0 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69e40040c0 con 0x7f69f40a67c0 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f2ffd640 4 set_mon_vals no callback set 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 10 monclient: dump: 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f69e4004260 con 0x7f69f40a67c0 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.252+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f69d40065c0 con 0x7f69d4002860 2024-08-19T10:01:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.253+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(415..415 src has 299..415) ==== 5481+0+0 (unknown 518798970 0 0) 0x7f69e404e1f0 con 0x7f69f40a67c0 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.253+0000 7f69f17fa640 10 client.174402.objecter ms_dispatch 0x7f69f4000b90 osd_map(415..415 src has 299..415) 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.253+0000 7f69f17fa640 3 client.174402.objecter handle_osd_map got epochs [415,415] > 0 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.253+0000 7f69f17fa640 3 client.174402.objecter handle_osd_map decoding full epoch 415 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.253+0000 7f69f17fa640 20 client.174402.objecter dump_active .. 0 homeless 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.253+0000 7f69f91fc640 10 monclient: get_auth_request con 0x7f69e0041520 auth_method 0 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f3fff640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6791/0 conn(0x7f69d4002860 legacy=0x7f69d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 client.174402.objecter ms_handle_connect 0x7f69d4002860 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 client.174402.objecter resend_mon_ops 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202577503 0 0) 0x7f69d40065c0 con 0x7f69d4002860 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient(hunting): my global_id is 27365 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f40a8130 con 0x7f69d4002860 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 client.174402.objecter ms_handle_connect 0x7f69e0041520 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2183718652 0 0) 0x7f69f40a8130 con 0x7f69d4002860 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69d40065c0 con 0x7f69d4002860 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69e8001d30 con 0x7f69d4002860 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 10 monclient: dump: 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.288 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2773168047 0 0) 0x7f69d40065c0 con 0x7f69d4002860 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.255+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69d400a160 con 0x7f69d4002860 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.256+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f69e8005490 con 0x7f69d4002860 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.256+0000 7f69f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.256+0000 7f69f17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.256+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6791/0 conn(0x7f69d4002860 legacy=0x7f69d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.256+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6791/0 conn(0x7f69d4002860 legacy=0x7f69d4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.262+0000 7f69fac5f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:41.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.262+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f69d4009e70 con 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f3fff640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6791/0 conn(0x7f69d4009130 legacy=0x7f69d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 10 client.174402.objecter ms_handle_connect 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 10 client.174402.objecter resend_mon_ops 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297691917 0 0) 0x7f69d4009e70 con 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 10 monclient(hunting): my global_id is 27374 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69d40065c0 con 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2209544871 0 0) 0x7f69d40065c0 con 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69d4009e70 con 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.265+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f69e8005c90 con 0x7f69d4009130 2024-08-19T10:01:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 10 monclient: dump: 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4175751157 0 0) 0x7f69d4009e70 con 0x7f69d4009130 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69d400c410 con 0x7f69d4009130 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3187249586) 0x7f69e8005fa0 con 0x7f69d4009130 2024-08-19T10:01:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6791/0 conn(0x7f69d4009130 legacy=0x7f69d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69f17fa640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6791/0 conn(0x7f69d4009130 legacy=0x7f69d40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:41.293 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 707532, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:01:41.294 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.295 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.296 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:41.297 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:41.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:41.302 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.303 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.306 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.307 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.307 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:41.307 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:41.307 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.310 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.310 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:41.311 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:41.313 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.317 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:41.317 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:41.317 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:41.318 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.320 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:41.321 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 10 librados: watch_flush enter 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 10 librados: watch_flush exit 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 20 client.174402.objecter shutdown clearing up homeless session... 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 10 client.174402.objecter successfully canceled tick 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.82:6812/2549104415 conn(0x7f69e0041520 legacy=0x7f69e00439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 10 monclient: shutdown 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.266+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 >> v1:172.21.15.1:6789/0 conn(0x7f69f40a67c0 legacy=0x7f69f413c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69f91fc640 1 -- 172.21.15.1:0/4100202102 reap_dead start 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69f91fc640 1 -- 172.21.15.1:0/4100202102 reap_dead start 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 shutdown_connections 2024-08-19T10:01:41.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 >> 172.21.15.1:0/4100202102 conn(0x7f69f40ac4b0 msgr2=0x7f69f413ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 shutdown_connections 2024-08-19T10:01:41.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69fac5f640 1 -- 172.21.15.1:0/4100202102 wait complete. 2024-08-19T10:01:41.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.267+0000 7f69fac5f640 1 librados: shutdown 2024-08-19T10:01:41.325 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:41.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:01:41.325 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:01:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.425+0000 7f70c029f640 10 monclient: get_monmap_and_config 2024-08-19T10:01:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.426+0000 7f70c029f640 10 monclient: build_initial_monmap 2024-08-19T10:01:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.426+0000 7f70c029f640 10 monclient: monmap: 2024-08-19T10:01:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:41.426941+0000 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:41.426941+0000 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:41.429 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.426+0000 7f70c029f640 1 Processor -- start 2024-08-19T10:01:41.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.426+0000 7f70c029f640 1 -- start start 2024-08-19T10:01:41.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.426+0000 7f70c029f640 10 monclient: init 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient: _add_conns ranks=[7,1,0,4,6,8,3,2,5] 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient(hunting): picked mon.noname-i con 0x7f70b80a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient(hunting): picked mon.noname-f con 0x7f70b80a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient(hunting): picked mon.noname-a con 0x7f70b813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b80a3b40 con 0x7f70b813f5c0 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b80a8130 con 0x7f70b80a6ba0 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b806ede0 con 0x7f70b80a67c0 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70c029f640 10 monclient(hunting): authenticate will time out at 1357.503307s 2024-08-19T10:01:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70be83c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f70b813f5c0 0x7f70b81419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47888/0 (socket says 172.21.15.1:47888) 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.427+0000 7f70be83c640 1 -- 172.21.15.1:0/1461373428 learned_addr learned my addr 172.21.15.1:0/1461373428 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70be83c640 1 -- 172.21.15.1:0/1461373428 >> v1:172.21.15.1:6789/0 conn(0x7f70b813f5c0 legacy=0x7f70b81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3154527689 0 0) 0x7f70b80a3b40 con 0x7f70b813f5c0 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 10 monclient(hunting): my global_id is 174411 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70a0003520 con 0x7f70b813f5c0 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd83a640 1 -- 172.21.15.1:0/1461373428 >> v1:172.21.15.82:6789/0 conn(0x7f70b80a6ba0 legacy=0x7f70b80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70be03b640 1 -- 172.21.15.1:0/1461373428 >> v1:172.21.15.82:6792/0 conn(0x7f70b80a67c0 legacy=0x7f70b80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773767861 0 0) 0x7f70a0003520 con 0x7f70b813f5c0 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 >> v1:172.21.15.82:6792/0 conn(0x7f70b80a67c0 legacy=0x7f70b80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 >> v1:172.21.15.82:6789/0 conn(0x7f70b80a6ba0 legacy=0x7f70b80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 1 monclient: found mon.noname-a 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.429+0000 7f70bd039640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b80a92a0 con 0x7f70b813f5c0 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70b0001ae0 con 0x7f70b813f5c0 2024-08-19T10:01:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 5 monclient: authenticate success, global_id 174411 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: dump: 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.438 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70b00048a0 con 0x7f70b813f5c0 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 1 -- 172.21.15.1:0/1461373428 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70b0004dd0 con 0x7f70b813f5c0 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70bd039640 10 monclient: dump: 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 4 set_mon_vals no callback set 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 10 monclient: shutdown 2024-08-19T10:01:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 1 -- 172.21.15.1:0/1461373428 >> v1:172.21.15.1:6789/0 conn(0x7f70b813f5c0 legacy=0x7f70b81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 1 -- 172.21.15.1:0/1461373428 shutdown_connections 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 1 -- 172.21.15.1:0/1461373428 >> 172.21.15.1:0/1461373428 conn(0x7f70b80ac4b0 msgr2=0x7f70b80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 1 -- 172.21.15.1:0/1461373428 shutdown_connections 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.430+0000 7f70c029f640 1 -- 172.21.15.1:0/1461373428 wait complete. 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 10 monclient: build_initial_monmap 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 10 monclient: monmap: 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 1 librados: starting msgr at 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 1 librados: starting objecter 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 1 Processor -- start 2024-08-19T10:01:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 1 -- start start 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 1 librados: setting wanted keys 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 1 librados: calling monclient init 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.431+0000 7f70c029f640 10 monclient: init 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient: _add_conns ranks=[5,4,6,8,0,3,1,7,2] 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient(hunting): picked mon.noname-h con 0x7f70b80a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient(hunting): picked mon.noname-c con 0x7f70b80a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient(hunting): picked mon.noname-d con 0x7f70b813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b813ae90 con 0x7f70b80a6ba0 2024-08-19T10:01:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b813b460 con 0x7f70b80a67c0 2024-08-19T10:01:41.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b813e570 con 0x7f70b813f5c0 2024-08-19T10:01:41.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:41.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70c029f640 10 monclient(hunting): authenticate will time out at 1357.508187s 2024-08-19T10:01:41.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70bd83a640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f70b80a6ba0 0x7f70b813c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44728/0 (socket says 172.21.15.1:44728) 2024-08-19T10:01:41.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.432+0000 7f70bd83a640 1 -- 172.21.15.1:0/285988089 learned_addr learned my addr 172.21.15.1:0/285988089 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70bd83a640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.1:6791/0 conn(0x7f70b80a6ba0 legacy=0x7f70b813c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 client.?.objecter ms_handle_connect 0x7f70b80a6ba0 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70be83c640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.1:6792/0 conn(0x7f70b813f5c0 legacy=0x7f70b81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 client.?.objecter ms_handle_connect 0x7f70b813f5c0 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 client.?.objecter ms_handle_connect 0x7f70b80a67c0 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70be03b640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6791/0 conn(0x7f70b80a67c0 legacy=0x7f70b813c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3711548863 0 0) 0x7f70b813ae90 con 0x7f70b80a6ba0 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): my global_id is 27751 2024-08-19T10:01:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70a4004eb0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146599004 0 0) 0x7f70b813e570 con 0x7f70b813f5c0 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): my global_id is 174426 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b813ae90 con 0x7f70b813f5c0 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2814270014 0 0) 0x7f70b806ede0 con 0x7f70b80a67c0 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): my global_id is 27383 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b813e570 con 0x7f70b80a67c0 2024-08-19T10:01:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3489430504 0 0) 0x7f70a4004eb0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b806ede0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70ac003610 con 0x7f70b80a6ba0 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): dump: 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.454 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.455 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3064253920 0 0) 0x7f70b813ae90 con 0x7f70b813f5c0 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70a4004eb0 con 0x7f70b813f5c0 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70b0001d30 con 0x7f70b813f5c0 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 10 monclient(hunting): dump: 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.434+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3127494280 0 0) 0x7f70b813e570 con 0x7f70b80a67c0 2024-08-19T10:01:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b813ae90 con 0x7f70b80a67c0 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70a8001fe0 con 0x7f70b80a67c0 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient(hunting): dump: 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1108940472 0 0) 0x7f70b806ede0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.1:6792/0 conn(0x7f70b813f5c0 legacy=0x7f70b81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6791/0 conn(0x7f70b80a67c0 legacy=0x7f70b813c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 monclient: found mon.c 2024-08-19T10:01:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b81399a0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 5 monclient: authenticate success, global_id 27751 2024-08-19T10:01:41.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 10 monclient: _renew_subs 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70b813bd00 con 0x7f70b80a6ba0 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 10 client.27751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 10 monclient: _renew_subs 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f70b813d020 con 0x7f70b80a6ba0 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 1 librados: init done 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70ac003b90 con 0x7f70b80a6ba0 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70ac0040c0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70bc838640 4 set_mon_vals no callback set 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70b6ffd640 10 monclient: dump: 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.435+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f70980065c0 con 0x7f7098002860 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f70ac004260 con 0x7f70b80a6ba0 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(415..415 src has 299..415) ==== 5481+0+0 (unknown 518798970 0 0) 0x7f70ac04e2d0 con 0x7f70b80a6ba0 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70b6ffd640 10 client.27751.objecter ms_dispatch 0x7f70b8000b90 osd_map(415..415 src has 299..415) 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70b6ffd640 3 client.27751.objecter handle_osd_map got epochs [415,415] > 0 2024-08-19T10:01:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70b6ffd640 3 client.27751.objecter handle_osd_map decoding full epoch 415 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70be83c640 10 monclient: get_auth_request con 0x7f70a4041b50 auth_method 0 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.436+0000 7f70b6ffd640 20 client.27751.objecter dump_active .. 0 homeless 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 client.27751.objecter ms_handle_connect 0x7f7098002860 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 client.27751.objecter resend_mon_ops 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70be03b640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6792/0 conn(0x7f7098002860 legacy=0x7f7098004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3475456683 0 0) 0x7f70980065c0 con 0x7f7098002860 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient(hunting): my global_id is 144898 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b806ede0 con 0x7f7098002860 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2491925983 0 0) 0x7f70b806ede0 con 0x7f7098002860 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70980065c0 con 0x7f7098002860 2024-08-19T10:01:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 client.27751.objecter ms_handle_connect 0x7f70a4041b50 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70a8004010 con 0x7f7098002860 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 10 monclient: dump: 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.465 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1324750424 0 0) 0x7f70980065c0 con 0x7f7098002860 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.438+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f709800a160 con 0x7f7098002860 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.439+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f70a8005b40 con 0x7f7098002860 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.439+0000 7f70b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.439+0000 7f70b6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.439+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6792/0 conn(0x7f7098002860 legacy=0x7f7098004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.439+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6792/0 conn(0x7f7098002860 legacy=0x7f7098004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.445+0000 7f70c029f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.445+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7098009e50 con 0x7f7098009110 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70be03b640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6792/0 conn(0x7f7098009110 legacy=0x7f70980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 client.27751.objecter ms_handle_connect 0x7f7098009110 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 client.27751.objecter resend_mon_ops 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704581714 0 0) 0x7f7098009e50 con 0x7f7098009110 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient(hunting): my global_id is 144907 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70980065c0 con 0x7f7098009110 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4111105823 0 0) 0x7f70980065c0 con 0x7f7098009110 2024-08-19T10:01:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7098009e50 con 0x7f7098009110 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f70a8005cf0 con 0x7f7098009110 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 10 monclient: dump: 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1938184441 0 0) 0x7f7098009e50 con 0x7f7098009110 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.448+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f709800c3f0 con 0x7f7098009110 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 4064142769) 0x7f70a8006060 con 0x7f7098009110 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70b6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6792/0 conn(0x7f7098009110 legacy=0x7f70980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70b6ffd640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6792/0 conn(0x7f7098009110 legacy=0x7f70980095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 425828, 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:41.470 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:41.471 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:41.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:41.474 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:41.475 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:41.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:41.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.478 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.483 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.484 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.485 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.487 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.488 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.489 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:41.490 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 10 librados: watch_flush enter 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 10 librados: watch_flush exit 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 20 client.27751.objecter shutdown clearing up homeless session... 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 10 client.27751.objecter successfully canceled tick 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.82:6812/2549104415 conn(0x7f70a4041b50 legacy=0x7f70a4044010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 10 monclient: shutdown 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 >> v1:172.21.15.1:6791/0 conn(0x7f70b80a6ba0 legacy=0x7f70b813c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70be83c640 1 -- 172.21.15.1:0/285988089 reap_dead start 2024-08-19T10:01:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70be83c640 1 -- 172.21.15.1:0/285988089 reap_dead start 2024-08-19T10:01:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 shutdown_connections 2024-08-19T10:01:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 >> 172.21.15.1:0/285988089 conn(0x7f70b80ac4b0 msgr2=0x7f70b8141cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 shutdown_connections 2024-08-19T10:01:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 -- 172.21.15.1:0/285988089 wait complete. 2024-08-19T10:01:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.449+0000 7f70c029f640 1 librados: shutdown 2024-08-19T10:01:41.494 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:41.494 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T10:01:41.494 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T10:01:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.593+0000 7fa7c7b8a640 10 monclient: get_monmap_and_config 2024-08-19T10:01:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.593+0000 7fa7c7b8a640 10 monclient: build_initial_monmap 2024-08-19T10:01:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.593+0000 7fa7c7b8a640 10 monclient: monmap: 2024-08-19T10:01:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:41.594412+0000 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:41.594412+0000 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:41.595 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 1 Processor -- start 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 1 -- start start 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient: init 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient: _add_conns ranks=[7,5,6,3,1,2,0,8,4] 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient(hunting): picked mon.noname-i con 0x7fa7c00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient(hunting): picked mon.noname-h con 0x7fa7c00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient(hunting): picked mon.noname-d con 0x7fa7c013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c00a3b40 con 0x7fa7c00a6ba0 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c00a8130 con 0x7fa7c013f5c0 2024-08-19T10:01:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c006ede0 con 0x7fa7c00a67c0 2024-08-19T10:01:41.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:41.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c7b8a640 10 monclient(hunting): authenticate will time out at 1357.670832s 2024-08-19T10:01:41.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c6127640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa7c013f5c0 0x7fa7c01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43062/0 (socket says 172.21.15.1:43062) 2024-08-19T10:01:41.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.594+0000 7fa7c6127640 1 -- 172.21.15.1:0/1043491185 learned_addr learned my addr 172.21.15.1:0/1043491185 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c6127640 1 -- 172.21.15.1:0/1043491185 >> v1:172.21.15.1:6792/0 conn(0x7fa7c013f5c0 legacy=0x7fa7c01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c5125640 1 -- 172.21.15.1:0/1043491185 >> v1:172.21.15.82:6791/0 conn(0x7fa7c00a6ba0 legacy=0x7fa7c00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c5926640 1 -- 172.21.15.1:0/1043491185 >> v1:172.21.15.82:6792/0 conn(0x7fa7c00a67c0 legacy=0x7fa7c00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838040611 0 0) 0x7fa7c00a8130 con 0x7fa7c013f5c0 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): my global_id is 174435 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ac003520 con 0x7fa7c013f5c0 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353453337 0 0) 0x7fa7c00a3b40 con 0x7fa7c00a6ba0 2024-08-19T10:01:41.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient(hunting): my global_id is 27392 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c00a8130 con 0x7fa7c00a6ba0 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1319991036 0 0) 0x7fa7ac003520 con 0x7fa7c013f5c0 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 >> v1:172.21.15.82:6792/0 conn(0x7fa7c00a67c0 legacy=0x7fa7c00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 >> v1:172.21.15.82:6791/0 conn(0x7fa7c00a6ba0 legacy=0x7fa7c00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 monclient: found mon.noname-d 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c00a92a0 con 0x7fa7c013f5c0 2024-08-19T10:01:41.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7b8001e60 con 0x7fa7c013f5c0 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c7b8a640 5 monclient: authenticate success, global_id 174435 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c7b8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: dump: 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c7b8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7b8004c20 con 0x7fa7c013f5c0 2024-08-19T10:01:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 1 -- 172.21.15.1:0/1043491185 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7b8005190 con 0x7fa7c013f5c0 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c4924640 10 monclient: dump: 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c7b8a640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.597+0000 7fa7c7b8a640 4 set_mon_vals no callback set 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 10 monclient: shutdown 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1043491185 >> v1:172.21.15.1:6792/0 conn(0x7fa7c013f5c0 legacy=0x7fa7c01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1043491185 shutdown_connections 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1043491185 >> 172.21.15.1:0/1043491185 conn(0x7fa7c00ac4b0 msgr2=0x7fa7c00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1043491185 shutdown_connections 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1043491185 wait complete. 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 10 monclient: build_initial_monmap 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.598+0000 7fa7c7b8a640 10 monclient: monmap: 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 librados: starting msgr at 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 librados: starting objecter 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 Processor -- start 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 -- start start 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 librados: setting wanted keys 2024-08-19T10:01:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 librados: calling monclient init 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient: init 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient: _add_conns ranks=[6,5,1,4,2,3,0,7,8] 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient(hunting): picked mon.noname-d con 0x7fa7c00a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient(hunting): picked mon.noname-h con 0x7fa7c00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient(hunting): picked mon.noname-f con 0x7fa7c013f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c013aa50 con 0x7fa7c013f5c0 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c00a17d0 con 0x7fa7c00a6ba0 2024-08-19T10:01:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c013a770 con 0x7fa7c00a67c0 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c7b8a640 10 monclient(hunting): authenticate will time out at 1357.675801s 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c5926640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa7c00a67c0 0x7fa7c01ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43066/0 (socket says 172.21.15.1:43066) 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.599+0000 7fa7c5926640 1 -- 172.21.15.1:0/1156186696 learned_addr learned my addr 172.21.15.1:0/1156186696 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.601+0000 7fa7c5926640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.1:6792/0 conn(0x7fa7c00a67c0 legacy=0x7fa7c01ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 client.?.objecter ms_handle_connect 0x7fa7c00a67c0 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1621617366 0 0) 0x7fa7c013a770 con 0x7fa7c00a67c0 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): my global_id is 174444 2024-08-19T10:01:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a4004eb0 con 0x7fa7c00a67c0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 client.?.objecter ms_handle_connect 0x7fa7c00a6ba0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c5125640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.82:6791/0 conn(0x7fa7c00a6ba0 legacy=0x7fa7c01d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 client.?.objecter ms_handle_connect 0x7fa7c013f5c0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c6127640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.82:6789/0 conn(0x7fa7c013f5c0 legacy=0x7fa7c01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3956387066 0 0) 0x7fa7a4004eb0 con 0x7fa7c00a67c0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c013a770 con 0x7fa7c00a67c0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7bc003610 con 0x7fa7c00a67c0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): dump: 2024-08-19T10:01:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.617 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 51883912 0 0) 0x7fa7c00a17d0 con 0x7fa7c00a6ba0 2024-08-19T10:01:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): my global_id is 27401 2024-08-19T10:01:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a4004eb0 con 0x7fa7c00a6ba0 2024-08-19T10:01:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368343318 0 0) 0x7fa7c013aa50 con 0x7fa7c013f5c0 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient(hunting): my global_id is 174403 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c00a17d0 con 0x7fa7c013f5c0 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1403635641 0 0) 0x7fa7c013a770 con 0x7fa7c00a67c0 2024-08-19T10:01:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.82:6791/0 conn(0x7fa7c00a6ba0 legacy=0x7fa7c01d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.82:6789/0 conn(0x7fa7c013f5c0 legacy=0x7fa7c01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 monclient: found mon.d 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c0139b90 con 0x7fa7c00a67c0 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 5 monclient: authenticate success, global_id 174444 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 10 monclient: _renew_subs 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c00a2330 con 0x7fa7c00a67c0 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 10 client.174444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 10 monclient: _renew_subs 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7c00a1010 con 0x7fa7c00a67c0 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7c7b8a640 1 librados: init done 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7bc003c00 con 0x7fa7c00a67c0 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa7bc0040f0 con 0x7fa7c00a67c0 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b7fff640 4 set_mon_vals no callback set 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.602+0000 7fa7b67fc640 10 monclient: dump: 2024-08-19T10:01:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.603+0000 7fa7c7b8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.603+0000 7fa7c7b8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.603+0000 7fa7c7b8a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.603+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a8005080 con 0x7fa7c00a67c0 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.603+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa7bc004290 con 0x7fa7c00a67c0 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.604+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(415..415 src has 299..415) ==== 5481+0+0 (unknown 518798970 0 0) 0x7fa7bc04e330 con 0x7fa7c00a67c0 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.604+0000 7fa7b67fc640 10 client.174444.objecter ms_dispatch 0x7fa7c0000b90 osd_map(415..415 src has 299..415) 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.604+0000 7fa7b67fc640 3 client.174444.objecter handle_osd_map got epochs [415,415] > 0 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.604+0000 7fa7b67fc640 3 client.174444.objecter handle_osd_map decoding full epoch 415 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.604+0000 7fa7b67fc640 20 client.174444.objecter dump_active .. 0 homeless 2024-08-19T10:01:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.604+0000 7fa7c5125640 10 monclient: get_auth_request con 0x7fa7a4041120 auth_method 0 2024-08-19T10:01:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.606+0000 7fa7b67fc640 10 client.174444.objecter ms_handle_connect 0x7fa7a4041120 2024-08-19T10:01:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.608+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa7bc0175f0 con 0x7fa7c00a67c0 2024-08-19T10:01:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.608+0000 7fa7b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.608+0000 7fa7b67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.868+0000 7fa7c7b8a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T10:01:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.868+0000 7fa7c7b8a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T10:01:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.868+0000 7fa7c7b8a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.868+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa7a8005c40 con 0x7fa7c00a67c0 2024-08-19T10:01:41.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.870+0000 7fa7b67fc640 1 -- 172.21.15.1:0/1156186696 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fa7bc01b4f0 con 0x7fa7c00a67c0 2024-08-19T10:01:41.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.870+0000 7fa7b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T10:01:41.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.870+0000 7fa7b67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 10 librados: watch_flush enter 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 10 librados: watch_flush exit 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 20 client.174444.objecter shutdown clearing up homeless session... 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 10 client.174444.objecter successfully canceled tick 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa7a4041120 legacy=0x7fa7a40435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 10 monclient: shutdown 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 >> v1:172.21.15.1:6792/0 conn(0x7fa7c00a67c0 legacy=0x7fa7c01ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 shutdown_connections 2024-08-19T10:01:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 >> 172.21.15.1:0/1156186696 conn(0x7fa7c00ac4b0 msgr2=0x7fa7c00a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 shutdown_connections 2024-08-19T10:01:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 -- 172.21.15.1:0/1156186696 wait complete. 2024-08-19T10:01:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:41.873+0000 7fa7c7b8a640 1 librados: shutdown 2024-08-19T10:01:41.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T10:01:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=872527168,kv=313361912,logm=2146492779}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=1151234885,config=2586702516,health=2941319762,kv=2485470190,logm=607508051}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=3179131515,config=2586702516,health=2941319762,kv=313361912,logm=2723069186}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=1332854661,config=2586702516,health=2377988878,kv=1251488192,logm=142242424}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=404357714,config=2586702516,health=872527168,kv=3949047626,logm=2396047023}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=25,config=1,health=12,kv=12,logm=50} crc {auth=726627786,config=1747058506,health=672568691,kv=1455257465,logm=192612721}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=2803982899,config=2586702516,health=672568691,kv=3960219694,logm=3298447111}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1151234885,config=2586702516,health=4241352263,kv=3949047626,logm=2131018679}) 2024-08-19T10:01:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.160+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=3363907131,config=2586702516,health=2941319762,kv=1455257465,logm=890963112}) 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3063711978}) 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615517550}) 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2805049566}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3408273713}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4265999176}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3248455984}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1950742104}) 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621913239}) 2024-08-19T10:01:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.164+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1869980571}) 2024-08-19T10:01:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729931989}) 2024-08-19T10:01:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3130075982}) 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.172+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=172197077}) 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=894244031}) 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3933469018}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=627311086}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1413924819}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862787177}) 2024-08-19T10:01:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.173+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1180231832}) 2024-08-19T10:01:42.552 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596238923}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=73920955}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2905796831}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4115612420}) 2024-08-19T10:01:42.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2081503413}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1410383915}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1074193155}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774988229}) 2024-08-19T10:01:42.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.550+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1212978973}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131426634}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250831746}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2054854200}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.559 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1028368623}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=741752084}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1505428445}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=926471631}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208459954}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.552+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=136828835}) 2024-08-19T10:01:42.560 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=858418823}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926651963}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4121625992}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=107687059}) 2024-08-19T10:01:42.561 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=653053578}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2315298606}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3202007614}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462029255}) 2024-08-19T10:01:42.562 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.553+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3008432090}) 2024-08-19T10:01:42.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2432697976}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595202952}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2114429667}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=698606995}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1152959025}) 2024-08-19T10:01:42.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2177286895}) 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3191950753}) 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554482362}) 2024-08-19T10:01:42.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.669+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=448375062}) 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2877864158}) 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257432174}) 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=378470187}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=801882748}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3776284728}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1848640080}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2852251796}) 2024-08-19T10:01:42.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129528229}) 2024-08-19T10:01:42.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=995676186}) 2024-08-19T10:01:42.885 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:01:42.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:01:42.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389040316}) 2024-08-19T10:01:42.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386402453}) 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3130413493}) 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2439675530}) 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.972 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1227760130}) 2024-08-19T10:01:42.972 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.972 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.968+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.972 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2617581310}) 2024-08-19T10:01:42.972 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.973 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.973 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3403751208}) 2024-08-19T10:01:42.973 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.973 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038113588}) 2024-08-19T10:01:42.973 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.969+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2024007690}) 2024-08-19T10:01:42.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072537887}) 2024-08-19T10:01:42.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1119273684}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=424899404}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3747893330}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2259104923}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3477290100}) 2024-08-19T10:01:42.985 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.985 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.985 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1099867335}) 2024-08-19T10:01:42.985 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.985 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300083379}) 2024-08-19T10:01:42.985 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.981+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1153049822}) 2024-08-19T10:01:42.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4132858404}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941293740}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295667639}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1767475061}) 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.982+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.990 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=586942523}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2702729405}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1946840146}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149679640}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.983+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3943960595}) 2024-08-19T10:01:42.991 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813258228}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237659253}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4230780911}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1083507043}) 2024-08-19T10:01:42.992 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=974255837}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1070012005}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948450117}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593450069}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.984+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3550190374}) 2024-08-19T10:01:42.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1170719014}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4119562160}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2808969520}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3923723782}) 2024-08-19T10:01:42.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3830307772}) 2024-08-19T10:01:42.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=771489444}) 2024-08-19T10:01:42.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1241208437}) 2024-08-19T10:01:42.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:42.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131546844}) 2024-08-19T10:01:42.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:42.985+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=342393444}) 2024-08-19T10:01:42.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.994+0000 7f7161fd9640 10 monclient: get_monmap_and_config 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.994+0000 7f7161fd9640 10 monclient: build_initial_monmap 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.994+0000 7f7161fd9640 10 monclient: monmap: 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:42.995578+0000 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:42.995578+0000 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 1 Processor -- start 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 1 -- start start 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 10 monclient: init 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 10 monclient: _add_conns ranks=[2,5,1,7,6,4,0,3,8] 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 10 monclient(hunting): picked mon.noname-b con 0x7f715c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 10 monclient(hunting): picked mon.noname-h con 0x7f715c0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 10 monclient(hunting): picked mon.noname-f con 0x7f715c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:43.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.995+0000 7f7161fd9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c0a3b40 con 0x7f715c13f5c0 2024-08-19T10:01:43.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.996+0000 7f7161fd9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c0a8130 con 0x7f715c0a67c0 2024-08-19T10:01:43.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.996+0000 7f7161fd9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c06ede0 con 0x7f715c0a6ba0 2024-08-19T10:01:43.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.996+0000 7f7161fd9640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.996+0000 7f7161fd9640 10 monclient(hunting): authenticate will time out at 1359.072042s 2024-08-19T10:01:43.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.996+0000 7f715b7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f715c0a67c0 0x7f715c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42738/0 (socket says 172.21.15.1:42738) 2024-08-19T10:01:43.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.996+0000 7f715b7fe640 1 -- 172.21.15.1:0/1211168759 learned_addr learned my addr 172.21.15.1:0/1211168759 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715b7fe640 1 -- 172.21.15.1:0/1211168759 >> v1:172.21.15.1:6790/0 conn(0x7f715c0a67c0 legacy=0x7f715c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715bfff640 1 -- 172.21.15.1:0/1211168759 >> v1:172.21.15.82:6789/0 conn(0x7f715c13f5c0 legacy=0x7f715c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715affd640 1 -- 172.21.15.1:0/1211168759 >> v1:172.21.15.82:6791/0 conn(0x7f715c0a6ba0 legacy=0x7f715c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448572897 0 0) 0x7f715c0a8130 con 0x7f715c0a67c0 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): my global_id is 184142 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7144003520 con 0x7f715c0a67c0 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936444603 0 0) 0x7f715c06ede0 con 0x7f715c0a6ba0 2024-08-19T10:01:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): my global_id is 27410 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715c0a8130 con 0x7f715c0a6ba0 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930840795 0 0) 0x7f715c0a3b40 con 0x7f715c13f5c0 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient(hunting): my global_id is 174412 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715c06ede0 con 0x7f715c13f5c0 2024-08-19T10:01:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 324605080 0 0) 0x7f7144003520 con 0x7f715c0a67c0 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 >> v1:172.21.15.82:6791/0 conn(0x7f715c0a6ba0 legacy=0x7f715c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 >> v1:172.21.15.82:6789/0 conn(0x7f715c13f5c0 legacy=0x7f715c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 monclient: found mon.noname-b 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c0a92a0 con 0x7f715c0a67c0 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f714c001ae0 con 0x7f715c0a67c0 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f7161fd9640 5 monclient: authenticate success, global_id 184142 2024-08-19T10:01:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f7161fd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: dump: 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f7161fd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f714c0048a0 con 0x7f715c0a67c0 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 1 -- 172.21.15.1:0/1211168759 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f714c004e10 con 0x7f715c0a67c0 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f715a7fc640 10 monclient: dump: 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f7161fd9640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.998+0000 7f7161fd9640 4 set_mon_vals no callback set 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 10 monclient: shutdown 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 1 -- 172.21.15.1:0/1211168759 >> v1:172.21.15.1:6790/0 conn(0x7f715c0a67c0 legacy=0x7f715c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 1 -- 172.21.15.1:0/1211168759 shutdown_connections 2024-08-19T10:01:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 1 -- 172.21.15.1:0/1211168759 >> 172.21.15.1:0/1211168759 conn(0x7f715c0ac4b0 msgr2=0x7f715c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 1 -- 172.21.15.1:0/1211168759 shutdown_connections 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 1 -- 172.21.15.1:0/1211168759 wait complete. 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:42.999+0000 7f7161fd9640 10 monclient: monmap: 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 librados: starting msgr at 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 librados: starting objecter 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 Processor -- start 2024-08-19T10:01:43.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 -- start start 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 librados: setting wanted keys 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 librados: calling monclient init 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient: init 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient: _add_conns ranks=[1,3,6,8,7,5,4,2,0] 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient(hunting): picked mon.noname-f con 0x7f715c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient(hunting): picked mon.noname-g con 0x7f715c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient(hunting): picked mon.noname-d con 0x7f715c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c13be80 con 0x7f715c0a67c0 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c1cf020 con 0x7f715c0a6ba0 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c0a9cc0 con 0x7f715c13f5c0 2024-08-19T10:01:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.000+0000 7f7161fd9640 10 monclient(hunting): authenticate will time out at 1359.076996s 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.001+0000 7f715bfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f715c13f5c0 0x7f715c1d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43068/0 (socket says 172.21.15.1:43068) 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.001+0000 7f715bfff640 1 -- 172.21.15.1:0/2078086162 learned_addr learned my addr 172.21.15.1:0/2078086162 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f715bfff640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.1:6792/0 conn(0x7f715c13f5c0 legacy=0x7f715c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 client.?.objecter ms_handle_connect 0x7f715c13f5c0 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 client.?.objecter ms_handle_connect 0x7f715c0a67c0 2024-08-19T10:01:43.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f715b7fe640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.82:6789/0 conn(0x7f715c0a67c0 legacy=0x7f715c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 client.?.objecter ms_handle_connect 0x7f715c0a6ba0 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f715affd640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.82:6790/0 conn(0x7f715c0a6ba0 legacy=0x7f715c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3686651236 0 0) 0x7f715c0a9cc0 con 0x7f715c13f5c0 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): my global_id is 174453 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7148004eb0 con 0x7f715c13f5c0 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2962104887 0 0) 0x7f715c13be80 con 0x7f715c0a67c0 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): my global_id is 174421 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715c0a9cc0 con 0x7f715c0a67c0 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3556257304 0 0) 0x7f715c1cf020 con 0x7f715c0a6ba0 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): my global_id is 184170 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715c13be80 con 0x7f715c0a6ba0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3960268511 0 0) 0x7f7148004eb0 con 0x7f715c13f5c0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f715c1cf020 con 0x7f715c13f5c0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2277705055 0 0) 0x7f715c0a9cc0 con 0x7f715c0a67c0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7148004eb0 con 0x7f715c0a67c0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f714c004d90 con 0x7f715c0a67c0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): dump: 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.025 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7154003990 con 0x7f715c13f5c0 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): dump: 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1494867991 0 0) 0x7f715c13be80 con 0x7f715c0a6ba0 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f715c0a9cc0 con 0x7f715c0a6ba0 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7150003610 con 0x7f715c0a6ba0 2024-08-19T10:01:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient(hunting): dump: 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1707910391 0 0) 0x7f715c1cf020 con 0x7f715c13f5c0 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.82:6790/0 conn(0x7f715c0a6ba0 legacy=0x7f715c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.82:6789/0 conn(0x7f715c0a67c0 legacy=0x7f715c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 monclient: found mon.d 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c13b010 con 0x7f715c13f5c0 2024-08-19T10:01:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7143fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7161fd9640 5 monclient: authenticate success, global_id 174453 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7161fd9640 10 monclient: _renew_subs 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7161fd9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.003+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f715c13bba0 con 0x7f715c13f5c0 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7161fd9640 10 client.174453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7161fd9640 10 monclient: _renew_subs 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7161fd9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f715c13d390 con 0x7f715c13f5c0 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7161fd9640 1 librados: init done 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7154003ec0 con 0x7f715c13f5c0 2024-08-19T10:01:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7143fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71540043f0 con 0x7f715c13f5c0 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7143fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7143fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7159ffb640 4 set_mon_vals no callback set 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.004+0000 7f7143fff640 10 monclient: dump: 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7161fd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7161fd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7161fd9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7154011e00 con 0x7f715c13f5c0 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713c005080 con 0x7f715c13f5c0 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(417..417 src has 299..417) ==== 5516+0+0 (unknown 2371035315 0 0) 0x7f715404e590 con 0x7f715c13f5c0 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7143fff640 10 client.174453.objecter ms_dispatch 0x7f715c000b90 osd_map(417..417 src has 299..417) 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7143fff640 3 client.174453.objecter handle_osd_map got epochs [417,417] > 0 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7143fff640 3 client.174453.objecter handle_osd_map decoding full epoch 417 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f7143fff640 20 client.174453.objecter dump_active .. 0 homeless 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.005+0000 7f715b7fe640 10 monclient: get_auth_request con 0x7f7148041880 auth_method 0 2024-08-19T10:01:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.007+0000 7f7143fff640 10 client.174453.objecter ms_handle_connect 0x7f7148041880 2024-08-19T10:01:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.009+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f7154017830 con 0x7f715c13f5c0 2024-08-19T10:01:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.009+0000 7f7143fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.009+0000 7f7143fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:43.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=859990250}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3891849931}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2036459991}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=910289454}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4029396264}) 2024-08-19T10:01:43.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1597807291}) 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=67684853}) 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.170+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228804537}) 2024-08-19T10:01:43.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.171+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2289017203}) 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616682045}) 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105533826}) 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921006425}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=170265938}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2865301227}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4082007178}) 2024-08-19T10:01:43.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=720668575}) 2024-08-19T10:01:43.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168488840}) 2024-08-19T10:01:43.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.183+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=803934344}) 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159584310}) 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613215559}) 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1166088572}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=571863307}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1731428}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4167777559}) 2024-08-19T10:01:43.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2277823747}) 2024-08-19T10:01:43.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612080211}) 2024-08-19T10:01:43.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.256+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2164292503}) 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851779114}) 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411682693}) 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.284 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3887677994}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2874757861}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2625766066}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1523339057}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3723950354}) 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815391486}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.257+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=875968275}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=768433146}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71507813}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=358783616}) 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2283662552}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=290347804}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4041539616}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137671791}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433677618}) 2024-08-19T10:01:43.287 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.259+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3217921455}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=480054914}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2045319364}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=301893174}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3966710683}) 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.288 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=70465235}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3639755963}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1122099305}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076834771}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.261+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3408033719}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.289 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2499858752}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013976150}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2950648324}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3016629932}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.290 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=124036392}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3021433075}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1530509706}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143840284}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1009992460}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895134625}) 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.291 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267686056}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1119937429}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=98335632}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3631650206}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=894328846}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1471754302}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050356199}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1145764469}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598333250}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543134878}) 2024-08-19T10:01:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2579840153}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1296518391}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4015894927}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2262087648}) 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2692407170}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697189264}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.266+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1152785662}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100060691}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168699541}) 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2907337404}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3623432462}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=870450510}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1419806709}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1684142105}) 2024-08-19T10:01:43.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443152532}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.267+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=208104221}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3673080217}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285046897}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566690340}) 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=178174191}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.268+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1721962954}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1031766984}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2051210921}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667266779}) 2024-08-19T10:01:43.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.269+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1961578636}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800441293}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799197959}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=961961896}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3004301237}) 2024-08-19T10:01:43.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1436636352}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4100635483}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4099331447}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152384894}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.270+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3339485506}) 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1527655186}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1486105151}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2126768290}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1979259416}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2952345837}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4222683012}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3690772868}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047770879}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.271+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1735618065}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427966078}) 2024-08-19T10:01:43.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2388381874}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032564165}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3730318258}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3251598817}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2965202841}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2694263286}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352221702}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=964600084}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999817806}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069910549}) 2024-08-19T10:01:43.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2907821457}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1259479638}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1368003259}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1444948173}) 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197703000}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162306812}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1783958925}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2090000972}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=591070855}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1545250240}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=46603873}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2602966208}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=665139714}) 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4238209000}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546278898}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4149421345}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790268927}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132196726}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3719434693}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1842697915}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1352395804}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2447825702}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2935358866}) 2024-08-19T10:01:43.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007702989}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2138709607}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505135461}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3204327520}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060975565}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3752112688}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2742613496}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3984940874}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=44728917}) 2024-08-19T10:01:43.311 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056932966}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1219900272}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060450360}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724081897}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2766756201}) 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.312 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2386410249}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1999409480}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1055122572}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3387309764}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.280+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230131575}) 2024-08-19T10:01:43.313 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.281+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3985103292}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3098642655}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520823572}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1628734458}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=623274075}) 2024-08-19T10:01:43.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2796259407}) 2024-08-19T10:01:43.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7161fd9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:01:43.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=335705429}) 2024-08-19T10:01:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7161fd9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:01:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7161fd9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f713c005370 con 0x7f715c13f5c0 2024-08-19T10:01:43.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3290283235}) 2024-08-19T10:01:43.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184088818}) 2024-08-19T10:01:43.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=606411751}) 2024-08-19T10:01:43.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7143fff640 1 -- 172.21.15.1:0/2078086162 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 1356691994) 0x7f715401b730 con 0x7f715c13f5c0 2024-08-19T10:01:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7143fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:01:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.282+0000 7f7143fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:43.318 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":106,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:01:43.322 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.322 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.322 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3884017754}) 2024-08-19T10:01:43.322 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.322 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=36842548}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2028283163}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1714472653}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=640477221}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.323 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2111266747}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2673898599}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534123568}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.283+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2804078592}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852043569}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1735826733}) 2024-08-19T10:01:43.324 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3465596100}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1209853413}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2887442708}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3215033148}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.325 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1801400200}) 2024-08-19T10:01:43.326 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.326 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672185662}) 2024-08-19T10:01:43.326 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3213820040}) 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 10 librados: watch_flush enter 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 10 librados: watch_flush exit 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 20 client.174453.objecter shutdown clearing up homeless session... 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 10 client.174453.objecter successfully canceled tick 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7148041880 legacy=0x7f7148043d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 10 monclient: shutdown 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.285+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 >> v1:172.21.15.1:6792/0 conn(0x7f715c13f5c0 legacy=0x7f715c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 shutdown_connections 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 >> 172.21.15.1:0/2078086162 conn(0x7f715c0ac4b0 msgr2=0x7f715c13eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 shutdown_connections 2024-08-19T10:01:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f7161fd9640 1 -- 172.21.15.1:0/2078086162 wait complete. 2024-08-19T10:01:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f7161fd9640 1 librados: shutdown 2024-08-19T10:01:43.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373317711}) 2024-08-19T10:01:43.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1831987476}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2968738026}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3491966937}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3930020258}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=253526428}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.286+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.287+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2484496055}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.287+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.287+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819992299}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.287+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2069692821}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2267265584}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=382176491}) 2024-08-19T10:01:43.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3794633216}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=95307993}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2899815439}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2173567629}) 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3337724290}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260417165}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.288+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1015895183}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409872816}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021854406}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=451094764}) 2024-08-19T10:01:43.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2584385508}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2880703332}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2631113112}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2390031126}) 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437751496}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.289+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2097916746}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056597406}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686131526}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3456453535}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3560660276}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2752418428}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2826193095}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3256578973}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128195440}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.290+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2266036955}) 2024-08-19T10:01:43.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294200027}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1994693764}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3414036944}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1464132156}) 2024-08-19T10:01:43.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=346655767}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=187372989}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1596572986}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175711930}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.292+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=81137302}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696078245}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599590165}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=700348898}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=455214358}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1161394838}) 2024-08-19T10:01:43.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=365330224}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=882531131}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520544670}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.294+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1221946061}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898418708}) 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626221218}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2950717686}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=812334030}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=373521748}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=355129056}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3953849904}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136547864}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.295+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2761596287}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=558007895}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587097942}) 2024-08-19T10:01:43.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2435686208}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3225021435}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1496286545}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2856840099}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1357028644}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738366248}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.296+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3632016471}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3030098619}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445761546}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4027817822}) 2024-08-19T10:01:43.361 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.297+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3780100408}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=265962550}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3010986315}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=348140101}) 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389590977}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.298+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2990984738}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170205500}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579513812}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2752164257}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296094165}) 2024-08-19T10:01:43.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3627720354}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4031138103}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1368404514}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438293941}) 2024-08-19T10:01:43.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.299+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=630074532}) 2024-08-19T10:01:43.365 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:01:43.365 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T10:01:43.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:01:43.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:01:43.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111171252}) 2024-08-19T10:01:43.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.436 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200958207}) 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4183690853}) 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1117495704}) 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.437 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=93532473}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=453375119}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3506460787}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256035182}) 2024-08-19T10:01:43.438 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.434+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1627837084}) 2024-08-19T10:01:43.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2328939274}) 2024-08-19T10:01:43.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.452 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883021754}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2366099615}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3606119315}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3843384369}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3233983544}) 2024-08-19T10:01:43.453 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1001987777}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154339568}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.436+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=624002598}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3215949611}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2933835308}) 2024-08-19T10:01:43.454 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3137960121}) 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1810865230}) 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.455 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=986646147}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=821327742}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2753118967}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436321801}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.438+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2951440041}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275780243}) 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.456 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541981804}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2828945627}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1096246770}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2670390677}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=541694064}) 2024-08-19T10:01:43.457 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1940036747}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315106367}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.439+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=813278150}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028829927}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2683704278}) 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.458 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2177308482}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4286097813}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1534333088}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1211608353}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.440+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.441+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.459 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.441+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3306540017}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.441+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.441+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3585531151}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.441+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2281008851}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495548287}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016641201}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173383708}) 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2363052034}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2611047248}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=635011102}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2359610502}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851519253}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.442+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=189374067}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208337314}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107360125}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782027251}) 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.462 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1773215203}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2896626144}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1520281637}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902776483}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563394840}) 2024-08-19T10:01:43.463 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.443+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=556067782}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436753775}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326128799}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.444+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2951628581}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=361121573}) 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.464 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4273074260}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=46297085}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3579789193}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967043931}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.445+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3013339944}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.465 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116863391}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644384075}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=415956915}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2138159488}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4011778953}) 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.466 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1980876165}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2809618641}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660954589}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.446+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3267094740}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1820335480}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2051165498}) 2024-08-19T10:01:43.467 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=957259998}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1925564892}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3111094260}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3311497083}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.468 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2803599331}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896534151}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.447+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=595059170}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549940097}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596593860}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952001299}) 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.469 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1113306509}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3288745361}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2488028132}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2916179207}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104496204}) 2024-08-19T10:01:43.470 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.449+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3226816652}) 2024-08-19T10:01:43.472 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.472 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.472 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972151044}) 2024-08-19T10:01:43.472 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.472 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.472 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004858276}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521092360}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3950445089}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3585731869}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=568008435}) 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.473 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.474 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=158338490}) 2024-08-19T10:01:43.474 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.474 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852422621}) 2024-08-19T10:01:43.474 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.461+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4063633187}) 2024-08-19T10:01:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.472+0000 7f1d0d420640 10 monclient: get_monmap_and_config 2024-08-19T10:01:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.472+0000 7f1d0d420640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.472+0000 7f1d0d420640 10 monclient: monmap: 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:43.473097+0000 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:43.473097+0000 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.472+0000 7f1d0d420640 1 Processor -- start 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.472+0000 7f1d0d420640 1 -- start start 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.472+0000 7f1d0d420640 10 monclient: init 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient: _add_conns ranks=[4,6,3,7,5,2,1,8,0] 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient(hunting): picked mon.noname-c con 0x7f1d080a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient(hunting): picked mon.noname-d con 0x7f1d080694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient(hunting): picked mon.noname-g con 0x7f1d080a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d080a3c70 con 0x7f1d080a46a0 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d080a33d0 con 0x7f1d080a6830 2024-08-19T10:01:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d080a36b0 con 0x7f1d080694b0 2024-08-19T10:01:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d0d420640 10 monclient(hunting): authenticate will time out at 1359.549310s 2024-08-19T10:01:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d06ffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f1d080a6830 0x7f1d080a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44738/0 (socket says 172.21.15.1:44738) 2024-08-19T10:01:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d06ffd640 1 -- 172.21.15.1:0/2813922414 learned_addr learned my addr 172.21.15.1:0/2813922414 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.473+0000 7f1d067fc640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f1d080694b0 0x7f1d080a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43076/0 (socket says 172.21.15.1:43076) 2024-08-19T10:01:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d06ffd640 1 -- 172.21.15.1:0/2813922414 >> v1:172.21.15.1:6791/0 conn(0x7f1d080a6830 legacy=0x7f1d080a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d067fc640 1 -- 172.21.15.1:0/2813922414 >> v1:172.21.15.1:6792/0 conn(0x7f1d080694b0 legacy=0x7f1d080a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d077fe640 1 -- 172.21.15.1:0/2813922414 >> v1:172.21.15.82:6790/0 conn(0x7f1d080a46a0 legacy=0x7f1d08141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546057169 0 0) 0x7f1d080a33d0 con 0x7f1d080a6830 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): my global_id is 27760 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cf4003520 con 0x7f1d080a6830 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832577112 0 0) 0x7f1d080a36b0 con 0x7f1d080694b0 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 10 monclient(hunting): my global_id is 174462 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d080a33d0 con 0x7f1d080694b0 2024-08-19T10:01:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.475+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 35397765 0 0) 0x7f1d080a3c70 con 0x7f1d080a46a0 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient(hunting): my global_id is 184179 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d080a36b0 con 0x7f1d080a46a0 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4165658554 0 0) 0x7f1cf4003520 con 0x7f1d080a6830 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 >> v1:172.21.15.1:6792/0 conn(0x7f1d080694b0 legacy=0x7f1d080a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 >> v1:172.21.15.82:6790/0 conn(0x7f1d080a46a0 legacy=0x7f1d08141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 monclient: found mon.noname-c 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0806dbb0 con 0x7f1d080a6830 2024-08-19T10:01:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cf8001ae0 con 0x7f1d080a6830 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 5 monclient: authenticate success, global_id 27760 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: dump: 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1cf80048a0 con 0x7f1d080a6830 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 1 -- 172.21.15.1:0/2813922414 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cf8004e10 con 0x7f1d080a6830 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 4 set_mon_vals no callback set 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d05ffb640 10 monclient: dump: 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 10 monclient: shutdown 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 1 -- 172.21.15.1:0/2813922414 >> v1:172.21.15.1:6791/0 conn(0x7f1d080a6830 legacy=0x7f1d080a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 1 -- 172.21.15.1:0/2813922414 shutdown_connections 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 1 -- 172.21.15.1:0/2813922414 >> 172.21.15.1:0/2813922414 conn(0x7f1d080a9020 msgr2=0x7f1d08068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 1 -- 172.21.15.1:0/2813922414 shutdown_connections 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.476+0000 7f1d0d420640 1 -- 172.21.15.1:0/2813922414 wait complete. 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 10 monclient: monmap: 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.493 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 1 librados: starting msgr at 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 1 librados: starting objecter 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 1 Processor -- start 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 1 -- start start 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 1 librados: setting wanted keys 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 1 librados: calling monclient init 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.477+0000 7f1d0d420640 10 monclient: init 2024-08-19T10:01:43.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient: _add_conns ranks=[2,5,6,3,1,7,8,0,4] 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient(hunting): picked mon.noname-b con 0x7f1d080694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient(hunting): picked mon.noname-h con 0x7f1d080a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient(hunting): picked mon.noname-d con 0x7f1d080a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d08138060 con 0x7f1d080694b0 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d08138340 con 0x7f1d080a46a0 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d08138620 con 0x7f1d080a6830 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d0d420640 10 monclient(hunting): authenticate will time out at 1359.554352s 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d06ffd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f1d080694b0 0x7f1d08141e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42748/0 (socket says 172.21.15.1:42748) 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.478+0000 7f1d06ffd640 1 -- 172.21.15.1:0/1516644138 learned_addr learned my addr 172.21.15.1:0/1516644138 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1d06ffd640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6790/0 conn(0x7f1d080694b0 legacy=0x7f1d08141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1d077fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6792/0 conn(0x7f1d080a6830 legacy=0x7f1d081d5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 client.?.objecter ms_handle_connect 0x7f1d080694b0 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 client.?.objecter ms_handle_connect 0x7f1d080a6830 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220259805 0 0) 0x7f1d08138060 con 0x7f1d080694b0 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): my global_id is 184151 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ce8004ee0 con 0x7f1d080694b0 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386967588 0 0) 0x7f1d08138620 con 0x7f1d080a6830 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 monclient(hunting): my global_id is 174471 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d08138060 con 0x7f1d080a6830 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 client.?.objecter ms_handle_connect 0x7f1d080a46a0 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1cff7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.480+0000 7f1d067fc640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.82:6791/0 conn(0x7f1d080a46a0 legacy=0x7f1d081ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2698175331 0 0) 0x7f1ce8004ee0 con 0x7f1d080694b0 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d08138620 con 0x7f1d080694b0 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 990933274 0 0) 0x7f1d08138060 con 0x7f1d080a6830 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ce8004ee0 con 0x7f1d080a6830 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cf8001d30 con 0x7f1d080694b0 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): dump: 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 386658443 0 0) 0x7f1d08138340 con 0x7f1d080a46a0 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): my global_id is 27419 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d08138060 con 0x7f1d080a46a0 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d00003990 con 0x7f1d080a6830 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient(hunting): dump: 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.500 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1805756202 0 0) 0x7f1d08138620 con 0x7f1d080694b0 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6792/0 conn(0x7f1d080a6830 legacy=0x7f1d081d5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.82:6791/0 conn(0x7f1d080a46a0 legacy=0x7f1d081ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 monclient: found mon.b 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d081425a0 con 0x7f1d080694b0 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 5 monclient: authenticate success, global_id 184151 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 10 monclient: _renew_subs 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d081cab60 con 0x7f1d080694b0 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 10 client.184151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 10 monclient: _renew_subs 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d081d7650 con 0x7f1d080694b0 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d0d420640 1 librados: init done 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1cf8005040 con 0x7f1d080694b0 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cf8005530 con 0x7f1d080694b0 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1d057fa640 4 set_mon_vals no callback set 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.481+0000 7f1cff7fe640 10 monclient: dump: 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.503 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1d0d420640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1cec0065c0 con 0x7f1cec002860 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1cf8011990 con 0x7f1d080694b0 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(417..417 src has 299..417) ==== 5516+0+0 (unknown 2371035315 0 0) 0x7f1cf804e220 con 0x7f1d080694b0 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1cff7fe640 10 client.184151.objecter ms_dispatch 0x7f1d08000b90 osd_map(417..417 src has 299..417) 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1cff7fe640 3 client.184151.objecter handle_osd_map got epochs [417,417] > 0 2024-08-19T10:01:43.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1cff7fe640 3 client.184151.objecter handle_osd_map decoding full epoch 417 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1cff7fe640 20 client.184151.objecter dump_active .. 0 homeless 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.482+0000 7f1d077fe640 10 monclient: get_auth_request con 0x7f1ce8041960 auth_method 0 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1d067fc640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6789/0 conn(0x7f1cec002860 legacy=0x7f1cec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 10 client.184151.objecter ms_handle_connect 0x7f1cec002860 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 10 client.184151.objecter resend_mon_ops 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481588415 0 0) 0x7f1cec0065c0 con 0x7f1cec002860 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 10 monclient(hunting): my global_id is 174420 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d08138620 con 0x7f1cec002860 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.484+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 602914852 0 0) 0x7f1d08138620 con 0x7f1cec002860 2024-08-19T10:01:43.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cec0065c0 con 0x7f1cec002860 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cf0001ab0 con 0x7f1cec002860 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 10 monclient: dump: 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1284936058 0 0) 0x7f1cec0065c0 con 0x7f1cec002860 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cec00a160 con 0x7f1cec002860 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 10 client.184151.objecter ms_handle_connect 0x7f1ce8041960 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1cf0004910 con 0x7f1cec002860 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6789/0 conn(0x7f1cec002860 legacy=0x7f1cec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.485+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6789/0 conn(0x7f1cec002860 legacy=0x7f1cec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.491+0000 7f1d0d420640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.491+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1cec009e70 con 0x7f1cec009130 2024-08-19T10:01:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.493+0000 7f1cff7fe640 10 client.184151.objecter ms_handle_connect 0x7f1cec009130 2024-08-19T10:01:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 client.184151.objecter resend_mon_ops 2024-08-19T10:01:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.493+0000 7f1d067fc640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6789/0 conn(0x7f1cec009130 legacy=0x7f1cec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131936847 0 0) 0x7f1cec009e70 con 0x7f1cec009130 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient(hunting): my global_id is 174429 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cec0065c0 con 0x7f1cec009130 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2933864937 0 0) 0x7f1cec0065c0 con 0x7f1cec009130 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cec009e70 con 0x7f1cec009130 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cf0005140 con 0x7f1cec009130 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient: dump: 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.510 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.511 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3517077438 0 0) 0x7f1cec009e70 con 0x7f1cec009130 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1cec00c410 con 0x7f1cec009130 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7687 (unknown 0 0 1014673186) 0x7f1cf00074a0 con 0x7f1cec009130 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6789/0 conn(0x7f1cec009130 legacy=0x7f1cec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.494+0000 7f1cff7fe640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6789/0 conn(0x7f1cec009130 legacy=0x7f1cec0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 153535, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:43.513 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.515 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:43.516 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.517 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:43.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.521 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:43.522 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.524 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:43.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.526 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:43.528 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.530 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.531 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:43.532 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 10 librados: watch_flush enter 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 10 librados: watch_flush exit 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 20 client.184151.objecter shutdown clearing up homeless session... 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 10 client.184151.objecter successfully canceled tick 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1ce8041960 legacy=0x7f1ce8043e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 10 monclient: shutdown 2024-08-19T10:01:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 >> v1:172.21.15.1:6790/0 conn(0x7f1d080694b0 legacy=0x7f1d08141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d077fe640 1 -- 172.21.15.1:0/1516644138 reap_dead start 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d077fe640 1 -- 172.21.15.1:0/1516644138 reap_dead start 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 shutdown_connections 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 >> 172.21.15.1:0/1516644138 conn(0x7f1d080a9020 msgr2=0x7f1d080a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 shutdown_connections 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 -- 172.21.15.1:0/1516644138 wait complete. 2024-08-19T10:01:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.495+0000 7f1d0d420640 1 librados: shutdown 2024-08-19T10:01:43.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:43.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:01:43.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:01:43.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.634+0000 7f4a62b3a640 10 monclient: get_monmap_and_config 2024-08-19T10:01:43.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.634+0000 7f4a62b3a640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.634+0000 7f4a62b3a640 10 monclient: monmap: 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:43.635638+0000 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:43.635638+0000 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.637 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.635+0000 7f4a62b3a640 1 Processor -- start 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.635+0000 7f4a62b3a640 1 -- start start 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.635+0000 7f4a62b3a640 10 monclient: init 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.635+0000 7f4a62b3a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.635+0000 7f4a62b3a640 10 monclient: _add_conns ranks=[1,2,0,4,6,5,8,7,3] 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.635+0000 7f4a62b3a640 10 monclient(hunting): picked mon.noname-f con 0x7f4a5c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5c0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c0a3b40 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c0a8130 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c06ede0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a62b3a640 10 monclient(hunting): authenticate will time out at 1359.712091s 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a610d7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4a5c13f5c0 0x7f4a5c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47936/0 (socket says 172.21.15.1:47936) 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a53fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f4a5c0a6ba0 0x7f4a5c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42750/0 (socket says 172.21.15.1:42750) 2024-08-19T10:01:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.636+0000 7f4a610d7640 1 -- 172.21.15.1:0/3098111201 learned_addr learned my addr 172.21.15.1:0/3098111201 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a610d7640 1 -- 172.21.15.1:0/3098111201 >> v1:172.21.15.1:6789/0 conn(0x7f4a5c13f5c0 legacy=0x7f4a5c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a53fff640 1 -- 172.21.15.1:0/3098111201 >> v1:172.21.15.1:6790/0 conn(0x7f4a5c0a6ba0 legacy=0x7f4a5c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a608d6640 1 -- 172.21.15.1:0/3098111201 >> v1:172.21.15.82:6789/0 conn(0x7f4a5c0a67c0 legacy=0x7f4a5c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289889969 0 0) 0x7f4a5c0a3b40 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): my global_id is 174438 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a44003520 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174783630 0 0) 0x7f4a5c06ede0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient(hunting): my global_id is 184160 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5c0a3b40 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3481645272 0 0) 0x7f4a44003520 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 >> v1:172.21.15.1:6790/0 conn(0x7f4a5c0a6ba0 legacy=0x7f4a5c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 >> v1:172.21.15.82:6789/0 conn(0x7f4a5c0a67c0 legacy=0x7f4a5c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 monclient: found mon.noname-a 2024-08-19T10:01:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c0a92a0 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a54001e60 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a62b3a640 5 monclient: authenticate success, global_id 174438 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a62b3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: dump: 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.646 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a62b3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a54004c20 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a62b3a640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 1 -- 172.21.15.1:0/3098111201 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a54005110 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a62b3a640 4 set_mon_vals no callback set 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.638+0000 7f4a537fe640 10 monclient: dump: 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 10 monclient: shutdown 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3098111201 >> v1:172.21.15.1:6789/0 conn(0x7f4a5c13f5c0 legacy=0x7f4a5c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3098111201 shutdown_connections 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3098111201 >> 172.21.15.1:0/3098111201 conn(0x7f4a5c0ac4b0 msgr2=0x7f4a5c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3098111201 shutdown_connections 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.639+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3098111201 wait complete. 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient: monmap: 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 1 librados: starting msgr at 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 1 librados: starting objecter 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 1 Processor -- start 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 1 -- start start 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 1 librados: setting wanted keys 2024-08-19T10:01:43.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 1 librados: calling monclient init 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient: init 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient: _add_conns ranks=[0,2,7,3,6,8,1,4,5] 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.640+0000 7f4a62b3a640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5c0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a62b3a640 10 monclient(hunting): picked mon.noname-i con 0x7f4a5c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a62b3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c13a100 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a62b3a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c13a3e0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a62b3a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c13a6c0 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a62b3a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a62b3a640 10 monclient(hunting): authenticate will time out at 1359.717061s 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a608d6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4a5c0a67c0 0x7f4a5c09fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47950/0 (socket says 172.21.15.1:47950) 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a53fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f4a5c0a6ba0 0x7f4a5c0a02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42752/0 (socket says 172.21.15.1:42752) 2024-08-19T10:01:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.641+0000 7f4a608d6640 1 -- 172.21.15.1:0/3541569566 learned_addr learned my addr 172.21.15.1:0/3541569566 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a608d6640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6789/0 conn(0x7f4a5c0a67c0 legacy=0x7f4a5c09fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a53fff640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a5c0a6ba0 legacy=0x7f4a5c0a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 client.?.objecter ms_handle_connect 0x7f4a5c0a67c0 2024-08-19T10:01:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 client.?.objecter ms_handle_connect 0x7f4a5c0a6ba0 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162826477 0 0) 0x7f4a5c13a100 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): my global_id is 174447 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a48004eb0 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104676295 0 0) 0x7f4a5c13a3e0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): my global_id is 184169 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a610d7640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.82:6792/0 conn(0x7f4a5c13f5c0 legacy=0x7f4a5c1d5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5c13a100 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 client.?.objecter ms_handle_connect 0x7f4a5c13f5c0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1678366530 0 0) 0x7f4a5c13a100 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a5c13a3e0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3233683353 0 0) 0x7f4a48004eb0 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a5c13a100 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a4c001970 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): dump: 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477865201 0 0) 0x7f4a5c13a6c0 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): my global_id is 144925 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a48004eb0 con 0x7f4a5c13f5c0 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a58003610 con 0x7f4a5c0a67c0 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient(hunting): dump: 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2438996783 0 0) 0x7f4a5c13a3e0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.82:6792/0 conn(0x7f4a5c13f5c0 legacy=0x7f4a5c1d5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6789/0 conn(0x7f4a5c0a67c0 legacy=0x7f4a5c09fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 monclient: found mon.b 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c0a09f0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 5 monclient: authenticate success, global_id 184169 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 10 monclient: _renew_subs 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5c139e20 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 10 client.184169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 10 monclient: _renew_subs 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.643+0000 7f4a62b3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4a5c1d72e0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a62b3a640 1 librados: init done 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a4c004dd0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a4c0052c0 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a52ffd640 4 set_mon_vals no callback set 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 10 monclient: dump: 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a62b3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a3c0065c0 con 0x7f4a3c002860 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.644+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4a4c011720 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.645+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(417..417 src has 299..417) ==== 5516+0+0 (unknown 2371035315 0 0) 0x7f4a4c04e260 con 0x7f4a5c0a6ba0 2024-08-19T10:01:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.645+0000 7f4a517fa640 10 client.184169.objecter ms_dispatch 0x7f4a5c000b90 osd_map(417..417 src has 299..417) 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.645+0000 7f4a517fa640 3 client.184169.objecter handle_osd_map got epochs [417,417] > 0 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.645+0000 7f4a517fa640 3 client.184169.objecter handle_osd_map decoding full epoch 417 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.645+0000 7f4a517fa640 20 client.184169.objecter dump_active .. 0 homeless 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.645+0000 7f4a610d7640 10 monclient: get_auth_request con 0x7f4a480416b0 auth_method 0 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.646+0000 7f4a517fa640 10 client.184169.objecter ms_handle_connect 0x7f4a3c002860 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.646+0000 7f4a517fa640 10 client.184169.objecter resend_mon_ops 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.646+0000 7f4a608d6640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a3c002860 legacy=0x7f4a3c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 530522375 0 0) 0x7f4a3c0065c0 con 0x7f4a3c002860 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient(hunting): my global_id is 184178 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5c13a3e0 con 0x7f4a3c002860 2024-08-19T10:01:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3105702423 0 0) 0x7f4a5c13a3e0 con 0x7f4a3c002860 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a3c0065c0 con 0x7f4a3c002860 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a58004140 con 0x7f4a3c002860 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient: dump: 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1390611229 0 0) 0x7f4a3c0065c0 con 0x7f4a3c002860 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a3c00a160 con 0x7f4a3c002860 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 client.184169.objecter ms_handle_connect 0x7f4a480416b0 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4a58004450 con 0x7f4a3c002860 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a3c002860 legacy=0x7f4a3c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.647+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a3c002860 legacy=0x7f4a3c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.653+0000 7f4a62b3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.653+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a3c009e70 con 0x7f4a3c009130 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a608d6640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a3c009130 legacy=0x7f4a3c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 client.184169.objecter ms_handle_connect 0x7f4a3c009130 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 client.184169.objecter resend_mon_ops 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252323308 0 0) 0x7f4a3c009e70 con 0x7f4a3c009130 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient(hunting): my global_id is 184187 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a3c0065c0 con 0x7f4a3c009130 2024-08-19T10:01:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 130825619 0 0) 0x7f4a3c0065c0 con 0x7f4a3c009130 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a3c009e70 con 0x7f4a3c009130 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4a58005cf0 con 0x7f4a3c009130 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient: dump: 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1144418047 0 0) 0x7f4a3c009e70 con 0x7f4a3c009130 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a3c00c410 con 0x7f4a3c009130 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 1728141749) 0x7f4a58008010 con 0x7f4a3c009130 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a3c009130 legacy=0x7f4a3c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.656+0000 7f4a517fa640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a3c009130 legacy=0x7f4a3c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 20904, 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:43.674 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:43.675 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:43.676 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:43.677 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:43.678 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.679 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:43.682 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.683 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:43.686 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:43.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:43.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.693 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:43.693 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:43.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 10 librados: watch_flush enter 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 10 librados: watch_flush exit 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 20 client.184169.objecter shutdown clearing up homeless session... 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 10 client.184169.objecter successfully canceled tick 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4a480416b0 legacy=0x7f4a48043b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 10 monclient: shutdown 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 >> v1:172.21.15.1:6790/0 conn(0x7f4a5c0a6ba0 legacy=0x7f4a5c0a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a610d7640 1 -- 172.21.15.1:0/3541569566 reap_dead start 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a610d7640 1 -- 172.21.15.1:0/3541569566 reap_dead start 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 shutdown_connections 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 >> 172.21.15.1:0/3541569566 conn(0x7f4a5c0ac4b0 msgr2=0x7f4a5c0aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 shutdown_connections 2024-08-19T10:01:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 -- 172.21.15.1:0/3541569566 wait complete. 2024-08-19T10:01:43.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.657+0000 7f4a62b3a640 1 librados: shutdown 2024-08-19T10:01:43.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:43.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:01:43.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:01:43.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=608501173}) 2024-08-19T10:01:43.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=349707815}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112740402}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1385011510}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3761427576}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2239059303}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.671+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3596809822}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.672+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.672+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774824433}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.672+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3599336052}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130829482}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216476532}) 2024-08-19T10:01:43.702 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3548029818}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2280478931}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1742625071}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3343941809}) 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.703 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3352228922}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835081722}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=398305525}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381303414}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2404494125}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1832051973}) 2024-08-19T10:01:43.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.682+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=253645102}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=363318363}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2008677417}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=702063111}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595108562}) 2024-08-19T10:01:43.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.683+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1181035506}) 2024-08-19T10:01:43.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.804+0000 7f574ecf6640 10 monclient: get_monmap_and_config 2024-08-19T10:01:43.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.804+0000 7f574ecf6640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.804+0000 7f574ecf6640 10 monclient: monmap: 2024-08-19T10:01:43.805 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:43.805242+0000 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:43.805242+0000 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.806 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.807 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.807 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.804+0000 7f574ecf6640 1 Processor -- start 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 1 -- start start 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient: init 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient: _add_conns ranks=[0,7,8,6,4,3,1,5,2] 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient(hunting): picked mon.noname-a con 0x7f57480a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient(hunting): picked mon.noname-i con 0x7f57480a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient(hunting): picked mon.noname-e con 0x7f574813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57480a3b40 con 0x7f57480a67c0 2024-08-19T10:01:43.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57480a8130 con 0x7f57480a6ba0 2024-08-19T10:01:43.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574806ede0 con 0x7f574813f5c0 2024-08-19T10:01:43.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ecf6640 10 monclient(hunting): authenticate will time out at 1359.881606s 2024-08-19T10:01:43.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ca92640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f57480a67c0 0x7f57480a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47960/0 (socket says 172.21.15.1:47960) 2024-08-19T10:01:43.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.805+0000 7f574ca92640 1 -- 172.21.15.1:0/2462397356 learned_addr learned my addr 172.21.15.1:0/2462397356 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.807+0000 7f574d293640 1 -- 172.21.15.1:0/2462397356 >> v1:172.21.15.1:6793/0 conn(0x7f574813f5c0 legacy=0x7f57481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.807+0000 7f574ca92640 1 -- 172.21.15.1:0/2462397356 >> v1:172.21.15.1:6789/0 conn(0x7f57480a67c0 legacy=0x7f57480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.807+0000 7f5747fff640 1 -- 172.21.15.1:0/2462397356 >> v1:172.21.15.82:6792/0 conn(0x7f57480a6ba0 legacy=0x7f57480a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3859096896 0 0) 0x7f574806ede0 con 0x7f574813f5c0 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): my global_id is 184175 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5738003520 con 0x7f574813f5c0 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635579562 0 0) 0x7f57480a3b40 con 0x7f57480a67c0 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): my global_id is 174456 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f574806ede0 con 0x7f57480a67c0 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574046872 0 0) 0x7f57480a8130 con 0x7f57480a6ba0 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient(hunting): my global_id is 144934 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57480a3b40 con 0x7f57480a6ba0 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2667817267 0 0) 0x7f574806ede0 con 0x7f57480a67c0 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 >> v1:172.21.15.1:6793/0 conn(0x7f574813f5c0 legacy=0x7f57481419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 >> v1:172.21.15.82:6792/0 conn(0x7f57480a6ba0 legacy=0x7f57480a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 monclient: found mon.noname-a 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57480a92a0 con 0x7f57480a67c0 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f573c001a40 con 0x7f57480a67c0 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 5 monclient: authenticate success, global_id 174456 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: dump: 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.816 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f573c001d10 con 0x7f57480a67c0 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 1 -- 172.21.15.1:0/2462397356 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f573c004be0 con 0x7f57480a67c0 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 4 set_mon_vals no callback set 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f57477fe640 10 monclient: dump: 2024-08-19T10:01:43.817 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.818 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 10 monclient: shutdown 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.808+0000 7f574ecf6640 1 -- 172.21.15.1:0/2462397356 >> v1:172.21.15.1:6789/0 conn(0x7f57480a67c0 legacy=0x7f57480a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 -- 172.21.15.1:0/2462397356 shutdown_connections 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 -- 172.21.15.1:0/2462397356 >> 172.21.15.1:0/2462397356 conn(0x7f57480ac4b0 msgr2=0x7f57480a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 -- 172.21.15.1:0/2462397356 shutdown_connections 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 -- 172.21.15.1:0/2462397356 wait complete. 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 10 monclient: monmap: 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:43.819 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 librados: starting msgr at 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 librados: starting objecter 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 Processor -- start 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 -- start start 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 librados: setting wanted keys 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 1 librados: calling monclient init 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.809+0000 7f574ecf6640 10 monclient: init 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient: _add_conns ranks=[8,5,2,3,0,6,7,1,4] 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient(hunting): picked mon.noname-e con 0x7f57480a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient(hunting): picked mon.noname-h con 0x7f57480a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient(hunting): picked mon.noname-b con 0x7f574813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5748139ad0 con 0x7f574813f5c0 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574813a990 con 0x7f57480a6ba0 2024-08-19T10:01:43.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5748142e70 con 0x7f57480a67c0 2024-08-19T10:01:43.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ecf6640 10 monclient(hunting): authenticate will time out at 1359.886613s 2024-08-19T10:01:43.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ca92640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f57480a67c0 0x7f57480a0010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42764/0 (socket says 172.21.15.1:42764) 2024-08-19T10:01:43.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.810+0000 7f574ca92640 1 -- 172.21.15.1:0/613669156 learned_addr learned my addr 172.21.15.1:0/613669156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f574ca92640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6793/0 conn(0x7f57480a67c0 legacy=0x7f57480a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 client.?.objecter ms_handle_connect 0x7f57480a67c0 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 client.?.objecter ms_handle_connect 0x7f574813f5c0 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f574d293640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6790/0 conn(0x7f574813f5c0 legacy=0x7f57481d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371734316 0 0) 0x7f5748142e70 con 0x7f57480a67c0 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): my global_id is 184184 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f572c004eb0 con 0x7f57480a67c0 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662710733 0 0) 0x7f5748139ad0 con 0x7f574813f5c0 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 monclient(hunting): my global_id is 184196 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5748142e70 con 0x7f574813f5c0 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f5747fff640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.82:6791/0 conn(0x7f57480a6ba0 legacy=0x7f57480a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 client.?.objecter ms_handle_connect 0x7f57480a6ba0 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.812+0000 7f57457fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232574684 0 0) 0x7f574813a990 con 0x7f57480a6ba0 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): my global_id is 27428 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5748139ad0 con 0x7f57480a6ba0 2024-08-19T10:01:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1313316351 0 0) 0x7f5748142e70 con 0x7f574813f5c0 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f574813a990 con 0x7f574813f5c0 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5740003990 con 0x7f574813f5c0 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): dump: 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3609208595 0 0) 0x7f572c004eb0 con 0x7f57480a67c0 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5748142e70 con 0x7f57480a67c0 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f573c004b20 con 0x7f57480a67c0 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): dump: 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.833 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1496310603 0 0) 0x7f5748139ad0 con 0x7f57480a6ba0 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f572c004eb0 con 0x7f57480a6ba0 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5734001350 con 0x7f57480a6ba0 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient(hunting): dump: 2024-08-19T10:01:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1841872943 0 0) 0x7f574813a990 con 0x7f574813f5c0 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6793/0 conn(0x7f57480a67c0 legacy=0x7f57480a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.82:6791/0 conn(0x7f57480a6ba0 legacy=0x7f57480a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 monclient: found mon.b 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57480a17d0 con 0x7f574813f5c0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 5 monclient: authenticate success, global_id 184196 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 10 monclient: _renew_subs 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f574813a6b0 con 0x7f574813f5c0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 10 client.184196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 10 monclient: _renew_subs 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f57480a0e30 con 0x7f574813f5c0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f574ecf6640 1 librados: init done 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5740003e60 con 0x7f574813f5c0 2024-08-19T10:01:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f57400043d0 con 0x7f574813f5c0 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f5746ffd640 4 set_mon_vals no callback set 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.813+0000 7f57457fa640 10 monclient: dump: 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f574ecf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57280065c0 con 0x7f5728002860 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5740004570 con 0x7f574813f5c0 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(418..418 src has 299..418) ==== 5516+0+0 (unknown 2446891144 0 0) 0x7f574004e410 con 0x7f574813f5c0 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f57457fa640 10 client.184196.objecter ms_dispatch 0x7f5748000b90 osd_map(418..418 src has 299..418) 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f57457fa640 3 client.184196.objecter handle_osd_map got epochs [418,418] > 0 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f57457fa640 3 client.184196.objecter handle_osd_map decoding full epoch 418 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f57457fa640 20 client.184196.objecter dump_active .. 0 homeless 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.814+0000 7f5747fff640 10 monclient: get_auth_request con 0x7f572c041800 auth_method 0 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f574ca92640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6791/0 conn(0x7f5728002860 legacy=0x7f5728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 10 client.184196.objecter ms_handle_connect 0x7f5728002860 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 10 client.184196.objecter resend_mon_ops 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992084883 0 0) 0x7f57280065c0 con 0x7f5728002860 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 10 monclient(hunting): my global_id is 27769 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f574813a990 con 0x7f5728002860 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.816+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 21824407 0 0) 0x7f574813a990 con 0x7f5728002860 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57280065c0 con 0x7f5728002860 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f573c001c90 con 0x7f5728002860 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 10 monclient: dump: 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3348693045 0 0) 0x7f57280065c0 con 0x7f5728002860 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f572800a160 con 0x7f5728002860 2024-08-19T10:01:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 10 client.184196.objecter ms_handle_connect 0x7f572c041800 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f573c0075d0 con 0x7f5728002860 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6791/0 conn(0x7f5728002860 legacy=0x7f5728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.817+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6791/0 conn(0x7f5728002860 legacy=0x7f5728004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.823+0000 7f574ecf6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.823+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5728009e70 con 0x7f5728009130 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 client.184196.objecter ms_handle_connect 0x7f5728009130 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 client.184196.objecter resend_mon_ops 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f574ca92640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6791/0 conn(0x7f5728009130 legacy=0x7f57280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758372350 0 0) 0x7f5728009e70 con 0x7f5728009130 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient(hunting): my global_id is 27778 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57280065c0 con 0x7f5728009130 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2084607660 0 0) 0x7f57280065c0 con 0x7f5728009130 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5728009e70 con 0x7f5728009130 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f573c005c80 con 0x7f5728009130 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient: dump: 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3322271460 0 0) 0x7f5728009e70 con 0x7f5728009130 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f572800c410 con 0x7f5728009130 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3572604570) 0x7f573c007860 con 0x7f5728009130 2024-08-19T10:01:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6791/0 conn(0x7f5728009130 legacy=0x7f57280095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.826+0000 7f57457fa640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6791/0 conn(0x7f5728009130 legacy=0x7f57280095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 710097, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:43.860 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:43.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:43.866 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:43.866 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 10 librados: watch_flush enter 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 10 librados: watch_flush exit 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 20 client.184196.objecter shutdown clearing up homeless session... 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 10 client.184196.objecter successfully canceled tick 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.82:6812/2549104415 conn(0x7f572c041800 legacy=0x7f572c043cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 10 monclient: shutdown 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 >> v1:172.21.15.1:6790/0 conn(0x7f574813f5c0 legacy=0x7f57481d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574d293640 1 -- 172.21.15.1:0/613669156 reap_dead start 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574d293640 1 -- 172.21.15.1:0/613669156 reap_dead start 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 shutdown_connections 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 >> 172.21.15.1:0/613669156 conn(0x7f57480ac4b0 msgr2=0x7f57480aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 shutdown_connections 2024-08-19T10:01:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 -- 172.21.15.1:0/613669156 wait complete. 2024-08-19T10:01:43.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.827+0000 7f574ecf6640 1 librados: shutdown 2024-08-19T10:01:43.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:43.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:01:43.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1269572655}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491318855}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1163330209}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=129744188}) 2024-08-19T10:01:43.880 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=424468373}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.877+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1238344226}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3257419984}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436026560}) 2024-08-19T10:01:43.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:43.878+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1957653835}) 2024-08-19T10:01:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.968+0000 7fd5fbcb3640 10 monclient: get_monmap_and_config 2024-08-19T10:01:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.968+0000 7fd5fbcb3640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.968+0000 7fd5fbcb3640 10 monclient: monmap: 2024-08-19T10:01:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:43.969068+0000 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:43.969068+0000 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.968+0000 7fd5fbcb3640 1 Processor -- start 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.968+0000 7fd5fbcb3640 1 -- start start 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.968+0000 7fd5fbcb3640 10 monclient: init 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient: _add_conns ranks=[4,8,2,0,7,3,5,6,1] 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient(hunting): picked mon.noname-c con 0x7fd5f40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient(hunting): picked mon.noname-e con 0x7fd5f40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient(hunting): picked mon.noname-b con 0x7fd5f413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f40a3b40 con 0x7fd5f413f5c0 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f40a8130 con 0x7fd5f40a67c0 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f406ede0 con 0x7fd5f40a6ba0 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fbcb3640 10 monclient(hunting): authenticate will time out at 1360.045418s 2024-08-19T10:01:43.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fa250640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fd5f413f5c0 0x7fd5f41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42788/0 (socket says 172.21.15.1:42788) 2024-08-19T10:01:43.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5f9a4f640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd5f40a67c0 0x7fd5f40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44764/0 (socket says 172.21.15.1:44764) 2024-08-19T10:01:43.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5f924e640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd5f40a6ba0 0x7fd5f40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42772/0 (socket says 172.21.15.1:42772) 2024-08-19T10:01:43.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.969+0000 7fd5fa250640 1 -- 172.21.15.1:0/1746252933 learned_addr learned my addr 172.21.15.1:0/1746252933 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f9a4f640 1 -- 172.21.15.1:0/1746252933 >> v1:172.21.15.1:6791/0 conn(0x7fd5f40a67c0 legacy=0x7fd5f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5fa250640 1 -- 172.21.15.1:0/1746252933 >> v1:172.21.15.1:6790/0 conn(0x7fd5f413f5c0 legacy=0x7fd5f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f924e640 1 -- 172.21.15.1:0/1746252933 >> v1:172.21.15.1:6793/0 conn(0x7fd5f40a6ba0 legacy=0x7fd5f40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358264169 0 0) 0x7fd5f40a8130 con 0x7fd5f40a67c0 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): my global_id is 27787 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5dc003520 con 0x7fd5f40a67c0 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104364139 0 0) 0x7fd5f40a3b40 con 0x7fd5f413f5c0 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): my global_id is 184205 2024-08-19T10:01:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f40a8130 con 0x7fd5f413f5c0 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137824988 0 0) 0x7fd5f406ede0 con 0x7fd5f40a6ba0 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 10 monclient(hunting): my global_id is 184193 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.971+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f40a3b40 con 0x7fd5f40a6ba0 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2703299752 0 0) 0x7fd5dc003520 con 0x7fd5f40a67c0 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 >> v1:172.21.15.1:6793/0 conn(0x7fd5f40a6ba0 legacy=0x7fd5f40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 >> v1:172.21.15.1:6790/0 conn(0x7fd5f413f5c0 legacy=0x7fd5f41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 monclient: found mon.noname-c 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f40a92a0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5ec001ae0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: dump: 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 5 monclient: authenticate success, global_id 27787 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5ec0047e0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 4 set_mon_vals no callback set 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 1 -- 172.21.15.1:0/1746252933 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5ec004d90 con 0x7fd5f40a67c0 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5f8a4d640 10 monclient: dump: 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 10 monclient: shutdown 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1746252933 >> v1:172.21.15.1:6791/0 conn(0x7fd5f40a67c0 legacy=0x7fd5f40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1746252933 shutdown_connections 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1746252933 >> 172.21.15.1:0/1746252933 conn(0x7fd5f40ac4b0 msgr2=0x7fd5f40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1746252933 shutdown_connections 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.972+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1746252933 wait complete. 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 10 monclient: build_initial_monmap 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 10 monclient: monmap: 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 1 librados: starting msgr at 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 1 librados: starting objecter 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 1 Processor -- start 2024-08-19T10:01:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 1 -- start start 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 1 librados: setting wanted keys 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 1 librados: calling monclient init 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.973+0000 7fd5fbcb3640 10 monclient: init 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient: _add_conns ranks=[0,7,8,6,4,5,2,3,1] 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient(hunting): picked mon.noname-a con 0x7fd5f40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient(hunting): picked mon.noname-i con 0x7fd5f40a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient(hunting): picked mon.noname-e con 0x7fd5f413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f40a1500 con 0x7fd5f40a67c0 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f413a350 con 0x7fd5f40a6ba0 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f413a630 con 0x7fd5f413f5c0 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5fbcb3640 10 monclient(hunting): authenticate will time out at 1360.050266s 2024-08-19T10:01:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5f9a4f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd5f40a67c0 0x7fd5f413eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47966/0 (socket says 172.21.15.1:47966) 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.974+0000 7fd5f9a4f640 1 -- 172.21.15.1:0/1587864323 learned_addr learned my addr 172.21.15.1:0/1587864323 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5fa250640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6793/0 conn(0x7fd5f413f5c0 legacy=0x7fd5f41d6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 client.?.objecter ms_handle_connect 0x7fd5f413f5c0 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 client.?.objecter ms_handle_connect 0x7fd5f40a67c0 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f9a4f640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6789/0 conn(0x7fd5f40a67c0 legacy=0x7fd5f413eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258982613 0 0) 0x7fd5f413a630 con 0x7fd5f413f5c0 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): my global_id is 184202 2024-08-19T10:01:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5e0004eb0 con 0x7fd5f413f5c0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055832943 0 0) 0x7fd5f40a1500 con 0x7fd5f40a67c0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): my global_id is 174465 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f413a630 con 0x7fd5f40a67c0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f924e640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6792/0 conn(0x7fd5f40a6ba0 legacy=0x7fd5f41d2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 client.?.objecter ms_handle_connect 0x7fd5f40a6ba0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3101586261 0 0) 0x7fd5e0004eb0 con 0x7fd5f413f5c0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5f40a1500 con 0x7fd5f413f5c0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5e4002090 con 0x7fd5f413f5c0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): dump: 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1656262965 0 0) 0x7fd5f413a630 con 0x7fd5f40a67c0 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5e0004eb0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4255146987 0 0) 0x7fd5f413a350 con 0x7fd5f40a6ba0 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): my global_id is 144943 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f413a630 con 0x7fd5f40a6ba0 2024-08-19T10:01:43.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5ec004cd0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.976+0000 7fd5f27fc640 10 monclient(hunting): dump: 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3680038379 0 0) 0x7fd5e0004eb0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6793/0 conn(0x7fd5f413f5c0 legacy=0x7fd5f41d6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6792/0 conn(0x7fd5f40a6ba0 legacy=0x7fd5f41d2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 monclient: found mon.a 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:43.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f4139a20 con 0x7fd5f40a67c0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 5 monclient: authenticate success, global_id 174465 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 10 monclient: _renew_subs 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f413f280 con 0x7fd5f40a67c0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 10 client.174465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 10 monclient: _renew_subs 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5f41cd9a0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5fbcb3640 1 librados: init done 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5ec0047e0 con 0x7fd5f40a67c0 2024-08-19T10:01:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5ec001a80 con 0x7fd5f40a67c0 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f3fff640 4 set_mon_vals no callback set 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.977+0000 7fd5f27fc640 10 monclient: dump: 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5fbcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd5d40065c0 con 0x7fd5d4002860 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd5ec011900 con 0x7fd5f40a67c0 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(418..418 src has 299..418) ==== 5516+0+0 (unknown 2446891144 0 0) 0x7fd5ec04e230 con 0x7fd5f40a67c0 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5f27fc640 10 client.174465.objecter ms_dispatch 0x7fd5f4000b90 osd_map(418..418 src has 299..418) 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5f27fc640 3 client.174465.objecter handle_osd_map got epochs [418,418] > 0 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5f27fc640 3 client.174465.objecter handle_osd_map decoding full epoch 418 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5f27fc640 20 client.174465.objecter dump_active .. 0 homeless 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.978+0000 7fd5fa250640 10 monclient: get_auth_request con 0x7fd5e0049dc0 auth_method 0 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f924e640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6792/0 conn(0x7fd5d4002860 legacy=0x7fd5d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 client.174465.objecter ms_handle_connect 0x7fd5d4002860 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 client.174465.objecter resend_mon_ops 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895810883 0 0) 0x7fd5d40065c0 con 0x7fd5d4002860 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient(hunting): my global_id is 174480 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5e0004eb0 con 0x7fd5d4002860 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 212312353 0 0) 0x7fd5e0004eb0 con 0x7fd5d4002860 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5d40065c0 con 0x7fd5d4002860 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5e80047e0 con 0x7fd5d4002860 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 monclient: dump: 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 10 client.174465.objecter ms_handle_connect 0x7fd5e0049dc0 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5fa250640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd5e0049dc0 legacy=0x7fd5e004c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5fa250640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd5e0049dc0 legacy=0x7fd5e004c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5fa250640 1 --1- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd5e0049dc0 0x7fd5e004c280 :-1 s=OPENED pgs=3915 cs=1 l=1).handle_message read tag failed 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5fa250640 1 --1- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd5e0049dc0 0x7fd5e004c280 :-1 s=OPENED pgs=3915 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.980+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd5e0049dc0 legacy=0x7fd5e004c280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3248291783 0 0) 0x7fd5d40065c0 con 0x7fd5d4002860 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5d400a160 con 0x7fd5d4002860 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd5e80048e0 con 0x7fd5d4002860 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6792/0 conn(0x7fd5d4002860 legacy=0x7fd5d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.981+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6792/0 conn(0x7fd5d4002860 legacy=0x7fd5d4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.987+0000 7fd5fbcb3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.987+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd5d4009e40 con 0x7fd5d4009100 2024-08-19T10:01:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.989+0000 7fd5f924e640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6792/0 conn(0x7fd5d4009100 legacy=0x7fd5d4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.989+0000 7fd5f27fc640 10 client.174465.objecter ms_handle_connect 0x7fd5d4009100 2024-08-19T10:01:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.989+0000 7fd5f27fc640 10 client.174465.objecter resend_mon_ops 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.989+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482705463 0 0) 0x7fd5d4009e40 con 0x7fd5d4009100 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient(hunting): my global_id is 174489 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5d40065c0 con 0x7fd5d4009100 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 611632014 0 0) 0x7fd5d40065c0 con 0x7fd5d4009100 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5d4009e40 con 0x7fd5d4009100 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd5e8005110 con 0x7fd5d4009100 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient: dump: 2024-08-19T10:01:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1248620533 0 0) 0x7fd5d4009e40 con 0x7fd5d4009100 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd5d400c3f0 con 0x7fd5d4009100 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 655185570) 0x7fd5e8007430 con 0x7fd5d4009100 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6792/0 conn(0x7fd5d4009100 legacy=0x7fd5d4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5f27fc640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6792/0 conn(0x7fd5d4009100 legacy=0x7fd5d4009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 154024, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:44.010 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.014 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:44.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:44.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:44.018 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.019 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.022 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.023 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:44.025 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.028 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:44.029 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fa250640 1 -- 172.21.15.1:0/1587864323 reap_dead start 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 10 librados: watch_flush enter 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 10 librados: watch_flush exit 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 20 client.174465.objecter shutdown clearing up homeless session... 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 10 client.174465.objecter successfully canceled tick 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 10 monclient: shutdown 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 >> v1:172.21.15.1:6789/0 conn(0x7fd5f40a67c0 legacy=0x7fd5f413eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 shutdown_connections 2024-08-19T10:01:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.990+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 >> 172.21.15.1:0/1587864323 conn(0x7fd5f40ac4b0 msgr2=0x7fd5f40aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.991+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 shutdown_connections 2024-08-19T10:01:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.991+0000 7fd5fbcb3640 1 -- 172.21.15.1:0/1587864323 wait complete. 2024-08-19T10:01:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:43.991+0000 7fd5fbcb3640 1 librados: shutdown 2024-08-19T10:01:44.032 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:44.032 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:01:44.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:01:44.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.170+0000 7fd2744a6640 10 monclient: get_monmap_and_config 2024-08-19T10:01:44.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.170+0000 7fd2744a6640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.170+0000 7fd2744a6640 10 monclient: monmap: 2024-08-19T10:01:44.171 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:44.171184+0000 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:44.171184+0000 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.173 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.170+0000 7fd2744a6640 1 Processor -- start 2024-08-19T10:01:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.170+0000 7fd2744a6640 1 -- start start 2024-08-19T10:01:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.170+0000 7fd2744a6640 10 monclient: init 2024-08-19T10:01:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient: _add_conns ranks=[1,7,0,4,6,5,2,3,8] 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient(hunting): picked mon.noname-f con 0x7fd26c0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient(hunting): picked mon.noname-i con 0x7fd26c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient(hunting): picked mon.noname-a con 0x7fd26c0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c0a3c70 con 0x7fd26c0a46a0 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c0a33d0 con 0x7fd26c0a6830 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c0a36b0 con 0x7fd26c0694b0 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd2744a6640 10 monclient(hunting): authenticate will time out at 1360.247565s 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd272a43640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd26c0a46a0 0x7fd26c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47968/0 (socket says 172.21.15.1:47968) 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.171+0000 7fd272a43640 1 -- 172.21.15.1:0/1832959484 learned_addr learned my addr 172.21.15.1:0/1832959484 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.173+0000 7fd272a43640 1 -- 172.21.15.1:0/1832959484 >> v1:172.21.15.1:6789/0 conn(0x7fd26c0a46a0 legacy=0x7fd26c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271a41640 1 -- 172.21.15.1:0/1832959484 >> v1:172.21.15.82:6792/0 conn(0x7fd26c0694b0 legacy=0x7fd26c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd272242640 1 -- 172.21.15.1:0/1832959484 >> v1:172.21.15.82:6789/0 conn(0x7fd26c0a6830 legacy=0x7fd26c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 291996088 0 0) 0x7fd26c0a36b0 con 0x7fd26c0694b0 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): my global_id is 144952 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd254003520 con 0x7fd26c0694b0 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282626456 0 0) 0x7fd26c0a33d0 con 0x7fd26c0a6830 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient(hunting): my global_id is 174439 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd26c0a36b0 con 0x7fd26c0a6830 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1440196458 0 0) 0x7fd254003520 con 0x7fd26c0694b0 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 >> v1:172.21.15.82:6789/0 conn(0x7fd26c0a6830 legacy=0x7fd26c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 >> v1:172.21.15.1:6789/0 conn(0x7fd26c0a46a0 legacy=0x7fd26c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 monclient: found mon.noname-i 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26c06dbb0 con 0x7fd26c0694b0 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd260001bf0 con 0x7fd26c0694b0 2024-08-19T10:01:44.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: dump: 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd2744a6640 5 monclient: authenticate success, global_id 144952 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd2744a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd260003610 con 0x7fd26c0694b0 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 1 -- 172.21.15.1:0/1832959484 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd260003b40 con 0x7fd26c0694b0 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd2744a6640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd2744a6640 4 set_mon_vals no callback set 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.174+0000 7fd271240640 10 monclient: dump: 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 10 monclient: shutdown 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 1 -- 172.21.15.1:0/1832959484 >> v1:172.21.15.82:6792/0 conn(0x7fd26c0694b0 legacy=0x7fd26c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 1 -- 172.21.15.1:0/1832959484 shutdown_connections 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 1 -- 172.21.15.1:0/1832959484 >> 172.21.15.1:0/1832959484 conn(0x7fd26c0a9020 msgr2=0x7fd26c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 1 -- 172.21.15.1:0/1832959484 shutdown_connections 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 1 -- 172.21.15.1:0/1832959484 wait complete. 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.175+0000 7fd2744a6640 10 monclient: monmap: 2024-08-19T10:01:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 librados: starting msgr at 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 librados: starting objecter 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 Processor -- start 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 -- start start 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 librados: setting wanted keys 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 librados: calling monclient init 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 10 monclient: init 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 10 monclient: _add_conns ranks=[6,2,0,7,3,1,5,8,4] 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 10 monclient(hunting): picked mon.noname-d con 0x7fd26c0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 10 monclient(hunting): picked mon.noname-b con 0x7fd26c0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 10 monclient(hunting): picked mon.noname-a con 0x7fd26c0a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.176+0000 7fd2744a6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c137d20 con 0x7fd26c0a6830 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd2744a6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c1382f0 con 0x7fd26c0a46a0 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd271a41640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fd26c0a46a0 0x7fd26c135640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42798/0 (socket says 172.21.15.1:42798) 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd272242640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fd26c0694b0 0x7fd26c134f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43116/0 (socket says 172.21.15.1:43116) 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd271a41640 1 -- 172.21.15.1:0/2553410891 learned_addr learned my addr 172.21.15.1:0/2553410891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c136830 con 0x7fd26c0694b0 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd2744a6640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.177+0000 7fd2744a6640 10 monclient(hunting): authenticate will time out at 1360.253143s 2024-08-19T10:01:44.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd272a43640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6789/0 conn(0x7fd26c0a6830 legacy=0x7fd26c135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd271a41640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6790/0 conn(0x7fd26c0a46a0 legacy=0x7fd26c135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd272242640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6792/0 conn(0x7fd26c0694b0 legacy=0x7fd26c134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 client.?.objecter ms_handle_connect 0x7fd26c0a6830 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 client.?.objecter ms_handle_connect 0x7fd26c0a46a0 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 client.?.objecter ms_handle_connect 0x7fd26c0694b0 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531852470 0 0) 0x7fd26c136830 con 0x7fd26c0694b0 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): my global_id is 174498 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd258004eb0 con 0x7fd26c0694b0 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1427070884 0 0) 0x7fd258004eb0 con 0x7fd26c0694b0 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd26c136830 con 0x7fd26c0694b0 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd25c002070 con 0x7fd26c0694b0 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): dump: 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.179+0000 7fd26affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563560364}) 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3271503031}) 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3895004468}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2538708417}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1053981192}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1203506973}) 2024-08-19T10:01:44.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2227063332}) 2024-08-19T10:01:44.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911044630}) 2024-08-19T10:01:44.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.181+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1360142803}) 2024-08-19T10:01:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793059392 0 0) 0x7fd26c137d20 con 0x7fd26c0a6830 2024-08-19T10:01:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): my global_id is 174474 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd258004eb0 con 0x7fd26c0a6830 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 486685893 0 0) 0x7fd258004eb0 con 0x7fd26c0a6830 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd26c137d20 con 0x7fd26c0a6830 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd264001c00 con 0x7fd26c0a6830 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 10 monclient(hunting): dump: 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.183+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1691015028 0 0) 0x7fd26c137d20 con 0x7fd26c0a6830 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6792/0 conn(0x7fd26c0694b0 legacy=0x7fd26c134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6790/0 conn(0x7fd26c0a46a0 legacy=0x7fd26c135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 1 monclient: found mon.a 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26c138b90 con 0x7fd26c0a6830 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 5 monclient: authenticate success, global_id 174474 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 10 monclient: _renew_subs 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd26c137630 con 0x7fd26c0a6830 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd264004a90 con 0x7fd26c0a6830 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd264004fc0 con 0x7fd26c0a6830 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 10 client.174474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd26affd640 10 monclient: dump: 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 10 monclient: _renew_subs 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd26c1d75f0 con 0x7fd26c0a6830 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd2744a6640 1 librados: init done 2024-08-19T10:01:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.184+0000 7fd270a3f640 4 set_mon_vals no callback set 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd2744a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd264005260 con 0x7fd26c0a6830 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd26c051b00 con 0x7fd26c09e570 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(418..418 src has 299..418) ==== 5516+0+0 (unknown 2446891144 0 0) 0x7fd26404e090 con 0x7fd26c0a6830 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd26affd640 10 client.174474.objecter ms_dispatch 0x7fd26c000b90 osd_map(418..418 src has 299..418) 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd26affd640 3 client.174474.objecter handle_osd_map got epochs [418,418] > 0 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd26affd640 3 client.174474.objecter handle_osd_map decoding full epoch 418 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd26affd640 20 client.174474.objecter dump_active .. 0 homeless 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.185+0000 7fd271a41640 10 monclient: get_auth_request con 0x7fd2580415c0 auth_method 0 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.187+0000 7fd272242640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6793/0 conn(0x7fd26c09e570 legacy=0x7fd26c09e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.187+0000 7fd26affd640 10 client.174474.objecter ms_handle_connect 0x7fd26c09e570 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.187+0000 7fd26affd640 10 client.174474.objecter resend_mon_ops 2024-08-19T10:01:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.188+0000 7fd26affd640 10 client.174474.objecter ms_handle_connect 0x7fd2580415c0 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889032795 0 0) 0x7fd26c051b00 con 0x7fd26c09e570 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient(hunting): my global_id is 184211 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd26c137d20 con 0x7fd26c09e570 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2627415110 0 0) 0x7fd26c137d20 con 0x7fd26c09e570 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd26c051b00 con 0x7fd26c09e570 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd25c004080 con 0x7fd26c09e570 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.214+0000 7fd26affd640 10 monclient: dump: 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.218 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409570108}) 2024-08-19T10:01:44.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3444945351 0 0) 0x7fd26c051b00 con 0x7fd26c09e570 2024-08-19T10:01:44.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd26c1386d0 con 0x7fd26c09e570 2024-08-19T10:01:44.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316874766}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711241684}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094254124}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3676746160}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3774941001}) 2024-08-19T10:01:44.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4164059128}) 2024-08-19T10:01:44.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742741857}) 2024-08-19T10:01:44.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.215+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3468446521}) 2024-08-19T10:01:44.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd25c005b40 con 0x7fd26c09e570 2024-08-19T10:01:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6793/0 conn(0x7fd26c09e570 legacy=0x7fd26c09e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.215+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6793/0 conn(0x7fd26c09e570 legacy=0x7fd26c09e950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.227+0000 7fd2744a6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.227+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd26c072fb0 con 0x7fd26c09f060 2024-08-19T10:01:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.229+0000 7fd272242640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6793/0 conn(0x7fd26c09f060 legacy=0x7fd26c1db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.229+0000 7fd26affd640 10 client.174474.objecter ms_handle_connect 0x7fd26c09f060 2024-08-19T10:01:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.229+0000 7fd26affd640 10 client.174474.objecter resend_mon_ops 2024-08-19T10:01:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794388646 0 0) 0x7fd26c072fb0 con 0x7fd26c09f060 2024-08-19T10:01:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient(hunting): my global_id is 184220 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd26c051b00 con 0x7fd26c09f060 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 416052628 0 0) 0x7fd26c051b00 con 0x7fd26c09f060 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd26c072fb0 con 0x7fd26c09f060 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd25c005d70 con 0x7fd26c09f060 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient: dump: 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.234 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3906875184 0 0) 0x7fd26c072fb0 con 0x7fd26c09f060 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd26c09f440 con 0x7fd26c09f060 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 2381024061) 0x7fd25c007260 con 0x7fd26c09f060 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6793/0 conn(0x7fd26c09f060 legacy=0x7fd26c1db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.230+0000 7fd26affd640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6793/0 conn(0x7fd26c09f060 legacy=0x7fd26c1db7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.235 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:44.236 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.236 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.236 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:44.236 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:44.236 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 21478, 2024-08-19T10:01:44.236 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:44.237 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.238 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.239 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.240 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:44.241 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.242 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.243 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.246 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:44.247 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.254 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.256 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:44.257 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 10 librados: watch_flush enter 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 10 librados: watch_flush exit 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 20 client.174474.objecter shutdown clearing up homeless session... 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 10 client.174474.objecter successfully canceled tick 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd2580415c0 legacy=0x7fd258043a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 10 monclient: shutdown 2024-08-19T10:01:44.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 >> v1:172.21.15.1:6789/0 conn(0x7fd26c0a6830 legacy=0x7fd26c135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd272a43640 1 -- 172.21.15.1:0/2553410891 reap_dead start 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd272a43640 1 -- 172.21.15.1:0/2553410891 reap_dead start 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 shutdown_connections 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 >> 172.21.15.1:0/2553410891 conn(0x7fd26c0a9020 msgr2=0x7fd26c0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 shutdown_connections 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 -- 172.21.15.1:0/2553410891 wait complete. 2024-08-19T10:01:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.231+0000 7fd2744a6640 1 librados: shutdown 2024-08-19T10:01:44.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:44.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:01:44.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:01:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.359+0000 7f7812260640 10 monclient: get_monmap_and_config 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.359+0000 7f7812260640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.359+0000 7f7812260640 10 monclient: monmap: 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:44.360508+0000 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:44.360508+0000 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.362 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 1 Processor -- start 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 1 -- start start 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient: init 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient: _add_conns ranks=[8,0,7,3,6,1,5,4,2] 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient(hunting): picked mon.noname-e con 0x7f780c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient(hunting): picked mon.noname-a con 0x7f780c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient(hunting): picked mon.noname-i con 0x7f780c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c0a3b40 con 0x7f780c0a6ba0 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c0a8130 con 0x7f780c13f5c0 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c06ede0 con 0x7f780c0a67c0 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.360+0000 7f7812260640 10 monclient(hunting): authenticate will time out at 1360.436910s 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.361+0000 7f780affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f780c0a6ba0 0x7f780c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47992/0 (socket says 172.21.15.1:47992) 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.361+0000 7f780affd640 1 -- 172.21.15.1:0/951238976 learned_addr learned my addr 172.21.15.1:0/951238976 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.361+0000 7f780b7fe640 1 --1- 172.21.15.1:0/951238976 >> v1:172.21.15.1:6793/0 conn(0x7f780c0a67c0 0x7f780c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42804/0 (socket says 172.21.15.1:42804) 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780affd640 1 -- 172.21.15.1:0/951238976 >> v1:172.21.15.1:6789/0 conn(0x7f780c0a6ba0 legacy=0x7f780c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780b7fe640 1 -- 172.21.15.1:0/951238976 >> v1:172.21.15.1:6793/0 conn(0x7f780c0a67c0 legacy=0x7f780c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821915406 0 0) 0x7f780c0a3b40 con 0x7f780c0a6ba0 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): my global_id is 174483 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77f4003520 con 0x7f780c0a6ba0 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109450686 0 0) 0x7f780c06ede0 con 0x7f780c0a67c0 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient(hunting): my global_id is 184229 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f780c0a3b40 con 0x7f780c0a67c0 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780bfff640 1 -- 172.21.15.1:0/951238976 >> v1:172.21.15.82:6792/0 conn(0x7f780c13f5c0 legacy=0x7f780c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 230408690 0 0) 0x7f780c0a3b40 con 0x7f780c0a67c0 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 >> v1:172.21.15.82:6792/0 conn(0x7f780c13f5c0 legacy=0x7f780c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 >> v1:172.21.15.1:6789/0 conn(0x7f780c0a6ba0 legacy=0x7f780c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 monclient: found mon.noname-e 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c0a92a0 con 0x7f780c0a67c0 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77fc001a10 con 0x7f780c0a67c0 2024-08-19T10:01:44.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f7812260640 5 monclient: authenticate success, global_id 184229 2024-08-19T10:01:44.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f7812260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: dump: 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.375 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f7812260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77fc001ce0 con 0x7f780c0a67c0 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 1 -- 172.21.15.1:0/951238976 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77fc004b20 con 0x7f780c0a67c0 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f780a7fc640 10 monclient: dump: 2024-08-19T10:01:44.376 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f7812260640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.363+0000 7f7812260640 4 set_mon_vals no callback set 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 10 monclient: shutdown 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 1 -- 172.21.15.1:0/951238976 >> v1:172.21.15.1:6793/0 conn(0x7f780c0a67c0 legacy=0x7f780c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 1 -- 172.21.15.1:0/951238976 shutdown_connections 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 1 -- 172.21.15.1:0/951238976 >> 172.21.15.1:0/951238976 conn(0x7f780c0ac4b0 msgr2=0x7f780c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 1 -- 172.21.15.1:0/951238976 shutdown_connections 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 1 -- 172.21.15.1:0/951238976 wait complete. 2024-08-19T10:01:44.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.364+0000 7f7812260640 10 monclient: monmap: 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 librados: starting msgr at 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 librados: starting objecter 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 Processor -- start 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 -- start start 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 librados: setting wanted keys 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 librados: calling monclient init 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient: init 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient: _add_conns ranks=[4,0,3,7,2,8,5,6,1] 2024-08-19T10:01:44.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient(hunting): picked mon.noname-c con 0x7f780c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient(hunting): picked mon.noname-a con 0x7f780c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient(hunting): picked mon.noname-g con 0x7f780c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c0a9dc0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c1cf020 con 0x7f780c13f5c0 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c142c80 con 0x7f780c0a67c0 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f7812260640 10 monclient(hunting): authenticate will time out at 1360.441908s 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f780affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f780c0a6ba0 0x7f780c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48004/0 (socket says 172.21.15.1:48004) 2024-08-19T10:01:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.365+0000 7f780affd640 1 -- 172.21.15.1:0/2554105730 learned_addr learned my addr 172.21.15.1:0/2554105730 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f780b7fe640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.1:6791/0 conn(0x7f780c0a67c0 legacy=0x7f780c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 client.?.objecter ms_handle_connect 0x7f780c0a67c0 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f780affd640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.1:6789/0 conn(0x7f780c0a6ba0 legacy=0x7f780c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 client.?.objecter ms_handle_connect 0x7f780c0a6ba0 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108065848 0 0) 0x7f780c0a9dc0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): my global_id is 174492 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77f8004eb0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922245674 0 0) 0x7f780c142c80 con 0x7f780c0a67c0 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): my global_id is 27796 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f780c0a9dc0 con 0x7f780c0a67c0 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f780bfff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6790/0 conn(0x7f780c13f5c0 legacy=0x7f780c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 client.?.objecter ms_handle_connect 0x7f780c13f5c0 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3711589330 0 0) 0x7f77f8004eb0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f780c142c80 con 0x7f780c0a6ba0 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f78000047e0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): dump: 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.390 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3536200285 0 0) 0x7f780c0a9dc0 con 0x7f780c0a67c0 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77f8004eb0 con 0x7f780c0a67c0 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77fc004b20 con 0x7f780c0a67c0 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): dump: 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298327469 0 0) 0x7f780c1cf020 con 0x7f780c13f5c0 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient(hunting): my global_id is 184188 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f780c0a9dc0 con 0x7f780c13f5c0 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1138000711 0 0) 0x7f780c142c80 con 0x7f780c0a6ba0 2024-08-19T10:01:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.1:6791/0 conn(0x7f780c0a67c0 legacy=0x7f780c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6790/0 conn(0x7f780c13f5c0 legacy=0x7f780c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 monclient: found mon.a 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c139f10 con 0x7f780c0a6ba0 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 5 monclient: authenticate success, global_id 174492 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 10 monclient: _renew_subs 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f780c0a1640 con 0x7f780c0a6ba0 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 10 client.174492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 10 monclient: _renew_subs 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f780c13f310 con 0x7f780c0a6ba0 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7812260640 1 librados: init done 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7800004ce0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7800005250 con 0x7f780c0a6ba0 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f7809ffb640 4 set_mon_vals no callback set 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:44.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.368+0000 7f77f3fff640 10 monclient: dump: 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.369+0000 7f7812260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.369+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f77ec0065c0 con 0x7f77ec002860 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.369+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f78000053f0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.370+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(418..418 src has 299..418) ==== 5516+0+0 (unknown 2446891144 0 0) 0x7f780004e2f0 con 0x7f780c0a6ba0 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.370+0000 7f77f3fff640 10 client.174492.objecter ms_dispatch 0x7f780c000b90 osd_map(418..418 src has 299..418) 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.370+0000 7f77f3fff640 3 client.174492.objecter handle_osd_map got epochs [418,418] > 0 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.370+0000 7f77f3fff640 3 client.174492.objecter handle_osd_map decoding full epoch 418 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.370+0000 7f77f3fff640 20 client.174492.objecter dump_active .. 0 homeless 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.370+0000 7f780bfff640 10 monclient: get_auth_request con 0x7f77f80416e0 auth_method 0 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.371+0000 7f780b7fe640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6789/0 conn(0x7f77ec002860 legacy=0x7f77ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 client.174492.objecter ms_handle_connect 0x7f77ec002860 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 client.174492.objecter resend_mon_ops 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1359595010 0 0) 0x7f77ec0065c0 con 0x7f77ec002860 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient(hunting): my global_id is 174448 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f780c142c80 con 0x7f77ec002860 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1828010239 0 0) 0x7f780c142c80 con 0x7f77ec002860 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77ec0065c0 con 0x7f77ec002860 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77fc001c60 con 0x7f77ec002860 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:01:44.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient: dump: 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 client.174492.objecter ms_handle_connect 0x7f77f80416e0 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3763422071 0 0) 0x7f77ec0065c0 con 0x7f77ec002860 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77ec00a160 con 0x7f77ec002860 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f77fc0051e0 con 0x7f77ec002860 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6789/0 conn(0x7f77ec002860 legacy=0x7f77ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.372+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6789/0 conn(0x7f77ec002860 legacy=0x7f77ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.378+0000 7f7812260640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.379+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f77ec009e50 con 0x7f77ec009110 2024-08-19T10:01:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f780b7fe640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6789/0 conn(0x7f77ec009110 legacy=0x7f77ec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 client.174492.objecter ms_handle_connect 0x7f77ec009110 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 client.174492.objecter resend_mon_ops 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963040354 0 0) 0x7f77ec009e50 con 0x7f77ec009110 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient(hunting): my global_id is 174457 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77ec0065c0 con 0x7f77ec009110 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 678226223 0 0) 0x7f77ec0065c0 con 0x7f77ec009110 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77ec009e50 con 0x7f77ec009110 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77fc005ae0 con 0x7f77ec009110 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.381+0000 7f77f3fff640 10 monclient: dump: 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2055003740 0 0) 0x7f77ec009e50 con 0x7f77ec009110 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77ec00c3f0 con 0x7f77ec009110 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7852 (unknown 0 0 1797722403) 0x7f77fc007da0 con 0x7f77ec009110 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6789/0 conn(0x7f77ec009110 legacy=0x7f77ec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f77f3fff640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6789/0 conn(0x7f77ec009110 legacy=0x7f77ec0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 154440, 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:44.425 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:44.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 10 librados: watch_flush enter 2024-08-19T10:01:44.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 10 librados: watch_flush exit 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 20 client.174492.objecter shutdown clearing up homeless session... 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 10 client.174492.objecter successfully canceled tick 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.82:6812/2549104415 conn(0x7f77f80416e0 legacy=0x7f77f8043ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 10 monclient: shutdown 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 >> v1:172.21.15.1:6789/0 conn(0x7f780c0a6ba0 legacy=0x7f780c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f780bfff640 1 -- 172.21.15.1:0/2554105730 reap_dead start 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f780bfff640 1 -- 172.21.15.1:0/2554105730 reap_dead start 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 shutdown_connections 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 >> 172.21.15.1:0/2554105730 conn(0x7f780c0ac4b0 msgr2=0x7f780c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 shutdown_connections 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.382+0000 7f7812260640 1 -- 172.21.15.1:0/2554105730 wait complete. 2024-08-19T10:01:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.383+0000 7f7812260640 1 librados: shutdown 2024-08-19T10:01:44.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:44.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:01:44.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:01:44.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.528+0000 7fe418a9f640 10 monclient: get_monmap_and_config 2024-08-19T10:01:44.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.528+0000 7fe418a9f640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.528+0000 7fe418a9f640 10 monclient: monmap: 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:44.529466+0000 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:44.529466+0000 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.530 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.531 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.531 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.531 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 1 Processor -- start 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 1 -- start start 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient: init 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient: _add_conns ranks=[3,6,5,7,8,0,1,4,2] 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient(hunting): picked mon.noname-g con 0x7fe4140a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:44.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient(hunting): picked mon.noname-d con 0x7fe4140a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient(hunting): picked mon.noname-h con 0x7fe41413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4140a3b40 con 0x7fe4140a67c0 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4140a8130 con 0x7fe41413f5c0 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe41406ede0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe418a9f640 10 monclient(hunting): authenticate will time out at 1360.605804s 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe411d9b640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe4140a6ba0 0x7fe4140a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43120/0 (socket says 172.21.15.1:43120) 2024-08-19T10:01:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.529+0000 7fe411d9b640 1 -- 172.21.15.1:0/3898024903 learned_addr learned my addr 172.21.15.1:0/3898024903 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe411d9b640 1 -- 172.21.15.1:0/3898024903 >> v1:172.21.15.1:6792/0 conn(0x7fe4140a6ba0 legacy=0x7fe4140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41259c640 1 -- 172.21.15.1:0/3898024903 >> v1:172.21.15.82:6790/0 conn(0x7fe4140a67c0 legacy=0x7fe4140a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230828670 0 0) 0x7fe41406ede0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): my global_id is 174507 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe412d9d640 1 -- 172.21.15.1:0/3898024903 >> v1:172.21.15.82:6791/0 conn(0x7fe41413f5c0 legacy=0x7fe4141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe400003520 con 0x7fe4140a6ba0 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267722945 0 0) 0x7fe4140a3b40 con 0x7fe4140a67c0 2024-08-19T10:01:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): my global_id is 184197 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe41406ede0 con 0x7fe4140a67c0 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519287454 0 0) 0x7fe4140a8130 con 0x7fe41413f5c0 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient(hunting): my global_id is 27437 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4140a3b40 con 0x7fe41413f5c0 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 218135647 0 0) 0x7fe400003520 con 0x7fe4140a6ba0 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 >> v1:172.21.15.82:6791/0 conn(0x7fe41413f5c0 legacy=0x7fe4141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 >> v1:172.21.15.82:6790/0 conn(0x7fe4140a67c0 legacy=0x7fe4140a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 monclient: found mon.noname-d 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4140a92a0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe418a9f640 5 monclient: authenticate success, global_id 174507 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe418a9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe3fc001ad0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: dump: 2024-08-19T10:01:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe418a9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe3fc0048a0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe418a9f640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 1 -- 172.21.15.1:0/3898024903 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe3fc004e10 con 0x7fe4140a6ba0 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe418a9f640 4 set_mon_vals no callback set 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.532+0000 7fe41159a640 10 monclient: dump: 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 10 monclient: shutdown 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 1 -- 172.21.15.1:0/3898024903 >> v1:172.21.15.1:6792/0 conn(0x7fe4140a6ba0 legacy=0x7fe4140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 1 -- 172.21.15.1:0/3898024903 shutdown_connections 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 1 -- 172.21.15.1:0/3898024903 >> 172.21.15.1:0/3898024903 conn(0x7fe4140ac4b0 msgr2=0x7fe4140a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 1 -- 172.21.15.1:0/3898024903 shutdown_connections 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 1 -- 172.21.15.1:0/3898024903 wait complete. 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.533+0000 7fe418a9f640 10 monclient: monmap: 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 librados: starting msgr at 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 librados: starting objecter 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 Processor -- start 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 -- start start 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 librados: setting wanted keys 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 librados: calling monclient init 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient: init 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient: _add_conns ranks=[4,1,8,0,5,2,6,7,3] 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient(hunting): picked mon.noname-c con 0x7fe4140a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient(hunting): picked mon.noname-f con 0x7fe4140a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient(hunting): picked mon.noname-e con 0x7fe41413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe414139a70 con 0x7fe4140a6ba0 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4140a9cc0 con 0x7fe4140a67c0 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe41413bdb0 con 0x7fe41413f5c0 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe418a9f640 10 monclient(hunting): authenticate will time out at 1360.610815s 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe41259c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fe4140a67c0 0x7fe4141ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44774/0 (socket says 172.21.15.1:44774) 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.534+0000 7fe41259c640 1 -- 172.21.15.1:0/1001459440 learned_addr learned my addr 172.21.15.1:0/1001459440 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe412d9d640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.1:6793/0 conn(0x7fe41413f5c0 legacy=0x7fe4141d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe41259c640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.1:6791/0 conn(0x7fe4140a67c0 legacy=0x7fe4141ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 client.?.objecter ms_handle_connect 0x7fe41413f5c0 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 client.?.objecter ms_handle_connect 0x7fe4140a67c0 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe411d9b640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6789/0 conn(0x7fe4140a6ba0 legacy=0x7fe4141ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510888664 0 0) 0x7fe41406ede0 con 0x7fe4140a67c0 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): my global_id is 27805 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f4004eb0 con 0x7fe4140a67c0 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 client.?.objecter ms_handle_connect 0x7fe4140a6ba0 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308354839 0 0) 0x7fe41413bdb0 con 0x7fe41413f5c0 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): my global_id is 184238 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe41406ede0 con 0x7fe41413f5c0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577177823 0 0) 0x7fe414139a70 con 0x7fe4140a6ba0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): my global_id is 174466 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe41413bdb0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 805010282 0 0) 0x7fe3f4004eb0 con 0x7fe4140a67c0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe414139a70 con 0x7fe4140a67c0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe40c003990 con 0x7fe4140a67c0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:01:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): dump: 2024-08-19T10:01:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.557 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1587127135 0 0) 0x7fe41406ede0 con 0x7fe41413f5c0 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3f4004eb0 con 0x7fe41413f5c0 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe404001cb0 con 0x7fe41413f5c0 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient(hunting): dump: 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2922880028 0 0) 0x7fe41413bdb0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe41406ede0 con 0x7fe4140a6ba0 2024-08-19T10:01:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2013118973 0 0) 0x7fe414139a70 con 0x7fe4140a67c0 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.1:6793/0 conn(0x7fe41413f5c0 legacy=0x7fe4141d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6789/0 conn(0x7fe4140a6ba0 legacy=0x7fe4141ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 monclient: found mon.c 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4141cf020 con 0x7fe4140a67c0 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 5 monclient: authenticate success, global_id 27805 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 10 monclient: _renew_subs 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:01:44.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe41413af40 con 0x7fe4140a67c0 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 10 client.27805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 10 monclient: _renew_subs 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fe41413a880 con 0x7fe4140a67c0 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe418a9f640 1 librados: init done 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe40c003f10 con 0x7fe4140a67c0 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe40c004400 con 0x7fe4140a67c0 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe410d99640 4 set_mon_vals no callback set 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.537+0000 7fe40affd640 10 monclient: dump: 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.538+0000 7fe418a9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.538+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe3f80065c0 con 0x7fe3f8002860 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.538+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe40c0045a0 con 0x7fe4140a67c0 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.539+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(418..418 src has 299..418) ==== 5516+0+0 (unknown 2446891144 0 0) 0x7fe40c04e640 con 0x7fe4140a67c0 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.539+0000 7fe40affd640 10 client.27805.objecter ms_dispatch 0x7fe414000b90 osd_map(418..418 src has 299..418) 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.539+0000 7fe40affd640 3 client.27805.objecter handle_osd_map got epochs [418,418] > 0 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.539+0000 7fe40affd640 3 client.27805.objecter handle_osd_map decoding full epoch 418 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.539+0000 7fe40affd640 20 client.27805.objecter dump_active .. 0 homeless 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.539+0000 7fe412d9d640 10 monclient: get_auth_request con 0x7fe3f40417c0 auth_method 0 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 client.27805.objecter ms_handle_connect 0x7fe3f8002860 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 client.27805.objecter resend_mon_ops 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe411d9b640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6790/0 conn(0x7fe3f8002860 legacy=0x7fe3f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617626998 0 0) 0x7fe3f80065c0 con 0x7fe3f8002860 2024-08-19T10:01:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient(hunting): my global_id is 184206 2024-08-19T10:01:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe414139a70 con 0x7fe3f8002860 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 708917025 0 0) 0x7fe414139a70 con 0x7fe3f8002860 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3f80065c0 con 0x7fe3f8002860 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe3fc001320 con 0x7fe3f8002860 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient: dump: 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 client.27805.objecter ms_handle_connect 0x7fe3f40417c0 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2710887464 0 0) 0x7fe3f80065c0 con 0x7fe3f8002860 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3f800a160 con 0x7fe3f8002860 2024-08-19T10:01:44.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe3fc001d20 con 0x7fe3f8002860 2024-08-19T10:01:44.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:44.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.541+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6790/0 conn(0x7fe3f8002860 legacy=0x7fe3f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.542+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6790/0 conn(0x7fe3f8002860 legacy=0x7fe3f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.548+0000 7fe418a9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:44.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.548+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe3f8009e70 con 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe411d9b640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6790/0 conn(0x7fe3f8009130 legacy=0x7fe3f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 10 client.27805.objecter ms_handle_connect 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 10 client.27805.objecter resend_mon_ops 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449676971 0 0) 0x7fe3f8009e70 con 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 10 monclient(hunting): my global_id is 184215 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.550+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f80065c0 con 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2102178175 0 0) 0x7fe3f80065c0 con 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3f8009e70 con 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe3fc005bb0 con 0x7fe3f8009130 2024-08-19T10:01:44.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 10 monclient: dump: 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3806621277 0 0) 0x7fe3f8009e70 con 0x7fe3f8009130 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3f800c410 con 0x7fe3f8009130 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3756919323) 0x7fe3fc007080 con 0x7fe3f8009130 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6790/0 conn(0x7fe3f8009130 legacy=0x7fe3f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe40affd640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6790/0 conn(0x7fe3f8009130 legacy=0x7fe3f80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 21805, 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:44.572 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.573 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:44.574 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.575 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.575 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.575 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.578 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.578 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.578 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:44.579 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.580 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.581 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.584 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.591 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.595 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:44.597 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 10 librados: watch_flush enter 2024-08-19T10:01:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 10 librados: watch_flush exit 2024-08-19T10:01:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 20 client.27805.objecter shutdown clearing up homeless session... 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 10 client.27805.objecter successfully canceled tick 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe3f40417c0 legacy=0x7fe3f4043c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 10 monclient: shutdown 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 >> v1:172.21.15.1:6791/0 conn(0x7fe4140a67c0 legacy=0x7fe4141ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.551+0000 7fe412d9d640 1 -- 172.21.15.1:0/1001459440 reap_dead start 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.552+0000 7fe412d9d640 1 -- 172.21.15.1:0/1001459440 reap_dead start 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.552+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 shutdown_connections 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.552+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 >> 172.21.15.1:0/1001459440 conn(0x7fe4140ac4b0 msgr2=0x7fe4141421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.552+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 shutdown_connections 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.552+0000 7fe418a9f640 1 -- 172.21.15.1:0/1001459440 wait complete. 2024-08-19T10:01:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.552+0000 7fe418a9f640 1 librados: shutdown 2024-08-19T10:01:44.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:44.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:01:44.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:01:44.603 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.586+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.603 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1831082816}) 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3331646475}) 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3611301167}) 2024-08-19T10:01:44.604 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1302821793}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1857592477}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3598425863}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1379454050}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007987093}) 2024-08-19T10:01:44.605 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.587+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1808661228}) 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058200951}) 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564319587}) 2024-08-19T10:01:44.607 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=850963274}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2469714246}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3332476626}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=312494327}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.608 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1574408886}) 2024-08-19T10:01:44.609 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.609 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115066020}) 2024-08-19T10:01:44.609 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.603+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4247904274}) 2024-08-19T10:01:44.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1807598793}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408612086}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1605618538}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2957246377}) 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=793037484}) 2024-08-19T10:01:44.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3017034039}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812357201}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581049952}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.676+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1529573987}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=112008631}) 2024-08-19T10:01:44.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273489530}) 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=208201853}) 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3385760078}) 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2703048274}) 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=788276559}) 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=411519358}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127645873}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.678+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=324378192}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646521341}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962440623}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=895299372}) 2024-08-19T10:01:44.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3232742578}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3010328931}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1574325905}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=20714685}) 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170797787}) 2024-08-19T10:01:44.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.680+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1970840249}) 2024-08-19T10:01:44.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899298037}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3468553325}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101143126}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=394597413}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2341619206}) 2024-08-19T10:01:44.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3556462089}) 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.688+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.689+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3775161890}) 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.689+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.689+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578333212}) 2024-08-19T10:01:44.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.689+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1383037433}) 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.711+0000 7f97cbccc640 10 monclient: get_monmap_and_config 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.711+0000 7f97cbccc640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.711+0000 7f97cbccc640 10 monclient: monmap: 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:44.712098+0000 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:44.712098+0000 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.713 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.711+0000 7f97cbccc640 1 Processor -- start 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.711+0000 7f97cbccc640 1 -- start start 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.711+0000 7f97cbccc640 10 monclient: init 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient: _add_conns ranks=[5,1,3,8,2,7,6,0,4] 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient(hunting): picked mon.noname-h con 0x7f97c40a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient(hunting): picked mon.noname-f con 0x7f97c40694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient(hunting): picked mon.noname-g con 0x7f97c40a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c40a3c70 con 0x7f97c40694b0 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c40a33d0 con 0x7f97c40a46a0 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c40a36b0 con 0x7f97c40a6830 2024-08-19T10:01:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97cbccc640 10 monclient(hunting): authenticate will time out at 1360.788517s 2024-08-19T10:01:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97c9267640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f97c40694b0 0x7f97c40a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:55734/0 (socket says 172.21.15.1:55734) 2024-08-19T10:01:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97ca269640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f97c40a46a0 0x7f97c4141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:38240/0 (socket says 172.21.15.1:38240) 2024-08-19T10:01:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.712+0000 7f97c9267640 1 -- 172.21.15.1:0/2834629001 learned_addr learned my addr 172.21.15.1:0/2834629001 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.714+0000 7f97ca269640 1 -- 172.21.15.1:0/2834629001 >> v1:172.21.15.82:6790/0 conn(0x7f97c40a46a0 legacy=0x7f97c4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.714+0000 7f97c9a68640 1 -- 172.21.15.1:0/2834629001 >> v1:172.21.15.82:6791/0 conn(0x7f97c40a6830 legacy=0x7f97c40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.714+0000 7f97c9267640 1 -- 172.21.15.1:0/2834629001 >> v1:172.21.15.82:6789/0 conn(0x7f97c40694b0 legacy=0x7f97c40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485973128 0 0) 0x7f97c40a33d0 con 0x7f97c40a46a0 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): my global_id is 184224 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b0003520 con 0x7f97c40a46a0 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3041647200 0 0) 0x7f97c40a3c70 con 0x7f97c40694b0 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): my global_id is 174475 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c40a33d0 con 0x7f97c40694b0 2024-08-19T10:01:44.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728291639 0 0) 0x7f97c40a36b0 con 0x7f97c40a6830 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient(hunting): my global_id is 27446 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c40a3c70 con 0x7f97c40a6830 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 218132389 0 0) 0x7f97b0003520 con 0x7f97c40a46a0 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 >> v1:172.21.15.82:6791/0 conn(0x7f97c40a6830 legacy=0x7f97c40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 >> v1:172.21.15.82:6789/0 conn(0x7f97c40694b0 legacy=0x7f97c40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 monclient: found mon.noname-g 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c406dbb0 con 0x7f97c40a46a0 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97cbccc640 5 monclient: authenticate success, global_id 184224 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97cbccc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97bc001e60 con 0x7f97c40a46a0 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: dump: 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97cbccc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97bc004b60 con 0x7f97c40a46a0 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97cbccc640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97cbccc640 4 set_mon_vals no callback set 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 1 -- 172.21.15.1:0/2834629001 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97bc005050 con 0x7f97c40a46a0 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.715+0000 7f97c8a66640 10 monclient: dump: 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.725 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 10 monclient: shutdown 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 1 -- 172.21.15.1:0/2834629001 >> v1:172.21.15.82:6790/0 conn(0x7f97c40a46a0 legacy=0x7f97c4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 1 -- 172.21.15.1:0/2834629001 shutdown_connections 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 1 -- 172.21.15.1:0/2834629001 >> 172.21.15.1:0/2834629001 conn(0x7f97c40a9020 msgr2=0x7f97c4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 1 -- 172.21.15.1:0/2834629001 shutdown_connections 2024-08-19T10:01:44.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 1 -- 172.21.15.1:0/2834629001 wait complete. 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.716+0000 7f97cbccc640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient: monmap: 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.728 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 librados: starting msgr at 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 librados: starting objecter 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 Processor -- start 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 -- start start 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 librados: setting wanted keys 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 librados: calling monclient init 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient: init 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient: _add_conns ranks=[3,8,5,0,4,1,7,6,2] 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient(hunting): picked mon.noname-g con 0x7f97c40694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient(hunting): picked mon.noname-e con 0x7f97c40a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient(hunting): picked mon.noname-h con 0x7f97c40a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c4138910 con 0x7f97c40694b0 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c4138bf0 con 0x7f97c40a6830 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c4137d80 con 0x7f97c40a46a0 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.717+0000 7f97cbccc640 10 monclient(hunting): authenticate will time out at 1360.793992s 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.718+0000 7f97c9267640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f97c40a46a0 0x7f97c4135720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42812/0 (socket says 172.21.15.1:42812) 2024-08-19T10:01:44.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.718+0000 7f97c9267640 1 -- 172.21.15.1:0/476347100 learned_addr learned my addr 172.21.15.1:0/476347100 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97c9267640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.1:6793/0 conn(0x7f97c40a46a0 legacy=0x7f97c4135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 client.?.objecter ms_handle_connect 0x7f97c40a46a0 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97c9a68640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6790/0 conn(0x7f97c40694b0 legacy=0x7f97c4135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 client.?.objecter ms_handle_connect 0x7f97c40694b0 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97ca269640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97c40a6830 legacy=0x7f97c41d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 client.?.objecter ms_handle_connect 0x7f97c40a6830 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114543726 0 0) 0x7f97c4137d80 con 0x7f97c40a46a0 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): my global_id is 184247 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97a8004eb0 con 0x7f97c40a46a0 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2882756090 0 0) 0x7f97c4138910 con 0x7f97c40694b0 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): my global_id is 184233 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c4137d80 con 0x7f97c40694b0 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 622218685 0 0) 0x7f97c4138bf0 con 0x7f97c40a6830 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): my global_id is 27455 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c4138910 con 0x7f97c40a6830 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2341281609 0 0) 0x7f97a8004eb0 con 0x7f97c40a46a0 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97c4138bf0 con 0x7f97c40a46a0 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97b80047e0 con 0x7f97c40a46a0 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): dump: 2024-08-19T10:01:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 329110899 0 0) 0x7f97c4138910 con 0x7f97c40a6830 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97a8004eb0 con 0x7f97c40a6830 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97bc001de0 con 0x7f97c40a6830 2024-08-19T10:01:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): dump: 2024-08-19T10:01:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3524592827 0 0) 0x7f97c4137d80 con 0x7f97c40694b0 2024-08-19T10:01:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97c4138910 con 0x7f97c40694b0 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97c0001cf0 con 0x7f97c40694b0 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient(hunting): dump: 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 943413473 0 0) 0x7f97c4138bf0 con 0x7f97c40a46a0 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97c40a6830 legacy=0x7f97c41d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6790/0 conn(0x7f97c40694b0 legacy=0x7f97c4135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 monclient: found mon.e 2024-08-19T10:01:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c4135e30 con 0x7f97c40a46a0 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.720+0000 7f97b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 5 monclient: authenticate success, global_id 184247 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 10 monclient: _renew_subs 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97c4138350 con 0x7f97c40a46a0 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 10 client.184247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 10 monclient: _renew_subs 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:01:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f97c41373f0 con 0x7f97c40a46a0 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 1 librados: init done 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97b8004ce0 con 0x7f97c40a46a0 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97b8005210 con 0x7f97c40a46a0 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b7fff640 4 set_mon_vals no callback set 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97b67fc640 10 monclient: dump: 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.747 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.721+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f97ac0065c0 con 0x7f97ac002860 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f97b80054b0 con 0x7f97c40a46a0 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(419..419 src has 299..419) ==== 5516+0+0 (unknown 2589855585 0 0) 0x7f97b804e340 con 0x7f97c40a46a0 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97b67fc640 10 client.184247.objecter ms_dispatch 0x7f97c4000b90 osd_map(419..419 src has 299..419) 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97b67fc640 3 client.184247.objecter handle_osd_map got epochs [419,419] > 0 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97b67fc640 3 client.184247.objecter handle_osd_map decoding full epoch 419 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97b67fc640 20 client.184247.objecter dump_active .. 0 homeless 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.722+0000 7f97ca269640 10 monclient: get_auth_request con 0x7f97a80418b0 auth_method 0 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 client.184247.objecter ms_handle_connect 0x7f97ac002860 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 client.184247.objecter resend_mon_ops 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97c9a68640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97ac002860 legacy=0x7f97ac004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085494452 0 0) 0x7f97ac0065c0 con 0x7f97ac002860 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient(hunting): my global_id is 27464 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c4138bf0 con 0x7f97ac002860 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3093023711 0 0) 0x7f97c4138bf0 con 0x7f97ac002860 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97ac0065c0 con 0x7f97ac002860 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97c0003c50 con 0x7f97ac002860 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 monclient: dump: 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 10 client.184247.objecter ms_handle_connect 0x7f97a80418b0 2024-08-19T10:01:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2507664516 0 0) 0x7f97ac0065c0 con 0x7f97ac002860 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.724+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97ac00a160 con 0x7f97ac002860 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.725+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f97c0003e80 con 0x7f97ac002860 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.725+0000 7f97b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.725+0000 7f97b67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.725+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97ac002860 legacy=0x7f97ac004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.725+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97ac002860 legacy=0x7f97ac004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.731+0000 7f97cbccc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.731+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f97ac009e70 con 0x7f97ac009130 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.733+0000 7f97b67fc640 10 client.184247.objecter ms_handle_connect 0x7f97ac009130 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.733+0000 7f97b67fc640 10 client.184247.objecter resend_mon_ops 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.733+0000 7f97c9a68640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97ac009130 legacy=0x7f97ac0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 864101846 0 0) 0x7f97ac009e70 con 0x7f97ac009130 2024-08-19T10:01:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient(hunting): my global_id is 27473 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97ac0065c0 con 0x7f97ac009130 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1436886025 0 0) 0x7f97ac0065c0 con 0x7f97ac009130 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97ac009e70 con 0x7f97ac009130 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f97c0004430 con 0x7f97ac009130 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient: dump: 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3917888476 0 0) 0x7f97ac009e70 con 0x7f97ac009130 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97ac00c3f0 con 0x7f97ac009130 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3277898999) 0x7f97c0005cf0 con 0x7f97ac009130 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97ac009130 legacy=0x7f97ac0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.734+0000 7f97b67fc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6791/0 conn(0x7f97ac009130 legacy=0x7f97ac0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:44.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 711000, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:44.756 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.757 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.758 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:44.759 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.761 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.762 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:44.767 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.768 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:44.775 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:44.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 10 librados: watch_flush enter 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 10 librados: watch_flush exit 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 20 client.184247.objecter shutdown clearing up homeless session... 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 10 client.184247.objecter successfully canceled tick 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.82:6812/2549104415 conn(0x7f97a80418b0 legacy=0x7f97a8043d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 10 monclient: shutdown 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 >> v1:172.21.15.1:6793/0 conn(0x7f97c40a46a0 legacy=0x7f97c4135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97ca269640 1 -- 172.21.15.1:0/476347100 reap_dead start 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97ca269640 1 -- 172.21.15.1:0/476347100 reap_dead start 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 shutdown_connections 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 >> 172.21.15.1:0/476347100 conn(0x7f97c40a9020 msgr2=0x7f97c40a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 shutdown_connections 2024-08-19T10:01:44.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 -- 172.21.15.1:0/476347100 wait complete. 2024-08-19T10:01:44.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.735+0000 7f97cbccc640 1 librados: shutdown 2024-08-19T10:01:44.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:44.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:01:44.779 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:01:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.877+0000 7f7f60993640 10 monclient: get_monmap_and_config 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.877+0000 7f7f60993640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.877+0000 7f7f60993640 10 monclient: monmap: 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:44.878759+0000 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:44.878759+0000 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.880 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.880 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.880 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.880 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.880 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.880 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.878+0000 7f7f60993640 1 Processor -- start 2024-08-19T10:01:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.878+0000 7f7f60993640 1 -- start start 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.878+0000 7f7f60993640 10 monclient: init 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.878+0000 7f7f60993640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 10 monclient: _add_conns ranks=[4,0,8,2,6,1,5,3,7] 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 10 monclient(hunting): picked mon.noname-c con 0x7f7f5c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 10 monclient(hunting): picked mon.noname-a con 0x7f7f5c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 10 monclient(hunting): picked mon.noname-e con 0x7f7f5c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c0a3b40 con 0x7f7f5c0a6ba0 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c0a8130 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c06ede0 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f60993640 10 monclient(hunting): authenticate will time out at 1360.955106s 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f5a59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f7f5c0a67c0 0x7f7f5c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:44780/0 (socket says 172.21.15.1:44780) 2024-08-19T10:01:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f59d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7f5c0a6ba0 0x7f7f5c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48016/0 (socket says 172.21.15.1:48016) 2024-08-19T10:01:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f5a59c640 1 -- 172.21.15.1:0/319803995 learned_addr learned my addr 172.21.15.1:0/319803995 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.879+0000 7f7f5ad9d640 1 --1- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6793/0 conn(0x7f7f5c13f5c0 0x7f7f5c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42826/0 (socket says 172.21.15.1:42826) 2024-08-19T10:01:44.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5a59c640 1 -- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6791/0 conn(0x7f7f5c0a67c0 legacy=0x7f7f5c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5ad9d640 1 -- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6793/0 conn(0x7f7f5c13f5c0 legacy=0x7f7f5c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f59d9b640 1 -- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6789/0 conn(0x7f7f5c0a6ba0 legacy=0x7f7f5c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920325597 0 0) 0x7f7f5c0a8130 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): my global_id is 27814 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f48003520 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915368856 0 0) 0x7f7f5c06ede0 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): my global_id is 184256 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f5c0a8130 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3905249055 0 0) 0x7f7f5c0a3b40 con 0x7f7f5c0a6ba0 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient(hunting): my global_id is 174501 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f5c06ede0 con 0x7f7f5c0a6ba0 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4259602420 0 0) 0x7f7f5c0a8130 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6791/0 conn(0x7f7f5c0a67c0 legacy=0x7f7f5c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6789/0 conn(0x7f7f5c0a6ba0 legacy=0x7f7f5c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 monclient: found mon.noname-e 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c0a92a0 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f60993640 5 monclient: authenticate success, global_id 184256 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7f4c001a50 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f60993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f5959a640 10 monclient: dump: 2024-08-19T10:01:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.881+0000 7f7f60993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f4c001d20 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f5959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f5959a640 1 -- 172.21.15.1:0/319803995 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7f4c004b20 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 4 set_mon_vals no callback set 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f5959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f5959a640 10 monclient: dump: 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 10 monclient: shutdown 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 1 -- 172.21.15.1:0/319803995 >> v1:172.21.15.1:6793/0 conn(0x7f7f5c13f5c0 legacy=0x7f7f5c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 1 -- 172.21.15.1:0/319803995 shutdown_connections 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 1 -- 172.21.15.1:0/319803995 >> 172.21.15.1:0/319803995 conn(0x7f7f5c0ac4b0 msgr2=0x7f7f5c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 1 -- 172.21.15.1:0/319803995 shutdown_connections 2024-08-19T10:01:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.882+0000 7f7f60993640 1 -- 172.21.15.1:0/319803995 wait complete. 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 10 monclient: build_initial_monmap 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 10 monclient: monmap: 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 1 librados: starting msgr at 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 1 librados: starting objecter 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 1 Processor -- start 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 1 -- start start 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 1 librados: setting wanted keys 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 1 librados: calling monclient init 2024-08-19T10:01:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.883+0000 7f7f60993640 10 monclient: init 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient: _add_conns ranks=[6,3,7,2,0,5,8,1,4] 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient(hunting): picked mon.noname-d con 0x7f7f5c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient(hunting): picked mon.noname-g con 0x7f7f5c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient(hunting): picked mon.noname-i con 0x7f7f5c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c13a800 con 0x7f7f5c0a6ba0 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c0a2860 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c0a2200 con 0x7f7f5c13f5c0 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f60993640 10 monclient(hunting): authenticate will time out at 1360.960207s 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f5a59c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f7f5c0a67c0 0x7f7f5c09fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43130/0 (socket says 172.21.15.1:43130) 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.884+0000 7f7f5a59c640 1 -- 172.21.15.1:0/1939622668 learned_addr learned my addr 172.21.15.1:0/1939622668 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f5a59c640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.1:6792/0 conn(0x7f7f5c0a67c0 legacy=0x7f7f5c09fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 client.?.objecter ms_handle_connect 0x7f7f5c0a67c0 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061830057 0 0) 0x7f7f5c0a2860 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): my global_id is 174516 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f3c004eb0 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 client.?.objecter ms_handle_connect 0x7f7f5c13f5c0 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f5ad9d640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f5c13f5c0 legacy=0x7f7f5c0a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 client.?.objecter ms_handle_connect 0x7f7f5c0a6ba0 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f59d9b640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6790/0 conn(0x7f7f5c0a6ba0 legacy=0x7f7f5c0a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3480280785 0 0) 0x7f7f3c004eb0 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f5c0a2860 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7f54001e10 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): dump: 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.901 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2366740650 0 0) 0x7f7f5c0a2860 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f5c13f5c0 legacy=0x7f7f5c0a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6790/0 conn(0x7f7f5c0a6ba0 legacy=0x7f7f5c0a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 monclient: found mon.d 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c139b50 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient: _finish_auth 0 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.886+0000 7f7f52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 5 monclient: authenticate success, global_id 174516 2024-08-19T10:01:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 10 monclient: _renew_subs 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5c0a1600 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 10 client.174516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 10 monclient: _renew_subs 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f5c139f60 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 1 librados: init done 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f54005150 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f52ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7f54005640 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f52ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:01:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f58d99640 4 set_mon_vals no callback set 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f52ffd640 10 monclient: dump: 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.905 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.887+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7f400065c0 con 0x7f7f40002860 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7f540058e0 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(419..419 src has 299..419) ==== 5516+0+0 (unknown 2589855585 0 0) 0x7f7f5404e750 con 0x7f7f5c0a67c0 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f52ffd640 10 client.174516.objecter ms_dispatch 0x7f7f5c000b90 osd_map(419..419 src has 299..419) 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f52ffd640 3 client.174516.objecter handle_osd_map got epochs [419,419] > 0 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f52ffd640 3 client.174516.objecter handle_osd_map decoding full epoch 419 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f52ffd640 20 client.174516.objecter dump_active .. 0 homeless 2024-08-19T10:01:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.888+0000 7f7f5ad9d640 10 monclient: get_auth_request con 0x7f7f3c040d60 auth_method 0 2024-08-19T10:01:44.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f59d9b640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f40002860 legacy=0x7f7f40004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 client.174516.objecter ms_handle_connect 0x7f7f40002860 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 client.174516.objecter resend_mon_ops 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1525396813 0 0) 0x7f7f400065c0 con 0x7f7f40002860 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient(hunting): my global_id is 144979 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f5c0a2860 con 0x7f7f40002860 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1505966323 0 0) 0x7f7f5c0a2860 con 0x7f7f40002860 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f400065c0 con 0x7f7f40002860 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7f44001320 con 0x7f7f40002860 2024-08-19T10:01:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 monclient: dump: 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 819436804 0 0) 0x7f7f400065c0 con 0x7f7f40002860 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f4000a160 con 0x7f7f40002860 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.890+0000 7f7f52ffd640 10 client.174516.objecter ms_handle_connect 0x7f7f3c040d60 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.891+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7f44004910 con 0x7f7f40002860 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.891+0000 7f7f52ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.891+0000 7f7f52ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.891+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f40002860 legacy=0x7f7f40004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.891+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f40002860 legacy=0x7f7f40004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.897+0000 7f7f60993640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.897+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7f40009e70 con 0x7f7f40009130 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f59d9b640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f40009130 legacy=0x7f7f400095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 10 client.174516.objecter ms_handle_connect 0x7f7f40009130 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 10 client.174516.objecter resend_mon_ops 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830460499 0 0) 0x7f7f40009e70 con 0x7f7f40009130 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 10 monclient(hunting): my global_id is 144988 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.899+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f400065c0 con 0x7f7f40009130 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2463130243 0 0) 0x7f7f400065c0 con 0x7f7f40009130 2024-08-19T10:01:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f40009e70 con 0x7f7f40009130 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7f44005180 con 0x7f7f40009130 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 10 monclient: dump: 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2464453567 0 0) 0x7f7f40009e70 con 0x7f7f40009130 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f4000c410 con 0x7f7f40009130 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3257132619) 0x7f7f44007440 con 0x7f7f40009130 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f40009130 legacy=0x7f7f400095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f52ffd640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6792/0 conn(0x7f7f40009130 legacy=0x7f7f400095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:01:44.916 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:01:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:01:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 106, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 429280, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:01:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:01:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:01:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:01:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:01:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.934 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.935 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:01:44.936 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:01:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 10 librados: watch_flush enter 2024-08-19T10:01:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 10 librados: watch_flush exit 2024-08-19T10:01:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 20 client.174516.objecter shutdown clearing up homeless session... 2024-08-19T10:01:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 10 client.174516.objecter successfully canceled tick 2024-08-19T10:01:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7f3c040d60 legacy=0x7f7f3c043220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 10 monclient: shutdown 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 >> v1:172.21.15.1:6792/0 conn(0x7f7f5c0a67c0 legacy=0x7f7f5c09fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f5ad9d640 1 -- 172.21.15.1:0/1939622668 reap_dead start 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f5ad9d640 1 -- 172.21.15.1:0/1939622668 reap_dead start 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 shutdown_connections 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.900+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 >> 172.21.15.1:0/1939622668 conn(0x7f7f5c0ac4b0 msgr2=0x7f7f5c0a4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.901+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 shutdown_connections 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.901+0000 7f7f60993640 1 -- 172.21.15.1:0/1939622668 wait complete. 2024-08-19T10:01:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:44.901+0000 7f7f60993640 1 librados: shutdown 2024-08-19T10:01:44.939 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:01:44.940 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d', 'c'] 2024-08-19T10:01:44.940 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-19T10:01:44.940 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T10:01:44.940 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T10:01:44.940 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T10:01:44.940 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:01:44.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727577787}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874856724}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2453885161}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1650085840}) 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.943+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3750042717}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=957775344}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=337139867}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028192232}) 2024-08-19T10:01:44.947 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:01:44.944+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3446890719}) 2024-08-19T10:01:44.951 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T10:01:44.952 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-19T10:01:44.952 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-19T10:01:44.952 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-19T10:01:44.952 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:01:45.007 INFO:tasks.ceph.mon.c:Stopped 2024-08-19T10:01:45.007 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-08-19T10:01:45.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:01:45.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.143+0000 7f24b5de9640 10 monclient: get_monmap_and_config 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.143+0000 7f24b5de9640 10 monclient: build_initial_monmap 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.143+0000 7f24b5de9640 10 monclient: monmap: 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:45.144878+0000 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:45.144878+0000 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:45.146 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:45.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.144+0000 7f24b5de9640 1 Processor -- start 2024-08-19T10:01:45.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.144+0000 7f24b5de9640 1 -- start start 2024-08-19T10:01:45.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.144+0000 7f24b5de9640 10 monclient: init 2024-08-19T10:01:45.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient: _add_conns ranks=[5,6,8,1,4,0,2,7,3] 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient(hunting): picked mon.noname-h con 0x7f24b00a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient(hunting): picked mon.noname-d con 0x7f24b00a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient(hunting): picked mon.noname-e con 0x7f24b013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b00a3b40 con 0x7f24b00a67c0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b00a8130 con 0x7f24b00a6ba0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b006ede0 con 0x7f24b013f5c0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24b5de9640 10 monclient(hunting): authenticate will time out at 1361.221299s 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24a6ffd640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f24b00a6ba0 legacy=0x7f24b00a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24affff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f24b013f5c0 0x7f24b01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:42832/0 (socket says 172.21.15.1:42832) 2024-08-19T10:01:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.145+0000 7f24affff640 1 -- 172.21.15.1:0/256676040 learned_addr learned my addr 172.21.15.1:0/256676040 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24affff640 1 -- 172.21.15.1:0/256676040 >> v1:172.21.15.1:6793/0 conn(0x7f24b013f5c0 legacy=0x7f24b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565954698 0 0) 0x7f24b006ede0 con 0x7f24b013f5c0 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24af7fe640 1 -- 172.21.15.1:0/256676040 >> v1:172.21.15.82:6791/0 conn(0x7f24b00a67c0 legacy=0x7f24b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 10 monclient(hunting): my global_id is 184265 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f249c003520 con 0x7f24b013f5c0 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.147+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 284646968 0 0) 0x7f24b00a3b40 con 0x7f24b00a67c0 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient(hunting): my global_id is 27482 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24b006ede0 con 0x7f24b00a67c0 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2388251585 0 0) 0x7f249c003520 con 0x7f24b013f5c0 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 >> v1:172.21.15.1:6792/0 conn(0x7f24b00a6ba0 legacy=0x7f24b00a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 >> v1:172.21.15.82:6791/0 conn(0x7f24b00a67c0 legacy=0x7f24b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: _finish_hunting 0 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 monclient: found mon.noname-e 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b00a92a0 con 0x7f24b013f5c0 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: _finish_auth 0 2024-08-19T10:01:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: _check_auth_tickets 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 5 monclient: authenticate success, global_id 184265 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24a8001e50 con 0x7f24b013f5c0 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: dump: 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24a8004c20 con 0x7f24b013f5c0 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 1 -- 172.21.15.1:0/256676040 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f24a8005190 con 0x7f24b013f5c0 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 4 set_mon_vals no callback set 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24aeffd640 10 monclient: dump: 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:45.156 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 10 monclient: shutdown 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 1 -- 172.21.15.1:0/256676040 >> v1:172.21.15.1:6793/0 conn(0x7f24b013f5c0 legacy=0x7f24b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 1 -- 172.21.15.1:0/256676040 shutdown_connections 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 1 -- 172.21.15.1:0/256676040 >> 172.21.15.1:0/256676040 conn(0x7f24b00ac4b0 msgr2=0x7f24b00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 1 -- 172.21.15.1:0/256676040 shutdown_connections 2024-08-19T10:01:45.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.148+0000 7f24b5de9640 1 -- 172.21.15.1:0/256676040 wait complete. 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 10 monclient: build_initial_monmap 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 10 monclient: monmap: 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:45.158 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 1 librados: starting msgr at 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 1 librados: starting objecter 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 1 Processor -- start 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 1 -- start start 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 1 librados: setting wanted keys 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 1 librados: calling monclient init 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.149+0000 7f24b5de9640 10 monclient: init 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient: _add_conns ranks=[6,0,4,1,5,3,2,7,8] 2024-08-19T10:01:45.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient(hunting): picked mon.noname-d con 0x7f24b00a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient(hunting): picked mon.noname-a con 0x7f24b00a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient(hunting): picked mon.noname-c con 0x7f24b013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b013a690 con 0x7f24b00a6ba0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b013a970 con 0x7f24b013f5c0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b00aa2a0 con 0x7f24b00a67c0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24b5de9640 10 monclient(hunting): authenticate will time out at 1361.226274s 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24af7fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f24b00a67c0 legacy=0x7f24b01ce4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24affff640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f24b013f5c0 legacy=0x7f24b01d6b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24a6ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24b00a6ba0 0x7f24b01d3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48026/0 (socket says 172.21.15.1:48026) 2024-08-19T10:01:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.150+0000 7f24a6ffd640 1 -- 172.21.15.1:0/2466041190 learned_addr learned my addr 172.21.15.1:0/2466041190 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24a6ffd640 1 -- 172.21.15.1:0/2466041190 >> v1:172.21.15.1:6789/0 conn(0x7f24b00a6ba0 legacy=0x7f24b01d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 client.?.objecter ms_handle_connect 0x7f24b00a6ba0 2024-08-19T10:01:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812238703 0 0) 0x7f24b013a690 con 0x7f24b00a6ba0 2024-08-19T10:01:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): _negotiate 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): my global_id is 174510 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2490004eb0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 366843781 0 0) 0x7f2490004eb0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24b013a690 con 0x7f24b00a6ba0 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2498004760 con 0x7f24b00a6ba0 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): dump: 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:45.165 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.152+0000 7f24acff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3196841146 0 0) 0x7f24b013a690 con 0x7f24b00a6ba0 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 >> v1:172.21.15.1:6792/0 conn(0x7f24b00a67c0 legacy=0x7f24b01ce4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 >> v1:172.21.15.1:6791/0 conn(0x7f24b013f5c0 legacy=0x7f24b01d6b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: _finish_hunting 0 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 monclient: found mon.a 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b013a3b0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: _finish_auth 0 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: _check_auth_tickets 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 5 monclient: authenticate success, global_id 174510 2024-08-19T10:01:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 10 monclient: _renew_subs 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f24b00a0d50 con 0x7f24b00a6ba0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 10 client.174510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 10 monclient: _renew_subs 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f24b01cedc0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24b5de9640 1 librados: init done 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2498004a00 con 0x7f24b00a6ba0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2498004f70 con 0x7f24b00a6ba0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24ae7fc640 4 set_mon_vals no callback set 2024-08-19T10:01:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.153+0000 7f24acff9640 10 monclient: dump: 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24b5de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24b5de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24b5de9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2494005080 con 0x7f24b00a6ba0 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f24980051c0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(419..419 src has 299..419) ==== 5516+0+0 (unknown 2589855585 0 0) 0x7f249804e420 con 0x7f24b00a6ba0 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24acff9640 10 client.174510.objecter ms_dispatch 0x7f24b0000b90 osd_map(419..419 src has 299..419) 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24acff9640 3 client.174510.objecter handle_osd_map got epochs [419,419] > 0 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24acff9640 3 client.174510.objecter handle_osd_map decoding full epoch 419 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24acff9640 20 client.174510.objecter dump_active .. 0 homeless 2024-08-19T10:01:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.154+0000 7f24af7fe640 10 monclient: get_auth_request con 0x7f2490040da0 auth_method 0 2024-08-19T10:01:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.157+0000 7f24acff9640 10 client.174510.objecter ms_handle_connect 0x7f2490040da0 2024-08-19T10:01:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.159+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f24980177a0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.159+0000 7f24acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:01:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.159+0000 7f24acff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:01:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24b5de9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24b5de9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24b5de9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2494005370 con 0x7f24b00a6ba0 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24acff9640 1 -- 172.21.15.1:0/2466041190 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 3952478108) 0x7f24980170a0 con 0x7f24b00a6ba0 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.416+0000 7f24acff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:01:45.418 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":106,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:01:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.418+0000 7f24b5de9640 10 librados: watch_flush enter 2024-08-19T10:01:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.418+0000 7f24b5de9640 10 librados: watch_flush exit 2024-08-19T10:01:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.418+0000 7f24b5de9640 20 client.174510.objecter shutdown clearing up homeless session... 2024-08-19T10:01:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.418+0000 7f24b5de9640 10 client.174510.objecter successfully canceled tick 2024-08-19T10:01:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2490040da0 legacy=0x7f2490043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 10 monclient: shutdown 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 >> v1:172.21.15.1:6789/0 conn(0x7f24b00a6ba0 legacy=0x7f24b01d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 shutdown_connections 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 >> 172.21.15.1:0/2466041190 conn(0x7f24b00ac4b0 msgr2=0x7f24b00a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 shutdown_connections 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 -- 172.21.15.1:0/2466041190 wait complete. 2024-08-19T10:01:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:45.419+0000 7f24b5de9640 1 librados: shutdown 2024-08-19T10:01:48.432 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.542+0000 7faa7cf3a640 10 monclient: get_monmap_and_config 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.542+0000 7faa7cf3a640 10 monclient: build_initial_monmap 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.542+0000 7faa7cf3a640 10 monclient: monmap: 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:01:48.543123+0000 2024-08-19T10:01:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:01:48.543123+0000 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:48.544 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.542+0000 7faa7cf3a640 1 Processor -- start 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 1 -- start start 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient: init 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient: _add_conns ranks=[7,2,0,3,4,5,6,1,8] 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient(hunting): picked mon.noname-i con 0x7faa780a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient(hunting): picked mon.noname-b con 0x7faa780694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient(hunting): picked mon.noname-a con 0x7faa780a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa780a3c70 con 0x7faa780a46a0 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa780a33d0 con 0x7faa780694b0 2024-08-19T10:01:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa780a36b0 con 0x7faa780a6830 2024-08-19T10:01:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa7cf3a640 10 monclient(hunting): authenticate will time out at 1364.619574s 2024-08-19T10:01:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa6ffff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7faa780694b0 0x7faa780a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52282/0 (socket says 172.21.15.1:52282) 2024-08-19T10:01:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa76d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faa780a46a0 0x7faa78141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45870/0 (socket says 172.21.15.1:45870) 2024-08-19T10:01:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.543+0000 7faa6ffff640 1 -- 172.21.15.1:0/1438992633 learned_addr learned my addr 172.21.15.1:0/1438992633 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.545+0000 7faa6ffff640 1 -- 172.21.15.1:0/1438992633 >> v1:172.21.15.1:6790/0 conn(0x7faa780694b0 legacy=0x7faa780a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.545+0000 7faa76d9d640 1 -- 172.21.15.1:0/1438992633 >> v1:172.21.15.1:6789/0 conn(0x7faa780a46a0 legacy=0x7faa78141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.545+0000 7faa7659c640 1 -- 172.21.15.1:0/1438992633 >> v1:172.21.15.82:6792/0 conn(0x7faa780a6830 legacy=0x7faa780a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459897995 0 0) 0x7faa780a3c70 con 0x7faa780a46a0 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient(hunting): my global_id is 174519 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa64003520 con 0x7faa780a46a0 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2951064547 0 0) 0x7faa64003520 con 0x7faa780a46a0 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 >> v1:172.21.15.82:6792/0 conn(0x7faa780a6830 legacy=0x7faa780a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 >> v1:172.21.15.1:6790/0 conn(0x7faa780694b0 legacy=0x7faa780a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 monclient: found mon.noname-a 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7806dbb0 con 0x7faa780a46a0 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: _finish_auth 0 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faa70001e60 con 0x7faa780a46a0 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa7cf3a640 5 monclient: authenticate success, global_id 174519 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa7cf3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:01:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: dump: 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa7cf3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa70004b60 con 0x7faa780a46a0 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 1 -- 172.21.15.1:0/1438992633 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faa70005090 con 0x7faa780a46a0 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa7cf3a640 10 monclient: get_monmap_and_config success 2024-08-19T10:01:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa7cf3a640 4 set_mon_vals no callback set 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.546+0000 7faa75d9b640 10 monclient: dump: 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:01:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 10 monclient: shutdown 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:01:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1438992633 >> v1:172.21.15.1:6789/0 conn(0x7faa780a46a0 legacy=0x7faa78141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1438992633 shutdown_connections 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1438992633 >> 172.21.15.1:0/1438992633 conn(0x7faa780a9020 msgr2=0x7faa78068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1438992633 shutdown_connections 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1438992633 wait complete. 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 10 monclient: build_initial_monmap 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.547+0000 7faa7cf3a640 10 monclient: monmap: 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:01:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 librados: starting msgr at 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 librados: starting objecter 2024-08-19T10:01:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 Processor -- start 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 -- start start 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 librados: setting wanted keys 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 librados: calling monclient init 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient: init 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient: _reopen_session rank -1 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient: _add_conns ranks=[3,2,4,6,1,7,5,8,0] 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient(hunting): picked mon.noname-g con 0x7faa780694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient(hunting): picked mon.noname-b con 0x7faa780a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient(hunting): picked mon.noname-c con 0x7faa780a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa78137320 con 0x7faa780a46a0 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa78135e30 con 0x7faa780694b0 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa78138280 con 0x7faa780a6830 2024-08-19T10:01:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa7cf3a640 10 monclient(hunting): authenticate will time out at 1364.624781s 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa76d9d640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7faa780a6830 legacy=0x7faa781d6400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa6ffff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7faa780a46a0 0x7faa78135720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:52294/0 (socket says 172.21.15.1:52294) 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.548+0000 7faa6ffff640 1 -- 172.21.15.1:0/1728739263 learned_addr learned my addr 172.21.15.1:0/1728739263 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.551+0000 7faa6ffff640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6790/0 conn(0x7faa780a46a0 legacy=0x7faa78135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa780a46a0 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa780694b0 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.551+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6790/0 conn(0x7faa780694b0 legacy=0x7faa78135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:48.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:48.749+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa780a6830 legacy=0x7faa781d6400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:49.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:49.150+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa780a6830 legacy=0x7faa781d6400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:49.951+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa780a6830 legacy=0x7faa781d6400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): tick 2024-08-19T10:01:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:01:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 monclient(hunting): continuing hunt 2024-08-19T10:01:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa780a6830 legacy=0x7faa781d6400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6790/0 conn(0x7faa780694b0 legacy=0x7faa78135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6790/0 conn(0x7faa780a46a0 legacy=0x7faa78135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient: _add_conns ranks=[6,0,7,1,8,5] 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-d con 0x7faa5c0038c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-a con 0x7faa5c0074b0 addr v1:172.21.15.1:6789/0 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-i con 0x7faa5c00b220 addr v1:172.21.15.82:6792/0 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa78138560 con 0x7faa5c0074b0 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa78138280 con 0x7faa5c0038c0 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c011100 con 0x7faa5c00b220 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa6dffb640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.549+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6792/0 conn(0x7faa5c0038c0 legacy=0x7faa5c005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:01:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.551+0000 7faa6ffff640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6789/0 conn(0x7faa5c0074b0 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c0074b0 2024-08-19T10:01:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c00b220 2024-08-19T10:01:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.551+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6792/0 conn(0x7faa5c00b220 legacy=0x7faa5c00d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:51.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:51.750+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6792/0 conn(0x7faa5c0038c0 legacy=0x7faa5c005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:01:52.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:52.151+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6792/0 conn(0x7faa5c0038c0 legacy=0x7faa5c005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:01:52.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:52.952+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6792/0 conn(0x7faa5c0038c0 legacy=0x7faa5c005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:01:54.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): tick 2024-08-19T10:01:54.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:01:54.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 monclient(hunting): continuing hunt 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6792/0 conn(0x7faa5c00b220 legacy=0x7faa5c00d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6792/0 conn(0x7faa5c0038c0 legacy=0x7faa5c005d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6789/0 conn(0x7faa5c0074b0 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient: _add_conns ranks=[5,1,8] 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-h con 0x7faa5c0114b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-f con 0x7faa5c010010 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 reap_dead start 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-e con 0x7faa5c010bd0 addr v1:172.21.15.1:6793/0 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c012400 con 0x7faa5c010010 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa78138280 con 0x7faa5c0114b0 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c00a120 con 0x7faa5c010bd0 2024-08-19T10:01:54.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.549+0000 7faa6dffb640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:54.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6793/0 conn(0x7faa5c010bd0 legacy=0x7faa5c019770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:54.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c010bd0 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c0114b0 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6791/0 conn(0x7faa5c0114b0 legacy=0x7faa5c0118f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c010010 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:54.551+0000 7faa6ffff640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6789/0 conn(0x7faa5c010010 legacy=0x7faa5c0104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): tick 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 monclient(hunting): continuing hunt 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6793/0 conn(0x7faa5c010bd0 legacy=0x7faa5c019770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6791/0 conn(0x7faa5c0114b0 legacy=0x7faa5c0118f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6789/0 conn(0x7faa5c010010 legacy=0x7faa5c0104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient: _add_conns ranks=[4,1,3,6,8,7,2,5,0] 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-c con 0x7faa5c00a730 addr v1:172.21.15.1:6791/0 2024-08-19T10:01:57.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-f con 0x7faa5c010010 addr v1:172.21.15.82:6789/0 2024-08-19T10:01:57.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): picked mon.noname-g con 0x7faa5c00ac30 addr v1:172.21.15.82:6790/0 2024-08-19T10:01:57.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c00a2f0 con 0x7faa5c010010 2024-08-19T10:01:57.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c005d20 con 0x7faa5c00ac30 2024-08-19T10:01:57.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c006fa0 con 0x7faa5c00a730 2024-08-19T10:01:57.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa6dffb640 10 monclient(hunting): _renew_subs 2024-08-19T10:01:57.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.549+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa5c00a730 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.552+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c010010 2024-08-19T10:01:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.552+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.552+0000 7faa6ffff640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6789/0 conn(0x7faa5c010010 legacy=0x7faa5c0104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.552+0000 7faa6effd640 10 client.?.objecter ms_handle_connect 0x7faa5c00ac30 2024-08-19T10:01:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.552+0000 7faa6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:01:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.552+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6790/0 conn(0x7faa5c00ac30 legacy=0x7faa5c00d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:01:57.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:57.750+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa5c00a730 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:58.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:58.151+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa5c00a730 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:58.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:58.953+0000 7faa7659c640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa5c00a730 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:01:59.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 697125660 0 0) 0x7faa5c005d20 con 0x7faa5c00ac30 2024-08-19T10:01:59.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:59.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:59.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:59.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): my global_id is 184251 2024-08-19T10:01:59.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa58004eb0 con 0x7faa5c00ac30 2024-08-19T10:01:59.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 143578727 0 0) 0x7faa5c00a2f0 con 0x7faa5c010010 2024-08-19T10:01:59.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): _negotiate 2024-08-19T10:01:59.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:01:59.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:01:59.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 10 monclient(hunting): my global_id is 174484 2024-08-19T10:01:59.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.705+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa5c005d20 con 0x7faa5c010010 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2457096003 0 0) 0x7faa58004eb0 con 0x7faa5c00ac30 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa5c00a2f0 con 0x7faa5c00ac30 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faa70001e60 con 0x7faa5c00ac30 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): dump: 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.001 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.002 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2298111410 0 0) 0x7faa5c005d20 con 0x7faa5c010010 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa58004eb0 con 0x7faa5c010010 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faa60001cc0 con 0x7faa5c010010 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:01:59.999+0000 7faa6effd640 10 monclient(hunting): dump: 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.003 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.004 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2847118866 0 0) 0x7faa5c00a2f0 con 0x7faa5c00ac30 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.1:6791/0 conn(0x7faa5c00a730 legacy=0x7faa5c009970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6789/0 conn(0x7faa5c010010 legacy=0x7faa5c0104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: _finish_hunting 0 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 monclient: found mon.g 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5c007230 con 0x7faa5c00ac30 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: _finish_auth 0 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: _check_auth_tickets 2024-08-19T10:02:00.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 5 monclient: authenticate success, global_id 184251 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 10 monclient: _renew_subs 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa781368c0 con 0x7faa5c00ac30 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa70005070 con 0x7faa5c00ac30 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 10 client.184251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 10 monclient: _renew_subs 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7faa700059c0 con 0x7faa5c00ac30 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7faa781425c0 con 0x7faa5c00ac30 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa7cf3a640 1 librados: init done 2024-08-19T10:02:00.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa6effd640 10 monclient: dump: 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.008+0000 7faa74d99640 4 set_mon_vals no callback set 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7faa70011e20 con 0x7faa5c00ac30 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(419..419 src has 299..419) ==== 5516+0+0 (unknown 2589855585 0 0) 0x7faa7004e3e0 con 0x7faa5c00ac30 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa6effd640 10 client.184251.objecter ms_dispatch 0x7faa78000b90 osd_map(419..419 src has 299..419) 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa6effd640 3 client.184251.objecter handle_osd_map got epochs [419,419] > 0 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa6effd640 3 client.184251.objecter handle_osd_map decoding full epoch 419 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa7cf3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa7cf3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa7cf3a640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:02:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.009+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa78136a90 con 0x7faa5c00ac30 2024-08-19T10:02:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.010+0000 7faa6effd640 20 client.184251.objecter dump_active .. 0 homeless 2024-08-19T10:02:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.010+0000 7faa7659c640 10 monclient: get_auth_request con 0x7faa58041480 auth_method 0 2024-08-19T10:02:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.012+0000 7faa6effd640 10 client.184251.objecter ms_handle_connect 0x7faa58041480 2024-08-19T10:02:00.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.015+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7faa78136a90 con 0x7faa5c00ac30 2024-08-19T10:02:00.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.015+0000 7faa6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.015+0000 7faa6effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:00.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.275+0000 7faa7cf3a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:02:00.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.275+0000 7faa7cf3a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:02:00.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.275+0000 7faa7cf3a640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:02:00.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.275+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 --> v1:172.21.15.82:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faa78051b00 con 0x7faa5c00ac30 2024-08-19T10:02:00.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.276+0000 7faa6effd640 1 -- 172.21.15.1:0/1728739263 <== mon.3 v1:172.21.15.82:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2691 (unknown 254746657 0 3158009363) 0x7faa700177a0 con 0x7faa5c00ac30 2024-08-19T10:02:00.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.276+0000 7faa6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:02:00.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.276+0000 7faa6effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:00.277 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":110,"quorum":[0,1,2,3,5,7,8],"quorum_names":["a","f","b","g","h","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 10 librados: watch_flush enter 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 10 librados: watch_flush exit 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 20 client.184251.objecter shutdown clearing up homeless session... 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 10 client.184251.objecter successfully canceled tick 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6812/2549104415 conn(0x7faa58041480 legacy=0x7faa58043940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 10 monclient: shutdown 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 >> v1:172.21.15.82:6790/0 conn(0x7faa5c00ac30 legacy=0x7faa5c00d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 reap_dead start 2024-08-19T10:02:00.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa76d9d640 1 -- 172.21.15.1:0/1728739263 reap_dead start 2024-08-19T10:02:00.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 shutdown_connections 2024-08-19T10:02:00.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 >> 172.21.15.1:0/1728739263 conn(0x7faa780a9020 msgr2=0x7faa780a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:00.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 shutdown_connections 2024-08-19T10:02:00.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 -- 172.21.15.1:0/1728739263 wait complete. 2024-08-19T10:02:00.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.278+0000 7faa7cf3a640 1 librados: shutdown 2024-08-19T10:02:00.290 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-08-19T10:02:00.290 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-08-19T10:02:00.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:02:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.389+0000 7f3f5797f640 10 monclient: get_monmap_and_config 2024-08-19T10:02:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.389+0000 7f3f5797f640 10 monclient: build_initial_monmap 2024-08-19T10:02:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.389+0000 7f3f5797f640 10 monclient: monmap: 2024-08-19T10:02:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:00.390167+0000 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:00.390167+0000 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:00.391 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:00.392 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.389+0000 7f3f5797f640 1 Processor -- start 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 1 -- start start 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient: init 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient: _add_conns ranks=[5,6,0,3,1,2,4,7,8] 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient(hunting): picked mon.noname-h con 0x7f3f500a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient(hunting): picked mon.noname-d con 0x7f3f500a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient(hunting): picked mon.noname-a con 0x7f3f5013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:02:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f500a3b40 con 0x7f3f5013f5c0 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f500a8130 con 0x7f3f500a67c0 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f5006ede0 con 0x7f3f500a6ba0 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f5797f640 10 monclient(hunting): authenticate will time out at 1376.466558s 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f54f1a640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f3f500a6ba0 legacy=0x7f3f500a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f55f1c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 0x7f3f501419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39730/0 (socket says 172.21.15.1:39730) 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.390+0000 7f3f55f1c640 1 -- 172.21.15.1:0/2328311500 learned_addr learned my addr 172.21.15.1:0/2328311500 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f55f1c640 1 -- 172.21.15.1:0/2328311500 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 legacy=0x7f3f501419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194355708 0 0) 0x7f3f500a3b40 con 0x7f3f5013f5c0 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f4ffff640 10 monclient(hunting): my global_id is 174528 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.392+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f38003520 con 0x7f3f5013f5c0 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5571b640 1 -- 172.21.15.1:0/2328311500 >> v1:172.21.15.82:6791/0 conn(0x7f3f500a67c0 legacy=0x7f3f500a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1578358988 0 0) 0x7f3f38003520 con 0x7f3f5013f5c0 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 >> v1:172.21.15.1:6792/0 conn(0x7f3f500a6ba0 legacy=0x7f3f500a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 >> v1:172.21.15.82:6791/0 conn(0x7f3f500a67c0 legacy=0x7f3f500a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:02:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 monclient: found mon.noname-a 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f500a92a0 con 0x7f3f5013f5c0 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: _finish_auth 0 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 5 monclient: authenticate success, global_id 174528 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f48001ae0 con 0x7f3f5013f5c0 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:02:00.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: dump: 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f480048a0 con 0x7f3f5013f5c0 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 1 -- 172.21.15.1:0/2328311500 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f48004e10 con 0x7f3f5013f5c0 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 4 set_mon_vals no callback set 2024-08-19T10:02:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f4ffff640 10 monclient: dump: 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.403 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 10 monclient: shutdown 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 1 -- 172.21.15.1:0/2328311500 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 legacy=0x7f3f501419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 1 -- 172.21.15.1:0/2328311500 shutdown_connections 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 1 -- 172.21.15.1:0/2328311500 >> 172.21.15.1:0/2328311500 conn(0x7f3f500ac4b0 msgr2=0x7f3f500a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 1 -- 172.21.15.1:0/2328311500 shutdown_connections 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.393+0000 7f3f5797f640 1 -- 172.21.15.1:0/2328311500 wait complete. 2024-08-19T10:02:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 10 monclient: build_initial_monmap 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 10 monclient: monmap: 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 1 librados: starting msgr at 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 1 librados: starting objecter 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 1 Processor -- start 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 1 -- start start 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 1 librados: setting wanted keys 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 1 librados: calling monclient init 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.394+0000 7f3f5797f640 10 monclient: init 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient: _add_conns ranks=[5,1,0,3,2,7,8,4,6] 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient(hunting): picked mon.noname-h con 0x7f3f500a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient(hunting): picked mon.noname-f con 0x7f3f500a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient(hunting): picked mon.noname-a con 0x7f3f5013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f500a9cc0 con 0x7f3f5013f5c0 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f50142b10 con 0x7f3f500a6ba0 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f5013a850 con 0x7f3f500a67c0 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f5797f640 10 monclient(hunting): authenticate will time out at 1376.471356s 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f55f1c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 0x7f3f501d6b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39744/0 (socket says 172.21.15.1:39744) 2024-08-19T10:02:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.395+0000 7f3f55f1c640 1 -- 172.21.15.1:0/1059526379 learned_addr learned my addr 172.21.15.1:0/1059526379 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f55f1c640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 legacy=0x7f3f501d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 client.?.objecter ms_handle_connect 0x7f3f5013f5c0 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f5571b640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.82:6791/0 conn(0x7f3f500a67c0 legacy=0x7f3f501ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 client.?.objecter ms_handle_connect 0x7f3f500a67c0 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 client.?.objecter ms_handle_connect 0x7f3f500a6ba0 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f54f1a640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.82:6789/0 conn(0x7f3f500a6ba0 legacy=0x7f3f501d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 717125635 0 0) 0x7f3f500a92a0 con 0x7f3f5013f5c0 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): my global_id is 174537 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f3c004ee0 con 0x7f3f5013f5c0 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700052738 0 0) 0x7f3f5013a850 con 0x7f3f500a67c0 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): my global_id is 27500 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f500a92a0 con 0x7f3f500a67c0 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830260786 0 0) 0x7f3f440040b0 con 0x7f3f500a6ba0 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): my global_id is 174493 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f5013a850 con 0x7f3f500a6ba0 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f55f1c640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 legacy=0x7f3f501d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f55f1c640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 legacy=0x7f3f501d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f55f1c640 1 --1- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 0x7f3f501d6b70 :-1 s=OPENED pgs=539 cs=1 l=1).handle_message read tag failed 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f55f1c640 1 --1- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 0x7f3f501d6b70 :-1 s=OPENED pgs=539 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.397+0000 7f3f4dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6789/0 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3810215190 0 0) 0x7f3f5013a850 con 0x7f3f500a6ba0 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f440040b0 con 0x7f3f500a6ba0 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f44004b20 con 0x7f3f500a6ba0 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:02:00.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): dump: 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.419 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2004060651 0 0) 0x7f3f500a92a0 con 0x7f3f500a67c0 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f5013a850 con 0x7f3f500a67c0 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f40003610 con 0x7f3f500a67c0 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:00.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient(hunting): dump: 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.421 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 33736393 0 0) 0x7f3f440040b0 con 0x7f3f500a6ba0 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.82:6791/0 conn(0x7f3f500a67c0 legacy=0x7f3f501ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5013f5c0 legacy=0x7f3f501d6b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:02:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 monclient: found mon.f 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5013ab30 con 0x7f3f500a6ba0 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: _finish_auth 0 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 5 monclient: authenticate success, global_id 174493 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 10 monclient: _renew_subs 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:02:00.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f500a17a0 con 0x7f3f500a6ba0 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f44004f70 con 0x7f3f500a6ba0 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f44005520 con 0x7f3f500a6ba0 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4dffb640 10 monclient: dump: 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 10 client.174493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f4f7fe640 4 set_mon_vals no callback set 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 10 monclient: _renew_subs 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f500a0e80 con 0x7f3f500a6ba0 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.398+0000 7f3f5797f640 1 librados: init done 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f5797f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3f500a27c0 con 0x7f3f501cebe0 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3f44005780 con 0x7f3f500a6ba0 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(419..419 src has 299..419) ==== 5516+0+0 (unknown 2589855585 0 0) 0x7f3f4404e720 con 0x7f3f500a6ba0 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f4dffb640 10 client.174493.objecter ms_dispatch 0x7f3f50000b90 osd_map(419..419 src has 299..419) 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f4dffb640 3 client.174493.objecter handle_osd_map got epochs [419,419] > 0 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f4dffb640 3 client.174493.objecter handle_osd_map decoding full epoch 419 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f4dffb640 20 client.174493.objecter dump_active .. 0 homeless 2024-08-19T10:02:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.399+0000 7f3f55f1c640 10 monclient: get_auth_request con 0x7f3f3c0415d0 auth_method 0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f5571b640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f501cebe0 legacy=0x7f3f5013fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 10 client.174493.objecter ms_handle_connect 0x7f3f501cebe0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 10 client.174493.objecter resend_mon_ops 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146167736 0 0) 0x7f3f500a27c0 con 0x7f3f501cebe0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 10 monclient(hunting): my global_id is 174546 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f440040b0 con 0x7f3f501cebe0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.401+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1864178093 0 0) 0x7f3f440040b0 con 0x7f3f501cebe0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f500a27c0 con 0x7f3f501cebe0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f40003e10 con 0x7f3f501cebe0 2024-08-19T10:02:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 10 monclient: dump: 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 10 client.174493.objecter ms_handle_connect 0x7f3f3c0415d0 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2260091827 0 0) 0x7f3f500a27c0 con 0x7f3f501cebe0 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f5006ee70 con 0x7f3f501cebe0 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3f40004100 con 0x7f3f501cebe0 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f501cebe0 legacy=0x7f3f5013fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.402+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f501cebe0 legacy=0x7f3f5013fe20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.408+0000 7f3f5797f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.408+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3f501cefc0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f5571b640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5002e1d0 legacy=0x7f3f50140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 client.174493.objecter ms_handle_connect 0x7f3f5002e1d0 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 client.174493.objecter resend_mon_ops 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297616247 0 0) 0x7f3f501cefc0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient(hunting): my global_id is 174555 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f500a27c0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2533561311 0 0) 0x7f3f500a27c0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f501cefc0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3f40005b00 con 0x7f3f5002e1d0 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient: dump: 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1316101775 0 0) 0x7f3f501cefc0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f500a11b0 con 0x7f3f5002e1d0 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7665 (unknown 0 0 3449138117) 0x7f3f40005b80 con 0x7f3f5002e1d0 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5002e1d0 legacy=0x7f3f50140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.411+0000 7f3f4dffb640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.1:6789/0 conn(0x7f3f5002e1d0 legacy=0x7f3f50140530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 170451, 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:00.434 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:00.436 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:00.437 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:00.438 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:00.439 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:00.440 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:00.441 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:00.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:00.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:00.446 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.446 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.452 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:00.454 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.455 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:00.457 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:00.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 10 librados: watch_flush enter 2024-08-19T10:02:00.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 10 librados: watch_flush exit 2024-08-19T10:02:00.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 20 client.174493.objecter shutdown clearing up homeless session... 2024-08-19T10:02:00.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 10 client.174493.objecter successfully canceled tick 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3f3c0415d0 legacy=0x7f3f3c043a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 10 monclient: shutdown 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 >> v1:172.21.15.82:6789/0 conn(0x7f3f500a6ba0 legacy=0x7f3f501d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f55f1c640 1 -- 172.21.15.1:0/1059526379 reap_dead start 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f55f1c640 1 -- 172.21.15.1:0/1059526379 reap_dead start 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 shutdown_connections 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 >> 172.21.15.1:0/1059526379 conn(0x7f3f500ac4b0 msgr2=0x7f3f500a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 shutdown_connections 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 -- 172.21.15.1:0/1059526379 wait complete. 2024-08-19T10:02:00.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.412+0000 7f3f5797f640 1 librados: shutdown 2024-08-19T10:02:00.461 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:00.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-08-19T10:02:00.461 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:02:00.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.559+0000 7f74d4704640 10 monclient: get_monmap_and_config 2024-08-19T10:02:00.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.559+0000 7f74d4704640 10 monclient: build_initial_monmap 2024-08-19T10:02:00.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.559+0000 7f74d4704640 10 monclient: monmap: 2024-08-19T10:02:00.560 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:00.560435+0000 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:00.560435+0000 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:00.562 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:00.562 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 1 Processor -- start 2024-08-19T10:02:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 1 -- start start 2024-08-19T10:02:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient: init 2024-08-19T10:02:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient: _add_conns ranks=[5,7,2,0,3,1,8,6,4] 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient(hunting): picked mon.noname-h con 0x7f74cc0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient(hunting): picked mon.noname-i con 0x7f74cc0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient(hunting): picked mon.noname-b con 0x7f74cc13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc0a3b40 con 0x7f74cc13f5c0 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc0a8130 con 0x7f74cc0a67c0 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc06ede0 con 0x7f74cc0a6ba0 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d4704640 10 monclient(hunting): authenticate will time out at 1376.636830s 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d2ca1640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f74cc13f5c0 0x7f74cc1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33012/0 (socket says 172.21.15.1:33012) 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.560+0000 7f74d2ca1640 1 -- 172.21.15.1:0/2829602760 learned_addr learned my addr 172.21.15.1:0/2829602760 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d2ca1640 1 -- 172.21.15.1:0/2829602760 >> v1:172.21.15.1:6790/0 conn(0x7f74cc13f5c0 legacy=0x7f74cc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d1c9f640 1 -- 172.21.15.1:0/2829602760 >> v1:172.21.15.82:6792/0 conn(0x7f74cc0a6ba0 legacy=0x7f74cc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d24a0640 1 -- 172.21.15.1:0/2829602760 >> v1:172.21.15.82:6791/0 conn(0x7f74cc0a67c0 legacy=0x7f74cc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997942903 0 0) 0x7f74cc0a3b40 con 0x7f74cc13f5c0 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): my global_id is 184214 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74b8003520 con 0x7f74cc13f5c0 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303391305 0 0) 0x7f74cc0a8130 con 0x7f74cc0a67c0 2024-08-19T10:02:00.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): my global_id is 27509 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74cc0a3b40 con 0x7f74cc0a67c0 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339865055 0 0) 0x7f74cc06ede0 con 0x7f74cc0a6ba0 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient(hunting): my global_id is 144997 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74cc0a8130 con 0x7f74cc0a6ba0 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 286339836 0 0) 0x7f74b8003520 con 0x7f74cc13f5c0 2024-08-19T10:02:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 >> v1:172.21.15.82:6792/0 conn(0x7f74cc0a6ba0 legacy=0x7f74cc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 >> v1:172.21.15.82:6791/0 conn(0x7f74cc0a67c0 legacy=0x7f74cc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: _finish_hunting 0 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 monclient: found mon.noname-b 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc0a92a0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: _finish_auth 0 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: _check_auth_tickets 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d4704640 5 monclient: authenticate success, global_id 184214 2024-08-19T10:02:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d4704640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74c4001e60 con 0x7f74cc13f5c0 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: dump: 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d4704640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74c4004c20 con 0x7f74cc13f5c0 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 1 -- 172.21.15.1:0/2829602760 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74c4005110 con 0x7f74cc13f5c0 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d4704640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d4704640 4 set_mon_vals no callback set 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.563+0000 7f74d149e640 10 monclient: dump: 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 10 monclient: shutdown 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 1 -- 172.21.15.1:0/2829602760 >> v1:172.21.15.1:6790/0 conn(0x7f74cc13f5c0 legacy=0x7f74cc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 1 -- 172.21.15.1:0/2829602760 shutdown_connections 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 1 -- 172.21.15.1:0/2829602760 >> 172.21.15.1:0/2829602760 conn(0x7f74cc0ac4b0 msgr2=0x7f74cc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 1 -- 172.21.15.1:0/2829602760 shutdown_connections 2024-08-19T10:02:00.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 1 -- 172.21.15.1:0/2829602760 wait complete. 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 10 monclient: build_initial_monmap 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.564+0000 7f74d4704640 10 monclient: monmap: 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:00.578 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:00.579 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:00.579 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:00.579 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:00.579 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:00.579 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 librados: starting msgr at 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 librados: starting objecter 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 Processor -- start 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 -- start start 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 librados: setting wanted keys 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 librados: calling monclient init 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient: init 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient: _add_conns ranks=[4,3,8,2,1,6,0,5,7] 2024-08-19T10:02:00.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient(hunting): picked mon.noname-c con 0x7f74cc0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient(hunting): picked mon.noname-g con 0x7f74cc0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient(hunting): picked mon.noname-e con 0x7f74cc13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d24a0640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f74cc0a67c0 legacy=0x7f74cc13c160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc13adf0 con 0x7f74cc0a6ba0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74c2ffd640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc139900 con 0x7f74cc0a67c0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc13bc60 con 0x7f74cc13f5c0 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d4704640 10 monclient(hunting): authenticate will time out at 1376.641821s 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d2ca1640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f74cc13f5c0 0x7f74cc1d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:59090/0 (socket says 172.21.15.1:59090) 2024-08-19T10:02:00.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.565+0000 7f74d2ca1640 1 -- 172.21.15.1:0/1904687677 learned_addr learned my addr 172.21.15.1:0/1904687677 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d2ca1640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6793/0 conn(0x7f74cc13f5c0 legacy=0x7f74cc1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 client.?.objecter ms_handle_connect 0x7f74cc13f5c0 2024-08-19T10:02:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886385279 0 0) 0x7f74cc13bc60 con 0x7f74cc13f5c0 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): my global_id is 184274 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74b0004eb0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 client.?.objecter ms_handle_connect 0x7f74cc0a6ba0 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d1c9f640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.82:6790/0 conn(0x7f74cc0a6ba0 legacy=0x7f74cc13c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652254235 0 0) 0x7f74cc13adf0 con 0x7f74cc0a6ba0 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): my global_id is 184260 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74cc13bc60 con 0x7f74cc0a6ba0 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 698274680 0 0) 0x7f74b0004eb0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74cc13adf0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74c40020b0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): dump: 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.591 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:00.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 325109848 0 0) 0x7f74cc13adf0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6791/0 conn(0x7f74cc0a67c0 legacy=0x7f74cc13c160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:02:00.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.82:6790/0 conn(0x7f74cc0a6ba0 legacy=0x7f74cc13c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 monclient: found mon.e 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc13b3c0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 5 monclient: authenticate success, global_id 184274 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 10 monclient: _renew_subs 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f74cc13a6d0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 10 client.184274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:00.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 10 monclient: _renew_subs 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f74cc13e6b0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d4704640 1 librados: init done 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74c40053c0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74c40058b0 con 0x7f74cc13f5c0 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74d0c9d640 4 set_mon_vals no callback set 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.568+0000 7f74c2ffd640 10 monclient: dump: 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.595 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.596 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74d4704640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74b40065c0 con 0x7f74b4002860 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f74c4011d10 con 0x7f74cc13f5c0 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(419..419 src has 299..419) ==== 5516+0+0 (unknown 2589855585 0 0) 0x7f74c404e580 con 0x7f74cc13f5c0 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74c2ffd640 10 client.184274.objecter ms_dispatch 0x7f74cc000b90 osd_map(419..419 src has 299..419) 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74c2ffd640 3 client.184274.objecter handle_osd_map got epochs [419,419] > 0 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.569+0000 7f74c2ffd640 3 client.184274.objecter handle_osd_map decoding full epoch 419 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.570+0000 7f74c2ffd640 20 client.184274.objecter dump_active .. 0 homeless 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.570+0000 7f74d1c9f640 10 monclient: get_auth_request con 0x7f74b0040e20 auth_method 0 2024-08-19T10:02:00.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 10 client.184274.objecter ms_handle_connect 0x7f74b4002860 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74d24a0640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6790/0 conn(0x7f74b4002860 legacy=0x7f74b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 10 client.184274.objecter resend_mon_ops 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 442908079 0 0) 0x7f74b40065c0 con 0x7f74b4002860 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 10 monclient(hunting): my global_id is 184223 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.571+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74cc13adf0 con 0x7f74b4002860 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1455360895 0 0) 0x7f74cc13adf0 con 0x7f74b4002860 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74b40065c0 con 0x7f74b4002860 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74c8001c60 con 0x7f74b4002860 2024-08-19T10:02:00.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 10 monclient: dump: 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.599 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3490215823 0 0) 0x7f74b40065c0 con 0x7f74b4002860 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74b400a160 con 0x7f74b4002860 2024-08-19T10:02:00.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 10 client.184274.objecter ms_handle_connect 0x7f74b0040e20 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f74c80067f0 con 0x7f74b4002860 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6790/0 conn(0x7f74b4002860 legacy=0x7f74b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.572+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6790/0 conn(0x7f74b4002860 legacy=0x7f74b4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.578+0000 7f74d4704640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.578+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74b4009e70 con 0x7f74b4009130 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74d24a0640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6790/0 conn(0x7f74b4009130 legacy=0x7f74b40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 client.184274.objecter ms_handle_connect 0x7f74b4009130 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 client.184274.objecter resend_mon_ops 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128963141 0 0) 0x7f74b4009e70 con 0x7f74b4009130 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient(hunting): my global_id is 184232 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74b40065c0 con 0x7f74b4009130 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1715685542 0 0) 0x7f74b40065c0 con 0x7f74b4009130 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74b4009e70 con 0x7f74b4009130 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74c8003d40 con 0x7f74b4009130 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient: dump: 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.602 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 303424697 0 0) 0x7f74b4009e70 con 0x7f74b4009130 2024-08-19T10:02:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f74b400c410 con 0x7f74b4009130 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 561413286) 0x7f74c80040b0 con 0x7f74b4009130 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6790/0 conn(0x7f74b4009130 legacy=0x7f74b40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.581+0000 7f74c2ffd640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6790/0 conn(0x7f74b4009130 legacy=0x7f74b40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 37829, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T10:02:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:00.629 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 10 librados: watch_flush enter 2024-08-19T10:02:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 10 librados: watch_flush exit 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 20 client.184274.objecter shutdown clearing up homeless session... 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 10 client.184274.objecter successfully canceled tick 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.82:6812/2549104415 conn(0x7f74b0040e20 legacy=0x7f74b00432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 10 monclient: shutdown 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 >> v1:172.21.15.1:6793/0 conn(0x7f74cc13f5c0 legacy=0x7f74cc1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d2ca1640 1 -- 172.21.15.1:0/1904687677 reap_dead start 2024-08-19T10:02:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 shutdown_connections 2024-08-19T10:02:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 >> 172.21.15.1:0/1904687677 conn(0x7f74cc0ac4b0 msgr2=0x7f74cc13ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 shutdown_connections 2024-08-19T10:02:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 -- 172.21.15.1:0/1904687677 wait complete. 2024-08-19T10:02:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.582+0000 7f74d4704640 1 librados: shutdown 2024-08-19T10:02:00.633 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:00.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-08-19T10:02:00.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:02:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.734+0000 7fc76017f640 10 monclient: get_monmap_and_config 2024-08-19T10:02:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.734+0000 7fc76017f640 10 monclient: build_initial_monmap 2024-08-19T10:02:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.734+0000 7fc76017f640 10 monclient: monmap: 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:00.735474+0000 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:00.735474+0000 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:00.737 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 1 Processor -- start 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 1 -- start start 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient: init 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient: _add_conns ranks=[0,1,3,5,8,2,4,6,7] 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7580a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:02:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient(hunting): picked mon.noname-f con 0x7fc7580a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient(hunting): picked mon.noname-g con 0x7fc75813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7580a3b40 con 0x7fc7580a67c0 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7580a8130 con 0x7fc7580a6ba0 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75806ede0 con 0x7fc75813f5c0 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc76017f640 10 monclient(hunting): authenticate will time out at 1376.811813s 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc75df1b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc7580a67c0 0x7fc7580a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39768/0 (socket says 172.21.15.1:39768) 2024-08-19T10:02:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.735+0000 7fc75df1b640 1 -- 172.21.15.1:0/2938963034 learned_addr learned my addr 172.21.15.1:0/2938963034 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75df1b640 1 -- 172.21.15.1:0/2938963034 >> v1:172.21.15.1:6789/0 conn(0x7fc7580a67c0 legacy=0x7fc7580a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915061458 0 0) 0x7fc7580a3b40 con 0x7fc7580a67c0 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient(hunting): _negotiate 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient(hunting): my global_id is 174564 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75e71c640 1 -- 172.21.15.1:0/2938963034 >> v1:172.21.15.82:6790/0 conn(0x7fc75813f5c0 legacy=0x7fc7581419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc744003520 con 0x7fc7580a67c0 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75d71a640 1 -- 172.21.15.1:0/2938963034 >> v1:172.21.15.82:6789/0 conn(0x7fc7580a6ba0 legacy=0x7fc7580a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 310287596 0 0) 0x7fc744003520 con 0x7fc7580a67c0 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 >> v1:172.21.15.82:6790/0 conn(0x7fc75813f5c0 legacy=0x7fc7581419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 >> v1:172.21.15.82:6789/0 conn(0x7fc7580a6ba0 legacy=0x7fc7580a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: _finish_hunting 0 2024-08-19T10:02:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 monclient: found mon.noname-a 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7580a92a0 con 0x7fc7580a67c0 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: _finish_auth 0 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: _check_auth_tickets 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc754001ae0 con 0x7fc7580a67c0 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc76017f640 5 monclient: authenticate success, global_id 174564 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:02:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: dump: 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc76017f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7540048a0 con 0x7fc7580a67c0 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 1 -- 172.21.15.1:0/2938963034 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc754004dd0 con 0x7fc7580a67c0 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc76017f640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc76017f640 4 set_mon_vals no callback set 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.738+0000 7fc75cf19640 10 monclient: dump: 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 10 monclient: shutdown 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 -- 172.21.15.1:0/2938963034 >> v1:172.21.15.1:6789/0 conn(0x7fc7580a67c0 legacy=0x7fc7580a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 -- 172.21.15.1:0/2938963034 shutdown_connections 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 -- 172.21.15.1:0/2938963034 >> 172.21.15.1:0/2938963034 conn(0x7fc7580ac4b0 msgr2=0x7fc7580a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 -- 172.21.15.1:0/2938963034 shutdown_connections 2024-08-19T10:02:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 -- 172.21.15.1:0/2938963034 wait complete. 2024-08-19T10:02:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 10 monclient: build_initial_monmap 2024-08-19T10:02:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 10 monclient: monmap: 2024-08-19T10:02:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 librados: starting msgr at 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.739+0000 7fc76017f640 1 librados: starting objecter 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 Processor -- start 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 -- start start 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 librados: setting wanted keys 2024-08-19T10:02:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 librados: calling monclient init 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient: init 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient: _add_conns ranks=[8,5,7,0,2,1,6,4,3] 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient(hunting): picked mon.noname-e con 0x7fc7580a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient(hunting): picked mon.noname-h con 0x7fc7580a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient(hunting): picked mon.noname-i con 0x7fc75813f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7581cf100 con 0x7fc7580a6ba0 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75813e3d0 con 0x7fc75813f5c0 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc758139b80 con 0x7fc7580a67c0 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc76017f640 10 monclient(hunting): authenticate will time out at 1376.816658s 2024-08-19T10:02:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc75df1b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc7580a67c0 0x7fc7581ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:59116/0 (socket says 172.21.15.1:59116) 2024-08-19T10:02:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.740+0000 7fc75df1b640 1 -- 172.21.15.1:0/1388107414 learned_addr learned my addr 172.21.15.1:0/1388107414 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.742+0000 7fc75df1b640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc7580a67c0 legacy=0x7fc7581ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.742+0000 7fc74e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7580a67c0 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.742+0000 7fc74e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.743+0000 7fc75e71c640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.82:6792/0 conn(0x7fc75813f5c0 legacy=0x7fc7581d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.743+0000 7fc74e7fc640 10 client.?.objecter ms_handle_connect 0x7fc75813f5c0 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.743+0000 7fc74e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.743+0000 7fc74e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7580a6ba0 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.743+0000 7fc74e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:00.743+0000 7fc75d71a640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.82:6791/0 conn(0x7fc7580a6ba0 legacy=0x7fc7581ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3160216992 0 0) 0x7fc758139b80 con 0x7fc7580a67c0 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): my global_id is 184283 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc73c004eb0 con 0x7fc7580a67c0 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2578538632 0 0) 0x7fc73c004eb0 con 0x7fc7580a67c0 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc758139b80 con 0x7fc7580a67c0 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc754004d90 con 0x7fc7580a67c0 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:01.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): dump: 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.189+0000 7fc74e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919406952 0 0) 0x7fc75813e3d0 con 0x7fc75813f5c0 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): my global_id is 145006 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc73c004eb0 con 0x7fc75813f5c0 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106939760 0 0) 0x7fc7581cf100 con 0x7fc7580a6ba0 2024-08-19T10:02:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 10 monclient(hunting): my global_id is 27518 2024-08-19T10:02:01.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.190+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc75813e3d0 con 0x7fc7580a6ba0 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 701050218 0 0) 0x7fc758139b80 con 0x7fc7580a67c0 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.82:6792/0 conn(0x7fc75813f5c0 legacy=0x7fc7581d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.82:6791/0 conn(0x7fc7580a6ba0 legacy=0x7fc7581ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 monclient: found mon.e 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75813bee0 con 0x7fc7580a67c0 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 5 monclient: authenticate success, global_id 184283 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 10 monclient: _renew_subs 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc75813b640 con 0x7fc7580a67c0 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 10 client.184283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 10 monclient: _renew_subs 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc754001a80 con 0x7fc7580a67c0 2024-08-19T10:02:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7580a2890 con 0x7fc7580a67c0 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc76017f640 1 librados: init done 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc754005700 con 0x7fc7580a67c0 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74ffff640 4 set_mon_vals no callback set 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.197+0000 7fc74e7fc640 10 monclient: dump: 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.254 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc754011b60 con 0x7fc7580a67c0 2024-08-19T10:02:01.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc76017f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc758143050 con 0x7fc75809da50 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(420..420 src has 299..420) ==== 5064+0+0 (unknown 2637308940 0 0) 0x7fc75404e1c0 con 0x7fc7580a67c0 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc74e7fc640 10 client.184283.objecter ms_dispatch 0x7fc758000b90 osd_map(420..420 src has 299..420) 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc74e7fc640 3 client.184283.objecter handle_osd_map got epochs [420,420] > 0 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.198+0000 7fc74e7fc640 3 client.184283.objecter handle_osd_map decoding full epoch 420 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.199+0000 7fc74e7fc640 20 client.184283.objecter dump_active .. 0 homeless 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 10 client.184283.objecter ms_handle_connect 0x7fc75809da50 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 10 client.184283.objecter resend_mon_ops 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc75d71a640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc75809da50 legacy=0x7fc75809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432217801 0 0) 0x7fc758143050 con 0x7fc75809da50 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 10 monclient(hunting): my global_id is 184292 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc758139b80 con 0x7fc75809da50 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.201+0000 7fc75e71c640 10 monclient: get_auth_request con 0x7fc73c041150 auth_method 0 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1286197887 0 0) 0x7fc758139b80 con 0x7fc75809da50 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc758143050 con 0x7fc75809da50 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc748001b60 con 0x7fc75809da50 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 10 monclient: dump: 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3842638373 0 0) 0x7fc758143050 con 0x7fc75809da50 2024-08-19T10:02:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc75813bb60 con 0x7fc75809da50 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc7480057c0 con 0x7fc75809da50 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc75809da50 legacy=0x7fc75809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.202+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc75809da50 legacy=0x7fc75809de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.204+0000 7fc74e7fc640 10 client.184283.objecter ms_handle_connect 0x7fc73c041150 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.209+0000 7fc76017f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.209+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc758068f60 con 0x7fc75809e540 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc75d71a640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc75809e540 legacy=0x7fc7581dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 10 client.184283.objecter ms_handle_connect 0x7fc75809e540 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 10 client.184283.objecter resend_mon_ops 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900754425 0 0) 0x7fc758068f60 con 0x7fc75809e540 2024-08-19T10:02:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 10 monclient(hunting): my global_id is 184301 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.211+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc758143050 con 0x7fc75809e540 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3125580803 0 0) 0x7fc758143050 con 0x7fc75809e540 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc758068f60 con 0x7fc75809e540 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc748003a90 con 0x7fc75809e540 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 10 monclient: dump: 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1612714749 0 0) 0x7fc758068f60 con 0x7fc75809e540 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc75809e920 con 0x7fc75809e540 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 3019923629) 0x7fc748005a90 con 0x7fc75809e540 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc75809e540 legacy=0x7fc7581dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc74e7fc640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc75809e540 legacy=0x7fc7581dbcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:01.262 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 38460, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:02:01.263 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.264 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.265 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:01.266 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.267 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:01.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:01.271 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:01.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:01.279 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.281 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.282 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.283 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:02:01.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.283 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.283 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:01.283 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 10 librados: watch_flush enter 2024-08-19T10:02:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 10 librados: watch_flush exit 2024-08-19T10:02:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 20 client.184283.objecter shutdown clearing up homeless session... 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 10 client.184283.objecter successfully canceled tick 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc73c041150 legacy=0x7fc73c043610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 10 monclient: shutdown 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.212+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 >> v1:172.21.15.1:6793/0 conn(0x7fc7580a67c0 legacy=0x7fc7581ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.213+0000 7fc75e71c640 1 -- 172.21.15.1:0/1388107414 reap_dead start 2024-08-19T10:02:01.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.213+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 shutdown_connections 2024-08-19T10:02:01.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.213+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 >> 172.21.15.1:0/1388107414 conn(0x7fc7580ac4b0 msgr2=0x7fc75813eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:01.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.213+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 shutdown_connections 2024-08-19T10:02:01.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.213+0000 7fc76017f640 1 -- 172.21.15.1:0/1388107414 wait complete. 2024-08-19T10:02:01.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.213+0000 7fc76017f640 1 librados: shutdown 2024-08-19T10:02:01.287 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:01.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-08-19T10:02:01.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:02:01.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.388+0000 7f94e8c9f640 10 monclient: get_monmap_and_config 2024-08-19T10:02:01.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.388+0000 7f94e8c9f640 10 monclient: build_initial_monmap 2024-08-19T10:02:01.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.388+0000 7f94e8c9f640 10 monclient: monmap: 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:01.389584+0000 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:01.389584+0000 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:01.390 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:01.391 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 1 Processor -- start 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 1 -- start start 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient: init 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient: _add_conns ranks=[3,4,2,6,8,7,0,1,5] 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient(hunting): picked mon.noname-g con 0x7f94e40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f94e40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f94e413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e40a3b40 con 0x7f94e413f5c0 2024-08-19T10:02:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e40a8130 con 0x7f94e40a67c0 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e406ede0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e8c9f640 10 monclient(hunting): authenticate will time out at 1377.465804s 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94d9d9b640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f94e40a6ba0 legacy=0x7f94e40a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e2d9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f94e413f5c0 0x7f94e41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33032/0 (socket says 172.21.15.1:33032) 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.389+0000 7f94e2d9d640 1 -- 172.21.15.1:0/2663849814 learned_addr learned my addr 172.21.15.1:0/2663849814 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e2d9d640 1 -- 172.21.15.1:0/2663849814 >> v1:172.21.15.1:6790/0 conn(0x7f94e413f5c0 legacy=0x7f94e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e259c640 1 -- 172.21.15.1:0/2663849814 >> v1:172.21.15.82:6790/0 conn(0x7f94e40a67c0 legacy=0x7f94e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 856482979 0 0) 0x7f94e40a3b40 con 0x7f94e413f5c0 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient(hunting): my global_id is 184241 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94d0003520 con 0x7f94e413f5c0 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4208840511 0 0) 0x7f94d0003520 con 0x7f94e413f5c0 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 >> v1:172.21.15.1:6791/0 conn(0x7f94e40a6ba0 legacy=0x7f94e40a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 >> v1:172.21.15.82:6790/0 conn(0x7f94e40a67c0 legacy=0x7f94e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 monclient: found mon.noname-b 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e40a92a0 con 0x7f94e413f5c0 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: _finish_auth 0 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94dc001e60 con 0x7f94e413f5c0 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e8c9f640 5 monclient: authenticate success, global_id 184241 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e8c9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: dump: 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.399 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e8c9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94dc004c20 con 0x7f94e413f5c0 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e8c9f640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 1 -- 172.21.15.1:0/2663849814 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94dc005190 con 0x7f94e413f5c0 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e8c9f640 4 set_mon_vals no callback set 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.392+0000 7f94e1d9b640 10 monclient: dump: 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 10 monclient: shutdown 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 1 -- 172.21.15.1:0/2663849814 >> v1:172.21.15.1:6790/0 conn(0x7f94e413f5c0 legacy=0x7f94e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 1 -- 172.21.15.1:0/2663849814 shutdown_connections 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 1 -- 172.21.15.1:0/2663849814 >> 172.21.15.1:0/2663849814 conn(0x7f94e40ac4b0 msgr2=0x7f94e40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 1 -- 172.21.15.1:0/2663849814 shutdown_connections 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 1 -- 172.21.15.1:0/2663849814 wait complete. 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 10 monclient: build_initial_monmap 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 10 monclient: monmap: 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.393+0000 7f94e8c9f640 1 librados: starting msgr at 2024-08-19T10:02:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 librados: starting objecter 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 Processor -- start 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 -- start start 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 librados: setting wanted keys 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 librados: calling monclient init 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient: init 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient: _add_conns ranks=[3,0,4,7,2,5,6,8,1] 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient(hunting): picked mon.noname-g con 0x7f94e40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f94e40a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f94e413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e40a9cc0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e4142b10 con 0x7f94e40a67c0 2024-08-19T10:02:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e413a850 con 0x7f94e413f5c0 2024-08-19T10:02:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e8c9f640 10 monclient(hunting): authenticate will time out at 1377.470773s 2024-08-19T10:02:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94e2d9d640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f94e413f5c0 legacy=0x7f94e41d6b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:02:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94d9d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f94e40a6ba0 0x7f94e41d3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39778/0 (socket says 172.21.15.1:39778) 2024-08-19T10:02:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.394+0000 7f94d9d9b640 1 -- 172.21.15.1:0/4204834075 learned_addr learned my addr 172.21.15.1:0/4204834075 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94d9d9b640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.1:6789/0 conn(0x7f94e40a6ba0 legacy=0x7f94e41d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 client.?.objecter ms_handle_connect 0x7f94e40a6ba0 2024-08-19T10:02:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 client.?.objecter ms_handle_connect 0x7f94e40a67c0 2024-08-19T10:02:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e259c640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6790/0 conn(0x7f94e40a67c0 legacy=0x7f94e41ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348822 0 0) 0x7f94cc003dd0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): my global_id is 174573 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94c4004ee0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385106918 0 0) 0x7f94e40a8130 con 0x7f94e40a67c0 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): my global_id is 184278 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94cc003dd0 con 0x7f94e40a67c0 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 613026705 0 0) 0x7f94c4004ee0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94e40a8130 con 0x7f94e40a6ba0 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94cc0048b0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): dump: 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 613302136 0 0) 0x7f94cc003dd0 con 0x7f94e40a67c0 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94c4004ee0 con 0x7f94e40a67c0 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94d4003610 con 0x7f94e40a67c0 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient(hunting): dump: 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1756130738 0 0) 0x7f94e40a8130 con 0x7f94e40a6ba0 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.1:6791/0 conn(0x7f94e413f5c0 legacy=0x7f94e41d6b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6790/0 conn(0x7f94e40a67c0 legacy=0x7f94e41ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 monclient: found mon.a 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:02:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e413ab30 con 0x7f94e40a6ba0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: _finish_auth 0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 5 monclient: authenticate success, global_id 174573 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 10 monclient: _renew_subs 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f94e40a17a0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 10 client.174573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 10 monclient: _renew_subs 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f94e40a0e80 con 0x7f94e40a6ba0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e8c9f640 1 librados: init done 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94cc004ad0 con 0x7f94e40a6ba0 2024-08-19T10:02:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94cc005000 con 0x7f94e40a6ba0 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94e159a640 4 set_mon_vals no callback set 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.397+0000 7f94db7fe640 10 monclient: dump: 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.398+0000 7f94e8c9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.398+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f94c80065c0 con 0x7f94c8002860 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.398+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f94cc005310 con 0x7f94e40a6ba0 2024-08-19T10:02:01.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.399+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(420..420 src has 299..420) ==== 5064+0+0 (unknown 2637308940 0 0) 0x7f94cc04e340 con 0x7f94e40a6ba0 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.399+0000 7f94db7fe640 10 client.174573.objecter ms_dispatch 0x7f94e4000b90 osd_map(420..420 src has 299..420) 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.399+0000 7f94db7fe640 3 client.174573.objecter handle_osd_map got epochs [420,420] > 0 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.399+0000 7f94db7fe640 3 client.174573.objecter handle_osd_map decoding full epoch 420 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.399+0000 7f94db7fe640 20 client.174573.objecter dump_active .. 0 homeless 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.399+0000 7f94e2d9d640 10 monclient: get_auth_request con 0x7f94c4041420 auth_method 0 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.400+0000 7f94db7fe640 10 client.174573.objecter ms_handle_connect 0x7f94c8002860 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.400+0000 7f94e259c640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8002860 legacy=0x7f94c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.400+0000 7f94db7fe640 10 client.174573.objecter resend_mon_ops 2024-08-19T10:02:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.401+0000 7f94db7fe640 10 client.174573.objecter ms_handle_connect 0x7f94c4041420 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 81040705 0 0) 0x7f94c80065c0 con 0x7f94c8002860 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient(hunting): my global_id is 174502 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94e40a8130 con 0x7f94c8002860 2024-08-19T10:02:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2816284966 0 0) 0x7f94e40a8130 con 0x7f94c8002860 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94c80065c0 con 0x7f94c8002860 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94d4003e10 con 0x7f94c8002860 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 10 monclient: dump: 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1937320648 0 0) 0x7f94c80065c0 con 0x7f94c8002860 2024-08-19T10:02:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.678+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94c800a160 con 0x7f94c8002860 2024-08-19T10:02:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.679+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f94d4004080 con 0x7f94c8002860 2024-08-19T10:02:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.679+0000 7f94db7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.679+0000 7f94db7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.679+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8002860 legacy=0x7f94c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.679+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8002860 legacy=0x7f94c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.685+0000 7f94e8c9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:01.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.685+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f94c8009e70 con 0x7f94c8009130 2024-08-19T10:02:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94e259c640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 legacy=0x7f94c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 10 client.174573.objecter ms_handle_connect 0x7f94c8009130 2024-08-19T10:02:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 10 client.174573.objecter resend_mon_ops 2024-08-19T10:02:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378924968 0 0) 0x7f94c8009e70 con 0x7f94c8009130 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 10 monclient(hunting): my global_id is 174511 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94c80065c0 con 0x7f94c8009130 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94e259c640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 legacy=0x7f94c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94e259c640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 legacy=0x7f94c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94e259c640 1 --1- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 0x7f94c80095c0 :-1 s=OPENED pgs=3525 cs=1 l=1).handle_message read tag failed 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94e259c640 1 --1- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 0x7f94c80095c0 :-1 s=OPENED pgs=3525 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 legacy=0x7f94c80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c8009130 legacy=0x7f94c80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f94c8009e70 con 0x7f94c404bab0 2024-08-19T10:02:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.688+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 33087674 0 0) 0x7f94c80065c0 con 0x7f94c8009130 2024-08-19T10:02:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.690+0000 7f94e259c640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c404bab0 legacy=0x7f94c404a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.690+0000 7f94db7fe640 10 client.174573.objecter ms_handle_connect 0x7f94c404bab0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.690+0000 7f94db7fe640 10 client.174573.objecter resend_mon_ops 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104395079 0 0) 0x7f94c8009e70 con 0x7f94c404bab0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient(hunting): my global_id is 174520 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94c80065c0 con 0x7f94c404bab0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2256605875 0 0) 0x7f94c80065c0 con 0x7f94c404bab0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94c8009e70 con 0x7f94c404bab0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94d4005cf0 con 0x7f94c404bab0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:02:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient: dump: 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 656982931 0 0) 0x7f94c8009e70 con 0x7f94c404bab0 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94d4004080 con 0x7f94c404bab0 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7830 (unknown 0 0 2669289043) 0x7f94d4007ea0 con 0x7f94c404bab0 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c404bab0 legacy=0x7f94c404a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94db7fe640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6789/0 conn(0x7f94c404bab0 legacy=0x7f94c404a030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.691+0000 7f94e2d9d640 1 -- 172.21.15.1:0/4204834075 reap_dead start 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 171750, 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:01.696 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.697 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.698 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.701 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.702 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:01.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.707 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:01.711 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.715 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:01.717 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 10 librados: watch_flush enter 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 10 librados: watch_flush exit 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 20 client.174573.objecter shutdown clearing up homeless session... 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 10 client.174573.objecter successfully canceled tick 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.82:6812/2549104415 conn(0x7f94c4041420 legacy=0x7f94c40438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 10 monclient: shutdown 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 >> v1:172.21.15.1:6789/0 conn(0x7f94e40a6ba0 legacy=0x7f94e41d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 shutdown_connections 2024-08-19T10:02:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 >> 172.21.15.1:0/4204834075 conn(0x7f94e40ac4b0 msgr2=0x7f94e40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:01.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 shutdown_connections 2024-08-19T10:02:01.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 -- 172.21.15.1:0/4204834075 wait complete. 2024-08-19T10:02:01.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.692+0000 7f94e8c9f640 1 librados: shutdown 2024-08-19T10:02:01.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:01.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-08-19T10:02:01.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:02:01.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.820+0000 7f0595914640 10 monclient: get_monmap_and_config 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.820+0000 7f0595914640 10 monclient: build_initial_monmap 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.820+0000 7f0595914640 10 monclient: monmap: 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:01.821658+0000 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:01.821658+0000 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:01.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:01.823 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 1 Processor -- start 2024-08-19T10:02:01.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 1 -- start start 2024-08-19T10:02:01.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient: init 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient: _add_conns ranks=[4,0,8,3,1,7,5,2,6] 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient(hunting): picked mon.noname-c con 0x7f05900a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient(hunting): picked mon.noname-a con 0x7f05900a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient(hunting): picked mon.noname-e con 0x7f059013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a3b40 con 0x7f05900a6ba0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a8130 con 0x7f05900a67c0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059006ede0 con 0x7f059013f5c0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.821+0000 7f0595914640 10 monclient(hunting): authenticate will time out at 1377.897991s 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.822+0000 7f058effd640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f05900a67c0 legacy=0x7f05900a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.822+0000 7f058f7fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f059013f5c0 0x7f05901419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:59146/0 (socket says 172.21.15.1:59146) 2024-08-19T10:02:01.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.822+0000 7f0587fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f05900a6ba0 0x7f05900a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39784/0 (socket says 172.21.15.1:39784) 2024-08-19T10:02:01.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.822+0000 7f058f7fe640 1 -- 172.21.15.1:0/525682255 learned_addr learned my addr 172.21.15.1:0/525682255 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058f7fe640 1 -- 172.21.15.1:0/525682255 >> v1:172.21.15.1:6793/0 conn(0x7f059013f5c0 legacy=0x7f05901419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f0587fff640 1 -- 172.21.15.1:0/525682255 >> v1:172.21.15.1:6789/0 conn(0x7f05900a6ba0 legacy=0x7f05900a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807889770 0 0) 0x7f059006ede0 con 0x7f059013f5c0 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): my global_id is 184310 2024-08-19T10:02:01.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f057c003520 con 0x7f059013f5c0 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259059408 0 0) 0x7f05900a3b40 con 0x7f05900a6ba0 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient(hunting): my global_id is 174582 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f059006ede0 con 0x7f05900a6ba0 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4188125848 0 0) 0x7f057c003520 con 0x7f059013f5c0 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 >> v1:172.21.15.1:6791/0 conn(0x7f05900a67c0 legacy=0x7f05900a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 >> v1:172.21.15.1:6789/0 conn(0x7f05900a6ba0 legacy=0x7f05900a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 monclient: found mon.noname-e 2024-08-19T10:02:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05900a92a0 con 0x7f059013f5c0 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0588001e50 con 0x7f059013f5c0 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f0595914640 5 monclient: authenticate success, global_id 184310 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f0595914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: dump: 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.832 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f0595914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0588004c20 con 0x7f059013f5c0 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 1 -- 172.21.15.1:0/525682255 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0588005150 con 0x7f059013f5c0 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f0595914640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f0595914640 4 set_mon_vals no callback set 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.824+0000 7f058e7fc640 10 monclient: dump: 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 10 monclient: shutdown 2024-08-19T10:02:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 1 -- 172.21.15.1:0/525682255 >> v1:172.21.15.1:6793/0 conn(0x7f059013f5c0 legacy=0x7f05901419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 1 -- 172.21.15.1:0/525682255 shutdown_connections 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 1 -- 172.21.15.1:0/525682255 >> 172.21.15.1:0/525682255 conn(0x7f05900ac4b0 msgr2=0x7f05900a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 1 -- 172.21.15.1:0/525682255 shutdown_connections 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 1 -- 172.21.15.1:0/525682255 wait complete. 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 10 monclient: build_initial_monmap 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.825+0000 7f0595914640 10 monclient: monmap: 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 librados: starting msgr at 2024-08-19T10:02:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 librados: starting objecter 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 Processor -- start 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 -- start start 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 librados: setting wanted keys 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 librados: calling monclient init 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient: init 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient: _add_conns ranks=[2,7,4,1,5,0,6,8,3] 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient(hunting): picked mon.noname-b con 0x7f05900a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient(hunting): picked mon.noname-i con 0x7f05900a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient(hunting): picked mon.noname-c con 0x7f059013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059013a510 con 0x7f05900a67c0 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059013a7f0 con 0x7f059013f5c0 2024-08-19T10:02:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a2250 con 0x7f05900a6ba0 2024-08-19T10:02:01.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:01.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f058f7fe640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f059013f5c0 legacy=0x7f05900a0a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:02:01.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f0595914640 10 monclient(hunting): authenticate will time out at 1377.902814s 2024-08-19T10:02:01.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f058effd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f05900a67c0 0x7f059009fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33034/0 (socket says 172.21.15.1:33034) 2024-08-19T10:02:01.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.826+0000 7f058effd640 1 -- 172.21.15.1:0/2338852997 learned_addr learned my addr 172.21.15.1:0/2338852997 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058effd640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.1:6790/0 conn(0x7f05900a67c0 legacy=0x7f059009fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 client.?.objecter ms_handle_connect 0x7f05900a67c0 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001354008 0 0) 0x7f059013a510 con 0x7f05900a67c0 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): my global_id is 184250 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0570004eb0 con 0x7f05900a67c0 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 client.?.objecter ms_handle_connect 0x7f05900a6ba0 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0587fff640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6792/0 conn(0x7f05900a6ba0 legacy=0x7f05900a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1948325444 0 0) 0x7f0570004eb0 con 0x7f05900a67c0 2024-08-19T10:02:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f059013a510 con 0x7f05900a67c0 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2207201709 0 0) 0x7f05900a2250 con 0x7f05900a6ba0 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): my global_id is 145015 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0570004eb0 con 0x7f05900a6ba0 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0580004760 con 0x7f05900a67c0 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): dump: 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.845 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1303870719 0 0) 0x7f059013a510 con 0x7f05900a67c0 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6792/0 conn(0x7f05900a6ba0 legacy=0x7f05900a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.1:6791/0 conn(0x7f059013f5c0 legacy=0x7f05900a0a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: _finish_hunting 0 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 monclient: found mon.b 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0590139a20 con 0x7f05900a67c0 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: _finish_auth 0 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: _check_auth_tickets 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 5 monclient: authenticate success, global_id 184250 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 10 monclient: _renew_subs 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05900a16f0 con 0x7f05900a67c0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 10 client.184250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 10 monclient: _renew_subs 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f05901d7260 con 0x7f05900a67c0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f0595914640 1 librados: init done 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0580004900 con 0x7f05900a67c0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0580004e70 con 0x7f05900a67c0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058dffb640 4 set_mon_vals no callback set 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.829+0000 7f058cff9640 10 monclient: dump: 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f0595914640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f05740065c0 con 0x7f0574002860 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0580005180 con 0x7f05900a67c0 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(420..420 src has 299..420) ==== 5064+0+0 (unknown 2637308940 0 0) 0x7f058004e1b0 con 0x7f05900a67c0 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058cff9640 10 client.184250.objecter ms_dispatch 0x7f0590000b90 osd_map(420..420 src has 299..420) 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058cff9640 3 client.184250.objecter handle_osd_map got epochs [420,420] > 0 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058cff9640 3 client.184250.objecter handle_osd_map decoding full epoch 420 2024-08-19T10:02:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058cff9640 20 client.184250.objecter dump_active .. 0 homeless 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.830+0000 7f058f7fe640 10 monclient: get_auth_request con 0x7f0570040f90 auth_method 0 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.832+0000 7f0587fff640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6790/0 conn(0x7f0574002860 legacy=0x7f0574004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.832+0000 7f058cff9640 10 client.184250.objecter ms_handle_connect 0x7f0574002860 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.832+0000 7f058cff9640 10 client.184250.objecter resend_mon_ops 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865194288 0 0) 0x7f05740065c0 con 0x7f0574002860 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient(hunting): my global_id is 184287 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f059013a510 con 0x7f0574002860 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 client.184250.objecter ms_handle_connect 0x7f0570040f90 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 187672556 0 0) 0x7f059013a510 con 0x7f0574002860 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05740065c0 con 0x7f0574002860 2024-08-19T10:02:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0578001a20 con 0x7f0574002860 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient: dump: 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3987422986 0 0) 0x7f05740065c0 con 0x7f0574002860 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f057400a160 con 0x7f0574002860 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0578006840 con 0x7f0574002860 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6790/0 conn(0x7f0574002860 legacy=0x7f0574004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.833+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6790/0 conn(0x7f0574002860 legacy=0x7f0574004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.840+0000 7f0595914640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:01.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.840+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0574009e70 con 0x7f0574009130 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f0587fff640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6790/0 conn(0x7f0574009130 legacy=0x7f05740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 10 client.184250.objecter ms_handle_connect 0x7f0574009130 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 10 client.184250.objecter resend_mon_ops 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270663472 0 0) 0x7f0574009e70 con 0x7f0574009130 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 10 monclient(hunting): my global_id is 184296 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.842+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05740065c0 con 0x7f0574009130 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2339184626 0 0) 0x7f05740065c0 con 0x7f0574009130 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0574009e70 con 0x7f0574009130 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0578004f10 con 0x7f0574009130 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 10 monclient: dump: 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.855 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1747830075 0 0) 0x7f0574009e70 con 0x7f0574009130 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f057400c3f0 con 0x7f0574009130 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 230932765) 0x7f0578006350 con 0x7f0574009130 2024-08-19T10:02:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6790/0 conn(0x7f0574009130 legacy=0x7f05740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058cff9640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6790/0 conn(0x7f0574009130 legacy=0x7f05740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39098, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:01.877 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:01.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 10 librados: watch_flush enter 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 10 librados: watch_flush exit 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 20 client.184250.objecter shutdown clearing up homeless session... 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 10 client.184250.objecter successfully canceled tick 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0570040f90 legacy=0x7f0570043450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 10 monclient: shutdown 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 >> v1:172.21.15.1:6790/0 conn(0x7f05900a67c0 legacy=0x7f059009fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.843+0000 7f058f7fe640 1 -- 172.21.15.1:0/2338852997 reap_dead start 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.844+0000 7f058f7fe640 1 -- 172.21.15.1:0/2338852997 reap_dead start 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.844+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 shutdown_connections 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.844+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 >> 172.21.15.1:0/2338852997 conn(0x7f05900ac4b0 msgr2=0x7f05900a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.844+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 shutdown_connections 2024-08-19T10:02:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.844+0000 7f0595914640 1 -- 172.21.15.1:0/2338852997 wait complete. 2024-08-19T10:02:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.844+0000 7f0595914640 1 librados: shutdown 2024-08-19T10:02:01.880 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:01.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-08-19T10:02:01.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:02:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.979+0000 7fbf5d3cf640 10 monclient: get_monmap_and_config 2024-08-19T10:02:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.979+0000 7fbf5d3cf640 10 monclient: build_initial_monmap 2024-08-19T10:02:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.979+0000 7fbf5d3cf640 10 monclient: monmap: 2024-08-19T10:02:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:01.980049+0000 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:01.980049+0000 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:01.982 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.979+0000 7fbf5d3cf640 1 Processor -- start 2024-08-19T10:02:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.979+0000 7fbf5d3cf640 1 -- start start 2024-08-19T10:02:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.979+0000 7fbf5d3cf640 10 monclient: init 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient: _add_conns ranks=[7,8,0,6,2,3,1,4,5] 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient(hunting): picked mon.noname-i con 0x7fbf580a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient(hunting): picked mon.noname-e con 0x7fbf580694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbf580a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf580a3c70 con 0x7fbf580a46a0 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf580a33d0 con 0x7fbf580a6830 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf580a36b0 con 0x7fbf580694b0 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf5d3cf640 10 monclient(hunting): authenticate will time out at 1378.056283s 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf4e7fc640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbf580694b0 0x7fbf580a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:59158/0 (socket says 172.21.15.1:59158) 2024-08-19T10:02:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf577fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbf580a46a0 0x7fbf58141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39796/0 (socket says 172.21.15.1:39796) 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.980+0000 7fbf4e7fc640 1 -- 172.21.15.1:0/2811566292 learned_addr learned my addr 172.21.15.1:0/2811566292 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf4e7fc640 1 -- 172.21.15.1:0/2811566292 >> v1:172.21.15.1:6793/0 conn(0x7fbf580694b0 legacy=0x7fbf580a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf577fe640 1 -- 172.21.15.1:0/2811566292 >> v1:172.21.15.1:6789/0 conn(0x7fbf580a46a0 legacy=0x7fbf58141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255010975 0 0) 0x7fbf580a3c70 con 0x7fbf580a46a0 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): my global_id is 174591 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf44003520 con 0x7fbf580a46a0 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf56ffd640 1 -- 172.21.15.1:0/2811566292 >> v1:172.21.15.82:6792/0 conn(0x7fbf580a6830 legacy=0x7fbf580a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726440633 0 0) 0x7fbf580a36b0 con 0x7fbf580694b0 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): my global_id is 184319 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf580a3c70 con 0x7fbf580694b0 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870975676 0 0) 0x7fbf580a33d0 con 0x7fbf580a6830 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _negotiate 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient(hunting): my global_id is 145024 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf580a36b0 con 0x7fbf580a6830 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 324590424 0 0) 0x7fbf44003520 con 0x7fbf580a46a0 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 >> v1:172.21.15.1:6793/0 conn(0x7fbf580694b0 legacy=0x7fbf580a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 >> v1:172.21.15.82:6792/0 conn(0x7fbf580a6830 legacy=0x7fbf580a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient: _finish_hunting 0 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 monclient: found mon.noname-a 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.982+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5806dbb0 con 0x7fbf580a46a0 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: _finish_auth 0 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: _check_auth_tickets 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf50001e60 con 0x7fbf580a46a0 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 5 monclient: authenticate success, global_id 174591 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:02:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: dump: 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf50004b60 con 0x7fbf580a46a0 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 1 -- 172.21.15.1:0/2811566292 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf500050d0 con 0x7fbf580a46a0 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf567fc640 10 monclient: dump: 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 4 set_mon_vals no callback set 2024-08-19T10:02:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 10 monclient: shutdown 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/2811566292 >> v1:172.21.15.1:6789/0 conn(0x7fbf580a46a0 legacy=0x7fbf58141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/2811566292 shutdown_connections 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/2811566292 >> 172.21.15.1:0/2811566292 conn(0x7fbf580a9020 msgr2=0x7fbf58068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/2811566292 shutdown_connections 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.983+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/2811566292 wait complete. 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 10 monclient: build_initial_monmap 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 10 monclient: monmap: 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 1 librados: starting msgr at 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 1 librados: starting objecter 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 1 Processor -- start 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 1 -- start start 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 1 librados: setting wanted keys 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 1 librados: calling monclient init 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.984+0000 7fbf5d3cf640 10 monclient: init 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient: _add_conns ranks=[5,8,3,2,7,0,6,4,1] 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient(hunting): picked mon.noname-h con 0x7fbf580694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient(hunting): picked mon.noname-e con 0x7fbf580a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient(hunting): picked mon.noname-g con 0x7fbf580a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf58136b50 con 0x7fbf580a6830 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf580a9d30 con 0x7fbf580694b0 2024-08-19T10:02:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf58138eb0 con 0x7fbf580a46a0 2024-08-19T10:02:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf5d3cf640 10 monclient(hunting): authenticate will time out at 1378.061310s 2024-08-19T10:02:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf4e7fc640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbf580a46a0 0x7fbf58135c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:59168/0 (socket says 172.21.15.1:59168) 2024-08-19T10:02:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.985+0000 7fbf4e7fc640 1 -- 172.21.15.1:0/1570401049 learned_addr learned my addr 172.21.15.1:0/1570401049 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4e7fc640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.1:6793/0 conn(0x7fbf580a46a0 legacy=0x7fbf58135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 client.?.objecter ms_handle_connect 0x7fbf580a46a0 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 client.?.objecter ms_handle_connect 0x7fbf580694b0 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf56ffd640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf580694b0 legacy=0x7fbf58135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf577fe640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6790/0 conn(0x7fbf580a6830 legacy=0x7fbf581d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464303713 0 0) 0x7fbf58138eb0 con 0x7fbf580a46a0 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): my global_id is 184328 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf38004eb0 con 0x7fbf580a46a0 2024-08-19T10:02:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 client.?.objecter ms_handle_connect 0x7fbf580a6830 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676011877 0 0) 0x7fbf580a36b0 con 0x7fbf580694b0 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): my global_id is 27527 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf58138eb0 con 0x7fbf580694b0 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49518103 0 0) 0x7fbf58136b50 con 0x7fbf580a6830 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): my global_id is 184305 2024-08-19T10:02:02.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf580a36b0 con 0x7fbf580a6830 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3971047404 0 0) 0x7fbf38004eb0 con 0x7fbf580a46a0 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf58136b50 con 0x7fbf580a46a0 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf40001d30 con 0x7fbf580a46a0 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): dump: 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.987+0000 7fbf4ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2116951257 0 0) 0x7fbf58136b50 con 0x7fbf580a46a0 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf580694b0 legacy=0x7fbf58135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6790/0 conn(0x7fbf580a6830 legacy=0x7fbf581d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 monclient: found mon.e 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf58137980 con 0x7fbf580a46a0 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: _finish_auth 0 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 5 monclient: authenticate success, global_id 184328 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 10 monclient: _renew_subs 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf581376b0 con 0x7fbf580a46a0 2024-08-19T10:02:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 10 client.184328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 10 monclient: _renew_subs 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf58136580 con 0x7fbf580a46a0 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf5d3cf640 1 librados: init done 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf40004dd0 con 0x7fbf580a46a0 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf400052c0 con 0x7fbf580a46a0 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf55ffb640 4 set_mon_vals no callback set 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 10 monclient: dump: 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.007 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.008 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.988+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbf40011720 con 0x7fbf580a46a0 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf5d3cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf3c0065c0 con 0x7fbf3c002860 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(420..420 src has 299..420) ==== 5064+0+0 (unknown 2637308940 0 0) 0x7fbf4004e130 con 0x7fbf580a46a0 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf4ffff640 10 client.184328.objecter ms_dispatch 0x7fbf58000b90 osd_map(420..420 src has 299..420) 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf4ffff640 3 client.184328.objecter handle_osd_map got epochs [420,420] > 0 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf4ffff640 3 client.184328.objecter handle_osd_map decoding full epoch 420 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf4ffff640 20 client.184328.objecter dump_active .. 0 homeless 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.989+0000 7fbf577fe640 10 monclient: get_auth_request con 0x7fbf38041250 auth_method 0 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 client.184328.objecter ms_handle_connect 0x7fbf3c002860 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf56ffd640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf3c002860 legacy=0x7fbf3c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 client.184328.objecter resend_mon_ops 2024-08-19T10:02:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268273367 0 0) 0x7fbf3c0065c0 con 0x7fbf3c002860 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient(hunting): my global_id is 27536 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf58136b50 con 0x7fbf3c002860 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 client.184328.objecter ms_handle_connect 0x7fbf38041250 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2732778956 0 0) 0x7fbf58136b50 con 0x7fbf3c002860 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf3c0065c0 con 0x7fbf3c002860 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf48003950 con 0x7fbf3c002860 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.991+0000 7fbf4ffff640 10 monclient: dump: 2024-08-19T10:02:02.010 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.011 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 757815425 0 0) 0x7fbf3c0065c0 con 0x7fbf3c002860 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf3c00a160 con 0x7fbf3c002860 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbf480057c0 con 0x7fbf3c002860 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf3c002860 legacy=0x7fbf3c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.992+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf3c002860 legacy=0x7fbf3c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.998+0000 7fbf5d3cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:01.998+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf3c009e50 con 0x7fbf3c009110 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf56ffd640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf3c009110 legacy=0x7fbf3c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 client.184328.objecter ms_handle_connect 0x7fbf3c009110 2024-08-19T10:02:02.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 client.184328.objecter resend_mon_ops 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819527310 0 0) 0x7fbf3c009e50 con 0x7fbf3c009110 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient(hunting): my global_id is 27545 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf3c0065c0 con 0x7fbf3c009110 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3954112869 0 0) 0x7fbf3c0065c0 con 0x7fbf3c009110 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf3c009e50 con 0x7fbf3c009110 2024-08-19T10:02:02.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf48004130 con 0x7fbf3c009110 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 10 monclient: dump: 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.015 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3642598441 0 0) 0x7fbf3c009e50 con 0x7fbf3c009110 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.001+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf3c00c3f0 con 0x7fbf3c009110 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7663 (unknown 0 0 827731269) 0x7fbf48004190 con 0x7fbf3c009110 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf4ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf3c009110 legacy=0x7fbf3c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf4ffff640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6791/0 conn(0x7fbf3c009110 legacy=0x7fbf3c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 728267, 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:02.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:02.018 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:02.020 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:02.021 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.022 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:02.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:02.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:02.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:02.034 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:02:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.037 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.037 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:02.037 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 10 librados: watch_flush enter 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 10 librados: watch_flush exit 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 20 client.184328.objecter shutdown clearing up homeless session... 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 10 client.184328.objecter successfully canceled tick 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbf38041250 legacy=0x7fbf38043710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 10 monclient: shutdown 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 >> v1:172.21.15.1:6793/0 conn(0x7fbf580a46a0 legacy=0x7fbf58135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf577fe640 1 -- 172.21.15.1:0/1570401049 reap_dead start 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf577fe640 1 -- 172.21.15.1:0/1570401049 reap_dead start 2024-08-19T10:02:02.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 shutdown_connections 2024-08-19T10:02:02.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 >> 172.21.15.1:0/1570401049 conn(0x7fbf580a9020 msgr2=0x7fbf580a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:02.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 shutdown_connections 2024-08-19T10:02:02.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 -- 172.21.15.1:0/1570401049 wait complete. 2024-08-19T10:02:02.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.002+0000 7fbf5d3cf640 1 librados: shutdown 2024-08-19T10:02:02.040 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:02.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-08-19T10:02:02.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.138+0000 7f559256b640 10 monclient: get_monmap_and_config 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.138+0000 7f559256b640 10 monclient: build_initial_monmap 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.139+0000 7f559256b640 10 monclient: monmap: 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:02:02.139910+0000 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:02:02.139910+0000 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:02.142 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.139+0000 7f559256b640 1 Processor -- start 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.139+0000 7f559256b640 1 -- start start 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.139+0000 7f559256b640 10 monclient: init 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient: _add_conns ranks=[6,7,3,2,4,1,0,8,5] 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient(hunting): picked mon.noname-d con 0x7f558c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient(hunting): picked mon.noname-i con 0x7f558c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient(hunting): picked mon.noname-g con 0x7f558c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:02:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c0a3b40 con 0x7f558c13f5c0 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c0a8130 con 0x7f558c0a67c0 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c06ede0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f559256b640 10 monclient(hunting): authenticate will time out at 1378.216296s 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f558bfff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f558c0a67c0 legacy=0x7f558c0a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f5590b08640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f558c13f5c0 0x7f558c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:58798/0 (socket says 172.21.15.1:58798) 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f5590b08640 1 -- 172.21.15.1:0/3114907790 learned_addr learned my addr 172.21.15.1:0/3114907790 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.140+0000 7f558b7fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f558c0a6ba0 0x7f558c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:45566/0 (socket says 172.21.15.1:45566) 2024-08-19T10:02:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558b7fe640 1 -- 172.21.15.1:0/3114907790 >> v1:172.21.15.82:6792/0 conn(0x7f558c0a6ba0 legacy=0x7f558c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f5590b08640 1 -- 172.21.15.1:0/3114907790 >> v1:172.21.15.82:6790/0 conn(0x7f558c13f5c0 legacy=0x7f558c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303500474 0 0) 0x7f558c06ede0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): my global_id is 145033 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5574003520 con 0x7f558c0a6ba0 2024-08-19T10:02:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884393507 0 0) 0x7f558c0a3b40 con 0x7f558c13f5c0 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 10 monclient(hunting): my global_id is 184314 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.142+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f558c06ede0 con 0x7f558c13f5c0 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 24837412 0 0) 0x7f5574003520 con 0x7f558c0a6ba0 2024-08-19T10:02:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 >> v1:172.21.15.1:6792/0 conn(0x7f558c0a67c0 legacy=0x7f558c0a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 >> v1:172.21.15.82:6790/0 conn(0x7f558c13f5c0 legacy=0x7f558c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: _finish_hunting 0 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 monclient: found mon.noname-i 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c0a92a0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: _finish_auth 0 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: _check_auth_tickets 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5580001ad0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 5 monclient: authenticate success, global_id 145033 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: dump: 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55800047e0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 10 monclient: get_monmap_and_config success 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 4 set_mon_vals no callback set 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 1 -- 172.21.15.1:0/3114907790 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5580004d50 con 0x7f558c0a6ba0 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f558affd640 10 monclient: dump: 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 10 monclient: shutdown 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 1 -- 172.21.15.1:0/3114907790 >> v1:172.21.15.82:6792/0 conn(0x7f558c0a6ba0 legacy=0x7f558c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 1 -- 172.21.15.1:0/3114907790 shutdown_connections 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 1 -- 172.21.15.1:0/3114907790 >> 172.21.15.1:0/3114907790 conn(0x7f558c0ac4b0 msgr2=0x7f558c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 1 -- 172.21.15.1:0/3114907790 shutdown_connections 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.143+0000 7f559256b640 1 -- 172.21.15.1:0/3114907790 wait complete. 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 10 monclient: build_initial_monmap 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 10 monclient: monmap: 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:02:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:02:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 1 librados: starting msgr at 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 1 librados: starting objecter 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 1 Processor -- start 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 1 -- start start 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 1 librados: setting wanted keys 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 1 librados: calling monclient init 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.144+0000 7f559256b640 10 monclient: init 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient: _reopen_session rank -1 2024-08-19T10:02:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient: _add_conns ranks=[4,1,5,7,3,6,2,0,8] 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient(hunting): picked mon.noname-c con 0x7f558c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient(hunting): picked mon.noname-f con 0x7f558c0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient(hunting): picked mon.noname-h con 0x7f558c13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f558bfff640 1 -- >> v1:172.21.15.1:6791/0 conn(0x7f558c0a67c0 legacy=0x7f558c13ebb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6791/0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c0a14e0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c0a9fa0 con 0x7f558c0a67c0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c0a5510 con 0x7f558c13f5c0 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient(hunting): _renew_subs 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f559256b640 10 monclient(hunting): authenticate will time out at 1378.221426s 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f558b7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f558c0a6ba0 0x7f558c1d2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:34750/0 (socket says 172.21.15.1:34750) 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f5590b08640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f558c13f5c0 0x7f558c1d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:41640/0 (socket says 172.21.15.1:41640) 2024-08-19T10:02:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.145+0000 7f558b7fe640 1 -- 172.21.15.1:0/172736690 learned_addr learned my addr 172.21.15.1:0/172736690 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:02:02.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5590b08640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6791/0 conn(0x7f558c13f5c0 legacy=0x7f558c1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 client.?.objecter ms_handle_connect 0x7f558c13f5c0 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f558b7fe640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6789/0 conn(0x7f558c0a6ba0 legacy=0x7f558c1d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 client.?.objecter ms_handle_connect 0x7f558c0a6ba0 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1445537221 0 0) 0x7f558c0a5510 con 0x7f558c13f5c0 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): my global_id is 27554 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5578004eb0 con 0x7f558c13f5c0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 290266388 0 0) 0x7f558c0a14e0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 10 monclient(hunting): my global_id is 174529 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.147+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f558c0a5510 con 0x7f558c0a6ba0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2244641545 0 0) 0x7f5578004eb0 con 0x7f558c13f5c0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f558c0a14e0 con 0x7f558c13f5c0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5584001f50 con 0x7f558c13f5c0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:02.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): dump: 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.169 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2589691752 0 0) 0x7f558c0a5510 con 0x7f558c0a6ba0 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5578004eb0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5580004cd0 con 0x7f558c0a6ba0 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:02:02.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient(hunting): dump: 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2641692670 0 0) 0x7f558c0a14e0 con 0x7f558c13f5c0 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.1:6791/0 conn(0x7f558c0a67c0 legacy=0x7f558c13ebb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6789/0 conn(0x7f558c0a6ba0 legacy=0x7f558c1d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 monclient: found mon.h 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c13a650 con 0x7f558c13f5c0 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: _finish_auth 0 2024-08-19T10:02:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 5 monclient: authenticate success, global_id 27554 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 10 monclient: _renew_subs 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f558c139db0 con 0x7f558c13f5c0 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 10 client.27554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 10 monclient: _renew_subs 2024-08-19T10:02:02.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f558c13f2c0 con 0x7f558c13f5c0 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f559256b640 1 librados: init done 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5584004e10 con 0x7f558c13f5c0 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5584005300 con 0x7f558c13f5c0 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f558a7fc640 4 set_mon_vals no callback set 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.148+0000 7f5588ff9640 10 monclient: dump: 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.174 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.175 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f559256b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f556c0065c0 con 0x7f556c002860 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5584011760 con 0x7f558c13f5c0 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(420..420 src has 299..420) ==== 5064+0+0 (unknown 2637308940 0 0) 0x7f558404e310 con 0x7f558c13f5c0 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f5588ff9640 10 client.27554.objecter ms_dispatch 0x7f558c000b90 osd_map(420..420 src has 299..420) 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f5588ff9640 3 client.27554.objecter handle_osd_map got epochs [420,420] > 0 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f5588ff9640 3 client.27554.objecter handle_osd_map decoding full epoch 420 2024-08-19T10:02:02.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f5588ff9640 20 client.27554.objecter dump_active .. 0 homeless 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.149+0000 7f558b7fe640 10 monclient: get_auth_request con 0x7f55780414f0 auth_method 0 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 10 client.27554.objecter ms_handle_connect 0x7f556c002860 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 10 client.27554.objecter resend_mon_ops 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f558bfff640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6792/0 conn(0x7f556c002860 legacy=0x7f556c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830552249 0 0) 0x7f556c0065c0 con 0x7f556c002860 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 10 monclient(hunting): my global_id is 145042 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.151+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f558c0a14e0 con 0x7f556c002860 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 702799527 0 0) 0x7f558c0a14e0 con 0x7f556c002860 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f556c0065c0 con 0x7f556c002860 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f557c004480 con 0x7f556c002860 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 10 monclient: dump: 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 10 client.27554.objecter ms_handle_connect 0x7f55780414f0 2024-08-19T10:02:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3574511595 0 0) 0x7f556c0065c0 con 0x7f556c002860 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f556c00a160 con 0x7f556c002860 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f557c0067c0 con 0x7f556c002860 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6792/0 conn(0x7f556c002860 legacy=0x7f556c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.152+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6792/0 conn(0x7f556c002860 legacy=0x7f556c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.158+0000 7f559256b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.158+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f556c009e70 con 0x7f556c009130 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f558bfff640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6792/0 conn(0x7f556c009130 legacy=0x7f556c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 client.27554.objecter ms_handle_connect 0x7f556c009130 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 client.27554.objecter resend_mon_ops 2024-08-19T10:02:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462953040 0 0) 0x7f556c009e70 con 0x7f556c009130 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient(hunting): my global_id is 145051 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f556c0065c0 con 0x7f556c009130 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3565203962 0 0) 0x7f556c0065c0 con 0x7f556c009130 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f556c009e70 con 0x7f556c009130 2024-08-19T10:02:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f557c004e10 con 0x7f556c009130 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 10 monclient: dump: 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:02:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:02:02.184 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2357718019 0 0) 0x7f556c009e70 con 0x7f556c009130 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.161+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f556c00c410 con 0x7f556c009130 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7663 (unknown 0 0 467026286) 0x7f557c006340 con 0x7f556c009130 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5588ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5588ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6792/0 conn(0x7f556c009130 legacy=0x7f556c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5588ff9640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6792/0 conn(0x7f556c009130 legacy=0x7f556c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 110, 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 446541, 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:02:02.186 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:02:02.187 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:02.188 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:02.189 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:02:02.190 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:02:02.191 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:02:02.192 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:02:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:02:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:02:02.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:02:02.204 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.205 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:02:02.206 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:02.207 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:02:02.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:02:02.209 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:02:02.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 10 librados: watch_flush enter 2024-08-19T10:02:02.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 10 librados: watch_flush exit 2024-08-19T10:02:02.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 20 client.27554.objecter shutdown clearing up homeless session... 2024-08-19T10:02:02.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 10 client.27554.objecter successfully canceled tick 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6812/2549104415 conn(0x7f55780414f0 legacy=0x7f55780439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 10 monclient: shutdown 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 >> v1:172.21.15.82:6791/0 conn(0x7f558c13f5c0 legacy=0x7f558c1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5590b08640 1 -- 172.21.15.1:0/172736690 reap_dead start 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f5590b08640 1 -- 172.21.15.1:0/172736690 reap_dead start 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 shutdown_connections 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 >> 172.21.15.1:0/172736690 conn(0x7f558c0ac4b0 msgr2=0x7f558c138770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 shutdown_connections 2024-08-19T10:02:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 -- 172.21.15.1:0/172736690 wait complete. 2024-08-19T10:02:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:02:02.162+0000 7f559256b640 1 librados: shutdown 2024-08-19T10:02:02.213 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:02:02.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (33655 ms) 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (49079 ms) 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3268 ms) 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.218 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3311 ms) 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3034 ms) 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3027 ms) 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4026 ms) 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40443 ms) 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.219 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (23998 ms) 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4931 ms) 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3016 ms) 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (46317 ms) 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14647 ms) 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-08-19T10:02:03.220 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5571 ms) 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4567 ms) 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: waiting for scrubs... 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: done waiting 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24133 ms) 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10731 ms) 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-08-19T10:02:03.221 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (20275 ms) 2024-08-19T10:02:03.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-08-19T10:02:03.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:02:03.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (25949 ms) 2024-08-19T10:02:03.222 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-08-19T10:02:59.723 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-08-19T10:02:59.721+0000 7f0f16240640 -1 snap_mapper.add_oid found existing snaps mapped on 99:a2a816e0:test-rados-api-smithi082-25997-97::foo:21, removing 2024-08-19T10:02:59.724 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-08-19T10:02:59.721+0000 7fba93e0d640 -1 snap_mapper.add_oid found existing snaps mapped on 99:a2a816e0:test-rados-api-smithi082-25997-97::foo:21, removing 2024-08-19T10:03:32.214 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T10:03:32.214 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T10:03:32.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T10:03:32.219 INFO:tasks.ceph.mon.d:Started 2024-08-19T10:03:32.219 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-19T10:03:32.220 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-19T10:03:32.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-19T10:03:32.222 INFO:tasks.ceph.mon.c:Started 2024-08-19T10:03:32.223 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:03:32.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:03:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.337+0000 7fb2c3884640 10 monclient: get_monmap_and_config 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.337+0000 7fb2c3884640 10 monclient: build_initial_monmap 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.337+0000 7fb2c3884640 10 monclient: monmap: 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:32.339818+0000 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:32.339818+0000 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:32.341 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:32.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.338+0000 7fb2c3884640 1 Processor -- start 2024-08-19T10:03:32.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.338+0000 7fb2c3884640 1 -- start start 2024-08-19T10:03:32.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.338+0000 7fb2c3884640 10 monclient: init 2024-08-19T10:03:32.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:32.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient: _add_conns ranks=[7,6,5,8,3,4,2,1,0] 2024-08-19T10:03:32.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient(hunting): picked mon.noname-i con 0x7fb2bc0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient(hunting): picked mon.noname-d con 0x7fb2bc0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient(hunting): picked mon.noname-h con 0x7fb2bc13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c0e1f640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fb2bc0a6ba0 0x7fb2bc0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:57828/0 (socket says 172.21.15.1:57828) 2024-08-19T10:03:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c0e1f640 1 -- 172.21.15.1:0/3148486696 learned_addr learned my addr 172.21.15.1:0/3148486696 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc0a3b40 con 0x7fb2bc13f5c0 2024-08-19T10:03:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc0a8130 con 0x7fb2bc0a6ba0 2024-08-19T10:03:32.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc06ede0 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:32.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.339+0000 7fb2c3884640 10 monclient(hunting): authenticate will time out at 1468.416889s 2024-08-19T10:03:32.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.341+0000 7fb2c0e1f640 1 -- 172.21.15.1:0/3148486696 >> v1:172.21.15.1:6792/0 conn(0x7fb2bc0a6ba0 legacy=0x7fb2bc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:32.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.342+0000 7fb2c1e21640 1 -- 172.21.15.1:0/3148486696 >> v1:172.21.15.82:6791/0 conn(0x7fb2bc13f5c0 legacy=0x7fb2bc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:32.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.342+0000 7fb2c1620640 1 -- 172.21.15.1:0/3148486696 >> v1:172.21.15.82:6792/0 conn(0x7fb2bc0a67c0 legacy=0x7fb2bc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:32.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2636118050 0 0) 0x7fb2bc06ede0 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:32.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:32.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:32.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 10 monclient(hunting): my global_id is 145060 2024-08-19T10:03:32.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2a8003520 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2903281358 0 0) 0x7fb2a8003520 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 >> v1:172.21.15.1:6792/0 conn(0x7fb2bc0a6ba0 legacy=0x7fb2bc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 >> v1:172.21.15.82:6791/0 conn(0x7fb2bc13f5c0 legacy=0x7fb2bc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 10 monclient: _finish_hunting 0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.346+0000 7fb2b3fff640 1 monclient: found mon.noname-i 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.82:6792/0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2bc0a92a0 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: _finish_auth 0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: _check_auth_tickets 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2b8001ae0 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: dump: 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:32.352 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2c3884640 5 monclient: authenticate success, global_id 145060 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2c3884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 <== mon.7 v1:172.21.15.82:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2b8003610 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 1 -- 172.21.15.1:0/3148486696 <== mon.7 v1:172.21.15.82:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2b8003b00 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2b3fff640 10 monclient: dump: 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2c3884640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.347+0000 7fb2c3884640 4 set_mon_vals no callback set 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 10 monclient: shutdown 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 >> v1:172.21.15.82:6792/0 conn(0x7fb2bc0a67c0 legacy=0x7fb2bc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 shutdown_connections 2024-08-19T10:03:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 >> 172.21.15.1:0/3148486696 conn(0x7fb2bc0ac4b0 msgr2=0x7fb2bc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 shutdown_connections 2024-08-19T10:03:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.348+0000 7fb2c3884640 1 -- 172.21.15.1:0/3148486696 wait complete. 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 10 monclient: build_initial_monmap 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 10 monclient: monmap: 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:32.359 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 1 librados: starting msgr at 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 1 librados: starting objecter 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 1 Processor -- start 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 1 -- start start 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 1 librados: setting wanted keys 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 1 librados: calling monclient init 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.349+0000 7fb2c3884640 10 monclient: init 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:32.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient: _add_conns ranks=[8,5,4,6,0,7,2,1,3] 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient(hunting): picked mon.noname-e con 0x7fb2bc0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient(hunting): picked mon.noname-h con 0x7fb2bc0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c1620640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fb2bc0a67c0 0x7fb2bc09e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:52722/0 (socket says 172.21.15.1:52722) 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c1620640 1 -- 172.21.15.1:0/175142294 learned_addr learned my addr 172.21.15.1:0/175142294 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc0aa510 con 0x7fb2bc13f5c0 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc0a1b50 con 0x7fb2bc0a6ba0 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc0a1e30 con 0x7fb2bc0a67c0 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.350+0000 7fb2c3884640 10 monclient(hunting): authenticate will time out at 1468.427968s 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.352+0000 7fb2c1620640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.1:6793/0 conn(0x7fb2bc0a67c0 legacy=0x7fb2bc09e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.352+0000 7fb2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2bc0a67c0 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.352+0000 7fb2b1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.352+0000 7fb2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2bc13f5c0 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.352+0000 7fb2b1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.353+0000 7fb2c1e21640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.1:6791/0 conn(0x7fb2bc13f5c0 legacy=0x7fb2bc1daa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.353+0000 7fb2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2bc0a6ba0 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.353+0000 7fb2c0e1f640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6791/0 conn(0x7fb2bc0a6ba0 legacy=0x7fb2bc09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:32.353+0000 7fb2b1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): tick 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 monclient(hunting): continuing hunt 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.1:6793/0 conn(0x7fb2bc0a67c0 legacy=0x7fb2bc09e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6791/0 conn(0x7fb2bc0a6ba0 legacy=0x7fb2bc09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.1:6791/0 conn(0x7fb2bc13f5c0 legacy=0x7fb2bc1daa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:35.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient: _add_conns ranks=[7,1,6,0,2,3] 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): picked mon.noname-i con 0x7fb2a4003960 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): picked mon.noname-f con 0x7fb2a40075f0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): picked mon.noname-d con 0x7fb2a400b360 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc0a0910 con 0x7fb2a40075f0 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a4010160 con 0x7fb2a400b360 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a4011400 con 0x7fb2a4003960 2024-08-19T10:03:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.350+0000 7fb2b0ff9640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2c1e21640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.1:6792/0 conn(0x7fb2a400b360 legacy=0x7fb2a400d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2a400b360 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2b1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2a4003960 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2b1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2a40075f0 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2b1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2c1620640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6792/0 conn(0x7fb2a4003960 legacy=0x7fb2a4005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:35.353+0000 7fb2c0e1f640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6789/0 conn(0x7fb2a40075f0 legacy=0x7fb2a4009a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:37.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.411+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2061425816 0 0) 0x7fb2a4011400 con 0x7fb2a4003960 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.411+0000 7fb2b1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.411+0000 7fb2b1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient(hunting): my global_id is 145069 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2a0004eb0 con 0x7fb2a4003960 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 627702706 0 0) 0x7fb2a0004eb0 con 0x7fb2a4003960 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2a4011400 con 0x7fb2a4003960 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2b8003b00 con 0x7fb2a4003960 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:03:37.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient(hunting): dump: 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1456419577 0 0) 0x7fb2a4011400 con 0x7fb2a4003960 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.1:6792/0 conn(0x7fb2a400b360 legacy=0x7fb2a400d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6789/0 conn(0x7fb2a40075f0 legacy=0x7fb2a4009a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 monclient: found mon.i 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a40126a0 con 0x7fb2a4003960 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient: _finish_auth 0 2024-08-19T10:03:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2c1e21640 1 -- 172.21.15.1:0/175142294 reap_dead start 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2c3884640 5 monclient: authenticate success, global_id 145069 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2c3884640 10 monclient: _renew_subs 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2c3884640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.412+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2bc0a2880 con 0x7fb2a4003960 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2b8001aa0 con 0x7fb2a4003960 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb2b8004430 con 0x7fb2a4003960 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b1ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b1ffb640 10 monclient: dump: 2024-08-19T10:03:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2c3884640 10 client.145069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2c3884640 10 monclient: _renew_subs 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2b37fe640 4 set_mon_vals no callback set 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2c3884640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2bc09e630 con 0x7fb2a4003960 2024-08-19T10:03:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.413+0000 7fb2c3884640 1 librados: init done 2024-08-19T10:03:37.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2c3884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:37.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2c3884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:37.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2c3884640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2bc0a0be0 con 0x7fb2a4003960 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb2b8011a80 con 0x7fb2a4003960 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(481..481 src has 299..481) ==== 5501+0+0 (unknown 220826006 0 0) 0x7fb2b804e240 con 0x7fb2a4003960 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2b1ffb640 10 client.145069.objecter ms_dispatch 0x7fb2bc000b90 osd_map(481..481 src has 299..481) 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2b1ffb640 3 client.145069.objecter handle_osd_map got epochs [481,481] > 0 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2b1ffb640 3 client.145069.objecter handle_osd_map decoding full epoch 481 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2b1ffb640 20 client.145069.objecter dump_active .. 0 homeless 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.414+0000 7fb2c0e1f640 10 monclient: get_auth_request con 0x7fb2a0040eb0 auth_method 0 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.417+0000 7fb2b1ffb640 10 client.145069.objecter ms_handle_connect 0x7fb2a0040eb0 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.419+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb2b8017670 con 0x7fb2a4003960 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.419+0000 7fb2b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.419+0000 7fb2b1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:37.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.680+0000 7fb2c3884640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:03:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.680+0000 7fb2c3884640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.680+0000 7fb2c3884640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.680+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb2bc068f60 con 0x7fb2a4003960 2024-08-19T10:03:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.681+0000 7fb2b1ffb640 1 -- 172.21.15.1:0/175142294 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2697 (unknown 254746657 0 1545334044) 0x7fb2b8016f70 con 0x7fb2a4003960 2024-08-19T10:03:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.681+0000 7fb2b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.681+0000 7fb2b1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":112,"quorum":[0,1,2,3,5,6,7,8],"quorum_names":["a","f","b","g","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:03:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 10 librados: watch_flush enter 2024-08-19T10:03:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 10 librados: watch_flush exit 2024-08-19T10:03:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 20 client.145069.objecter shutdown clearing up homeless session... 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 10 client.145069.objecter successfully canceled tick 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb2a0040eb0 legacy=0x7fb2a0043370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 10 monclient: shutdown 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 >> v1:172.21.15.82:6792/0 conn(0x7fb2a4003960 legacy=0x7fb2a4005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 shutdown_connections 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 >> 172.21.15.1:0/175142294 conn(0x7fb2bc0ac4b0 msgr2=0x7fb2bc0aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 shutdown_connections 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 -- 172.21.15.1:0/175142294 wait complete. 2024-08-19T10:03:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:37.683+0000 7fb2c3884640 1 librados: shutdown 2024-08-19T10:03:40.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.815+0000 7ff2a8386640 10 monclient: get_monmap_and_config 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.815+0000 7ff2a8386640 10 monclient: build_initial_monmap 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.815+0000 7ff2a8386640 10 monclient: monmap: 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:40.816154+0000 2024-08-19T10:03:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:40.816154+0000 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:40.817 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.815+0000 7ff2a8386640 1 Processor -- start 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.815+0000 7ff2a8386640 1 -- start start 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.815+0000 7ff2a8386640 10 monclient: init 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient: _add_conns ranks=[4,0,8,2,5,1,6,3,7] 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient(hunting): picked mon.noname-c con 0x7ff2a00a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient(hunting): picked mon.noname-a con 0x7ff2a00694b0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient(hunting): picked mon.noname-e con 0x7ff2a00a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a00a3c70 con 0x7ff2a00694b0 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a00a33d0 con 0x7ff2a00a6830 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a00a36b0 con 0x7ff2a00a46a0 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a8386640 10 monclient(hunting): authenticate will time out at 1476.892653s 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a5921640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff2a00694b0 0x7ff2a00a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57140/0 (socket says 172.21.15.1:57140) 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a6122640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff2a00a6830 0x7ff2a00a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52456/0 (socket says 172.21.15.1:52456) 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a5921640 1 -- 172.21.15.1:0/690768539 learned_addr learned my addr 172.21.15.1:0/690768539 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.816+0000 7ff2a6923640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff2a00a46a0 0x7ff2a0141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41708/0 (socket says 172.21.15.1:41708) 2024-08-19T10:03:40.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.818+0000 7ff2a6923640 1 -- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6793/0 conn(0x7ff2a00a46a0 legacy=0x7ff2a0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:40.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.818+0000 7ff2a5921640 1 -- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6789/0 conn(0x7ff2a00694b0 legacy=0x7ff2a00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:40.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.818+0000 7ff2a6122640 1 -- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6791/0 conn(0x7ff2a00a6830 legacy=0x7ff2a00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:40.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156172705 0 0) 0x7ff2a00a3c70 con 0x7ff2a00694b0 2024-08-19T10:03:40.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): _negotiate 2024-08-19T10:03:40.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): my global_id is 174618 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff28c003520 con 0x7ff2a00694b0 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991259485 0 0) 0x7ff2a00a36b0 con 0x7ff2a00a46a0 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): _negotiate 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient(hunting): my global_id is 184337 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2a00a3c70 con 0x7ff2a00a46a0 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3488607877 0 0) 0x7ff2a00a3c70 con 0x7ff2a00a46a0 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6791/0 conn(0x7ff2a00a6830 legacy=0x7ff2a00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:40.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6789/0 conn(0x7ff2a00694b0 legacy=0x7ff2a00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: _finish_hunting 0 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 monclient: found mon.noname-e 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a006dbb0 con 0x7ff2a00a46a0 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: _finish_auth 0 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: _check_auth_tickets 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff298001dc0 con 0x7ff2a00a46a0 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a8386640 5 monclient: authenticate success, global_id 184337 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a8386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:40.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: dump: 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:40.831 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a8386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a6923640 1 --1- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6793/0 conn(0x7ff2a00a46a0 0x7ff2a0141850 :-1 s=OPENED pgs=3572 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7ff298002090 config(0 keys) 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff298002090 con 0x7ff2a00a46a0 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a5120640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a8386640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.819+0000 7ff2a8386640 4 set_mon_vals no callback set 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 10 monclient: shutdown 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 -- 172.21.15.1:0/690768539 >> v1:172.21.15.1:6793/0 conn(0x7ff2a00a46a0 legacy=0x7ff2a0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a5120640 1 -- 172.21.15.1:0/690768539 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff298004ea0 con 0x7ff2a00a46a0 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a5120640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 -- 172.21.15.1:0/690768539 shutdown_connections 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 -- 172.21.15.1:0/690768539 >> 172.21.15.1:0/690768539 conn(0x7ff2a00a9020 msgr2=0x7ff2a0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 -- 172.21.15.1:0/690768539 shutdown_connections 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 -- 172.21.15.1:0/690768539 wait complete. 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 10 monclient: build_initial_monmap 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 10 monclient: monmap: 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 librados: starting msgr at 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.820+0000 7ff2a8386640 1 librados: starting objecter 2024-08-19T10:03:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 Processor -- start 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 -- start start 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 librados: setting wanted keys 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 librados: calling monclient init 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient: init 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient: _add_conns ranks=[8,1,4,0,7,6,3,5,2] 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient(hunting): picked mon.noname-e con 0x7ff2a00694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient(hunting): picked mon.noname-f con 0x7ff2a00a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient(hunting): picked mon.noname-c con 0x7ff2a00a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a0138140 con 0x7ff2a00a46a0 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a013aa90 con 0x7ff2a00a6830 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a00aa0a0 con 0x7ff2a00694b0 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a8386640 10 monclient(hunting): authenticate will time out at 1476.897621s 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a6122640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff2a00694b0 0x7ff2a0142000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41710/0 (socket says 172.21.15.1:41710) 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.821+0000 7ff2a6122640 1 -- 172.21.15.1:0/1541210476 learned_addr learned my addr 172.21.15.1:0/1541210476 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff2a6122640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.1:6793/0 conn(0x7ff2a00694b0 legacy=0x7ff2a0142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff296ffd640 10 client.?.objecter ms_handle_connect 0x7ff2a00694b0 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff296ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff2a6923640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.1:6791/0 conn(0x7ff2a00a6830 legacy=0x7ff2a01d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff296ffd640 10 client.?.objecter ms_handle_connect 0x7ff2a00a6830 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff296ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.823+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418718041 0 0) 0x7ff2a00aa0a0 con 0x7ff2a00694b0 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): my global_id is 184346 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff284004eb0 con 0x7ff2a00694b0 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a5921640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.82:6789/0 conn(0x7ff2a00a46a0 legacy=0x7ff2a01d2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 client.?.objecter ms_handle_connect 0x7ff2a00a46a0 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1324819432 0 0) 0x7ff2a013aa90 con 0x7ff2a00a6830 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): my global_id is 194125 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2a00aa0a0 con 0x7ff2a00a6830 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3602728136 0 0) 0x7ff2a00aa0a0 con 0x7ff2a00a6830 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2a013aa90 con 0x7ff2a00a6830 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3445219111 0 0) 0x7ff284004eb0 con 0x7ff2a00694b0 2024-08-19T10:03:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2a00aa0a0 con 0x7ff2a00694b0 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219892758 0 0) 0x7ff2a0138140 con 0x7ff2a00a46a0 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): my global_id is 174547 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff284004eb0 con 0x7ff2a00a46a0 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff298004ea0 con 0x7ff2a00a6830 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): dump: 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:40.839 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff29c001d00 con 0x7ff2a00694b0 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient(hunting): dump: 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1110884020 0 0) 0x7ff2a013aa90 con 0x7ff2a00a6830 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.1:6793/0 conn(0x7ff2a00694b0 legacy=0x7ff2a0142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.82:6789/0 conn(0x7ff2a00a46a0 legacy=0x7ff2a01d2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 monclient: found mon.c 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a0138730 con 0x7ff2a00a6830 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: _finish_auth 0 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 5 monclient: authenticate success, global_id 194125 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 10 monclient: _renew_subs 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a013a7b0 con 0x7ff2a00a6830 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 10 client.194125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 10 monclient: _renew_subs 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2a0139290 con 0x7ff2a00a6830 2024-08-19T10:03:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a8386640 1 librados: init done 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff298001de0 con 0x7ff2a00a6830 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff298005790 con 0x7ff2a00a6830 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff2a491f640 4 set_mon_vals no callback set 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.824+0000 7ff296ffd640 10 monclient: dump: 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.825+0000 7ff2a8386640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.825+0000 7ff2a8386640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.825+0000 7ff2a8386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.825+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff288005080 con 0x7ff2a00a6830 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.825+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff298011bf0 con 0x7ff2a00a6830 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.826+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(484..484 src has 299..484) ==== 4967+0+0 (unknown 3313500721 0 0) 0x7ff29804e1e0 con 0x7ff2a00a6830 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.826+0000 7ff296ffd640 10 client.194125.objecter ms_dispatch 0x7ff2a0000b90 osd_map(484..484 src has 299..484) 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.826+0000 7ff296ffd640 3 client.194125.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.826+0000 7ff296ffd640 3 client.194125.objecter handle_osd_map decoding full epoch 484 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.826+0000 7ff296ffd640 20 client.194125.objecter dump_active .. 0 homeless 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.826+0000 7ff2a6122640 10 monclient: get_auth_request con 0x7ff2840416d0 auth_method 0 2024-08-19T10:03:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.828+0000 7ff296ffd640 10 client.194125.objecter ms_handle_connect 0x7ff2840416d0 2024-08-19T10:03:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.830+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff2980177c0 con 0x7ff2a00a6830 2024-08-19T10:03:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.830+0000 7ff296ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:40.830+0000 7ff296ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:41.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff2a8386640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff2a8386640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff2a8386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff288005370 con 0x7ff2a00a6830 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff296ffd640 1 -- 172.21.15.1:0/1541210476 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 735911810) 0x7ff2980170c0 con 0x7ff2a00a6830 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff296ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.087+0000 7ff296ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:41.089 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":118,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:03:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 10 librados: watch_flush enter 2024-08-19T10:03:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 10 librados: watch_flush exit 2024-08-19T10:03:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 20 client.194125.objecter shutdown clearing up homeless session... 2024-08-19T10:03:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 10 client.194125.objecter successfully canceled tick 2024-08-19T10:03:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff2840416d0 legacy=0x7ff284043b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 10 monclient: shutdown 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.089+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 >> v1:172.21.15.1:6791/0 conn(0x7ff2a00a6830 legacy=0x7ff2a01d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.090+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 shutdown_connections 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.090+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 >> 172.21.15.1:0/1541210476 conn(0x7ff2a00a9020 msgr2=0x7ff2a00a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.090+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 shutdown_connections 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.090+0000 7ff2a8386640 1 -- 172.21.15.1:0/1541210476 wait complete. 2024-08-19T10:03:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.090+0000 7ff2a8386640 1 librados: shutdown 2024-08-19T10:03:41.101 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:03:41.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:03:41.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.200+0000 7f196983f640 10 monclient: get_monmap_and_config 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.200+0000 7f196983f640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.200+0000 7f196983f640 10 monclient: monmap: 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:41.201584+0000 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:41.201584+0000 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.202 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.203 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 1 Processor -- start 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 1 -- start start 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 10 monclient: init 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 10 monclient: _add_conns ranks=[0,7,2,1,8,3,6,4,5] 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 10 monclient(hunting): picked mon.noname-a con 0x7f19640a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.201+0000 7f196983f640 10 monclient(hunting): picked mon.noname-i con 0x7f19640694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f196983f640 10 monclient(hunting): picked mon.noname-b con 0x7f19640a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f196983f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19640a3c70 con 0x7f19640a6830 2024-08-19T10:03:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f196983f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19640a33d0 con 0x7f19640a46a0 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f196983f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19640a36b0 con 0x7f19640694b0 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f196983f640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f196983f640 10 monclient(hunting): authenticate will time out at 1477.278067s 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f1962ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f19640a6830 0x7f19640a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57144/0 (socket says 172.21.15.1:57144) 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.202+0000 7f1962ffd640 1 -- 172.21.15.1:0/4052756348 learned_addr learned my addr 172.21.15.1:0/4052756348 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19637fe640 1 -- 172.21.15.1:0/4052756348 >> v1:172.21.15.1:6790/0 conn(0x7f19640a46a0 legacy=0x7f1964141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f1962ffd640 1 -- 172.21.15.1:0/4052756348 >> v1:172.21.15.1:6789/0 conn(0x7f19640a6830 legacy=0x7f19640a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f195bfff640 1 -- 172.21.15.1:0/4052756348 >> v1:172.21.15.82:6792/0 conn(0x7f19640694b0 legacy=0x7f19640a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790223790 0 0) 0x7f19640a33d0 con 0x7f19640a46a0 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): my global_id is 184259 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1950003520 con 0x7f19640a46a0 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2081267663 0 0) 0x7f19640a3c70 con 0x7f19640a6830 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient(hunting): my global_id is 174627 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19640a33d0 con 0x7f19640a6830 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 68529605 0 0) 0x7f1950003520 con 0x7f19640a46a0 2024-08-19T10:03:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 >> v1:172.21.15.82:6792/0 conn(0x7f19640694b0 legacy=0x7f19640a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 >> v1:172.21.15.1:6789/0 conn(0x7f19640a6830 legacy=0x7f19640a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 monclient: found mon.noname-b 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196406dbb0 con 0x7f19640a46a0 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f195c001eb0 con 0x7f19640a46a0 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f196983f640 5 monclient: authenticate success, global_id 184259 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f196983f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: dump: 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.213 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f196983f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f195c003990 con 0x7f19640a46a0 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 1 -- 172.21.15.1:0/4052756348 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f195c003f00 con 0x7f19640a46a0 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f196983f640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f196983f640 4 set_mon_vals no callback set 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.204+0000 7f19627fc640 10 monclient: dump: 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 10 monclient: shutdown 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 1 -- 172.21.15.1:0/4052756348 >> v1:172.21.15.1:6790/0 conn(0x7f19640a46a0 legacy=0x7f1964141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 1 -- 172.21.15.1:0/4052756348 shutdown_connections 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 1 -- 172.21.15.1:0/4052756348 >> 172.21.15.1:0/4052756348 conn(0x7f19640a9020 msgr2=0x7f1964068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 1 -- 172.21.15.1:0/4052756348 shutdown_connections 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.205+0000 7f196983f640 1 -- 172.21.15.1:0/4052756348 wait complete. 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient: monmap: 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 1 librados: starting msgr at 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 1 librados: starting objecter 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 1 Processor -- start 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 1 -- start start 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 1 librados: setting wanted keys 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 1 librados: calling monclient init 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient: init 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient: _add_conns ranks=[7,2,5,8,6,1,4,3,0] 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient(hunting): picked mon.noname-i con 0x7f19640694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.206+0000 7f196983f640 10 monclient(hunting): picked mon.noname-b con 0x7f19640a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f196983f640 10 monclient(hunting): picked mon.noname-h con 0x7f19640a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f196983f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19641367b0 con 0x7f19640a46a0 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f196983f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19641375d0 con 0x7f19640a6830 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f196983f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19641378b0 con 0x7f19640694b0 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f196983f640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f196983f640 10 monclient(hunting): authenticate will time out at 1477.283036s 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f195bfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f19640a46a0 0x7f1964135630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45530/0 (socket says 172.21.15.1:45530) 2024-08-19T10:03:41.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.207+0000 7f195bfff640 1 -- 172.21.15.1:0/3954999675 learned_addr learned my addr 172.21.15.1:0/3954999675 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f195bfff640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6790/0 conn(0x7f19640a46a0 legacy=0x7f1964135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 client.?.objecter ms_handle_connect 0x7f19640a46a0 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843576070 0 0) 0x7f19641367b0 con 0x7f19640a46a0 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): my global_id is 184268 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1944004eb0 con 0x7f19640a46a0 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 client.?.objecter ms_handle_connect 0x7f19640694b0 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1962ffd640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.82:6792/0 conn(0x7f19640694b0 legacy=0x7f1964134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f19637fe640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.82:6791/0 conn(0x7f19640a6830 legacy=0x7f19641d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 client.?.objecter ms_handle_connect 0x7f19640a6830 2024-08-19T10:03:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4053021720 0 0) 0x7f1944004eb0 con 0x7f19640a46a0 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19641367b0 con 0x7f19640a46a0 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f194c003610 con 0x7f19640a46a0 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): dump: 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140486600 0 0) 0x7f19641378b0 con 0x7f19640694b0 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): my global_id is 145087 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1944004eb0 con 0x7f19640694b0 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1305822042 0 0) 0x7f19641375d0 con 0x7f19640a6830 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient(hunting): my global_id is 27572 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19641378b0 con 0x7f19640a6830 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1604004843 0 0) 0x7f19641367b0 con 0x7f19640a46a0 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.82:6792/0 conn(0x7f19640694b0 legacy=0x7f1964134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.82:6791/0 conn(0x7f19640a6830 legacy=0x7f19641d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 monclient: found mon.b 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1964135d40 con 0x7f19640a46a0 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 5 monclient: authenticate success, global_id 184268 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 10 monclient: _renew_subs 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1964136390 con 0x7f19640a46a0 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 10 client.184268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 10 monclient: _renew_subs 2024-08-19T10:03:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f19640a1700 con 0x7f19640a46a0 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f196983f640 1 librados: init done 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.209+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f194c003bd0 con 0x7f19640a46a0 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1960ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f194c004100 con 0x7f19640a46a0 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1960ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1961ffb640 4 set_mon_vals no callback set 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1960ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1960ff9640 10 monclient: dump: 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f196983f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f19480065c0 con 0x7f1948002860 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.210+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f194c0042a0 con 0x7f19640a46a0 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.211+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(484..484 src has 299..484) ==== 4967+0+0 (unknown 3313500721 0 0) 0x7f194c04e0c0 con 0x7f19640a46a0 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.211+0000 7f1960ff9640 10 client.184268.objecter ms_dispatch 0x7f1964000b90 osd_map(484..484 src has 299..484) 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.211+0000 7f1960ff9640 3 client.184268.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.211+0000 7f1960ff9640 3 client.184268.objecter handle_osd_map decoding full epoch 484 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.211+0000 7f1960ff9640 20 client.184268.objecter dump_active .. 0 homeless 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.211+0000 7f19637fe640 10 monclient: get_auth_request con 0x7f19440410c0 auth_method 0 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.212+0000 7f1960ff9640 10 client.184268.objecter ms_handle_connect 0x7f1948002860 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.212+0000 7f1962ffd640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6789/0 conn(0x7f1948002860 legacy=0x7f1948004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.212+0000 7f1960ff9640 10 client.184268.objecter resend_mon_ops 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2148726330 0 0) 0x7f19480065c0 con 0x7f1948002860 2024-08-19T10:03:41.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient(hunting): my global_id is 174636 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19641367b0 con 0x7f1948002860 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2780823152 0 0) 0x7f19641367b0 con 0x7f1948002860 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19480065c0 con 0x7f1948002860 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f19540018c0 con 0x7f1948002860 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient: dump: 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.233 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1952637100 0 0) 0x7f19480065c0 con 0x7f1948002860 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f194800a160 con 0x7f1948002860 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 client.184268.objecter ms_handle_connect 0x7f19440410c0 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1954004870 con 0x7f1948002860 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6789/0 conn(0x7f1948002860 legacy=0x7f1948004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.213+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6789/0 conn(0x7f1948002860 legacy=0x7f1948004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.219+0000 7f196983f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.219+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1948009e70 con 0x7f1948009130 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 client.184268.objecter ms_handle_connect 0x7f1948009130 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 client.184268.objecter resend_mon_ops 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1962ffd640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6789/0 conn(0x7f1948009130 legacy=0x7f19480095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691557607 0 0) 0x7f1948009e70 con 0x7f1948009130 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient(hunting): my global_id is 174645 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19480065c0 con 0x7f1948009130 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3392415577 0 0) 0x7f19480065c0 con 0x7f1948009130 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1948009e70 con 0x7f1948009130 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1954004fe0 con 0x7f1948009130 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient: dump: 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.236 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.237 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3193348528 0 0) 0x7f1948009e70 con 0x7f1948009130 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f194800c3f0 con 0x7f1948009130 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7687 (unknown 0 0 2481036994) 0x7f19540072e0 con 0x7f1948009130 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6789/0 conn(0x7f1948009130 legacy=0x7f19480095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.222+0000 7f1960ff9640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6789/0 conn(0x7f1948009130 legacy=0x7f19480095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:41.238 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 271263, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:41.239 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.241 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:41.242 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.243 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.244 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.245 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:41.246 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:41.247 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.248 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:41.250 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.251 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:41.253 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.258 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.259 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:41.259 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:41.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 10 librados: watch_flush enter 2024-08-19T10:03:41.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 10 librados: watch_flush exit 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 20 client.184268.objecter shutdown clearing up homeless session... 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 10 client.184268.objecter successfully canceled tick 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.82:6812/2549104415 conn(0x7f19440410c0 legacy=0x7f1944043580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 10 monclient: shutdown 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 >> v1:172.21.15.1:6790/0 conn(0x7f19640a46a0 legacy=0x7f1964135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f19637fe640 1 -- 172.21.15.1:0/3954999675 reap_dead start 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f19637fe640 1 -- 172.21.15.1:0/3954999675 reap_dead start 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 shutdown_connections 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 >> 172.21.15.1:0/3954999675 conn(0x7f19640a9020 msgr2=0x7f19640a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 shutdown_connections 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 -- 172.21.15.1:0/3954999675 wait complete. 2024-08-19T10:03:41.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.223+0000 7f196983f640 1 librados: shutdown 2024-08-19T10:03:41.262 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:41.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:03:41.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.361+0000 7f0c06b6b640 10 monclient: get_monmap_and_config 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.361+0000 7f0c06b6b640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.361+0000 7f0c06b6b640 10 monclient: monmap: 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:41.362760+0000 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:41.362760+0000 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.364 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.362+0000 7f0c06b6b640 1 Processor -- start 2024-08-19T10:03:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.362+0000 7f0c06b6b640 1 -- start start 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.362+0000 7f0c06b6b640 10 monclient: init 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient: _add_conns ranks=[7,0,5,3,1,2,6,4,8] 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient(hunting): picked mon.noname-i con 0x7f0c000a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f0c000a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient(hunting): picked mon.noname-h con 0x7f0c0013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000a3b40 con 0x7f0c000a6ba0 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000a8130 con 0x7f0c0013f5c0 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0006ede0 con 0x7f0c000a67c0 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0c06b6b640 10 monclient(hunting): authenticate will time out at 1477.439212s 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0bfffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c000a6ba0 0x7f0c000a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57164/0 (socket says 172.21.15.1:57164) 2024-08-19T10:03:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.363+0000 7f0bfffff640 1 -- 172.21.15.1:0/1988248195 learned_addr learned my addr 172.21.15.1:0/1988248195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bfffff640 1 -- 172.21.15.1:0/1988248195 >> v1:172.21.15.1:6789/0 conn(0x7f0c000a6ba0 legacy=0x7f0c000a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0c04907640 1 -- 172.21.15.1:0/1988248195 >> v1:172.21.15.82:6792/0 conn(0x7f0c000a67c0 legacy=0x7f0c000a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3856076530 0 0) 0x7f0c000a3b40 con 0x7f0c000a6ba0 2024-08-19T10:03:41.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0c05108640 1 -- 172.21.15.1:0/1988248195 >> v1:172.21.15.82:6791/0 conn(0x7f0c0013f5c0 legacy=0x7f0c001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): my global_id is 174654 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bec003520 con 0x7f0c000a6ba0 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469867498 0 0) 0x7f0c0006ede0 con 0x7f0c000a67c0 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): my global_id is 145096 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c000a3b40 con 0x7f0c000a67c0 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557895034 0 0) 0x7f0c000a8130 con 0x7f0c0013f5c0 2024-08-19T10:03:41.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient(hunting): my global_id is 27581 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c0006ede0 con 0x7f0c0013f5c0 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 335385678 0 0) 0x7f0bec003520 con 0x7f0c000a6ba0 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 >> v1:172.21.15.82:6792/0 conn(0x7f0c000a67c0 legacy=0x7f0c000a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 >> v1:172.21.15.82:6791/0 conn(0x7f0c0013f5c0 legacy=0x7f0c001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 monclient: found mon.noname-a 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c000a92a0 con 0x7f0c000a6ba0 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0bf0001e60 con 0x7f0c000a6ba0 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0c06b6b640 5 monclient: authenticate success, global_id 174654 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0bff7fe640 10 monclient: dump: 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.373 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.365+0000 7f0c06b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bf0004c20 con 0x7f0c000a6ba0 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0bff7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0bff7fe640 1 -- 172.21.15.1:0/1988248195 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0bf0005190 con 0x7f0c000a6ba0 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0bff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0bff7fe640 10 monclient: dump: 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 4 set_mon_vals no callback set 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 10 monclient: shutdown 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1988248195 >> v1:172.21.15.1:6789/0 conn(0x7f0c000a6ba0 legacy=0x7f0c000a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1988248195 shutdown_connections 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1988248195 >> 172.21.15.1:0/1988248195 conn(0x7f0c000ac4b0 msgr2=0x7f0c000a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1988248195 shutdown_connections 2024-08-19T10:03:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.366+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1988248195 wait complete. 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 10 monclient: monmap: 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.377 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 1 librados: starting msgr at 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 1 librados: starting objecter 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 1 Processor -- start 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 1 -- start start 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 1 librados: setting wanted keys 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 1 librados: calling monclient init 2024-08-19T10:03:41.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 10 monclient: init 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.367+0000 7f0c06b6b640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 10 monclient: _add_conns ranks=[0,1,8,4,5,7,3,2,6] 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f0c000a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 10 monclient(hunting): picked mon.noname-f con 0x7f0c000a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 10 monclient(hunting): picked mon.noname-e con 0x7f0c0013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c001cf100 con 0x7f0c000a67c0 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000aa140 con 0x7f0c000a6ba0 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000a9cc0 con 0x7f0c0013f5c0 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c06b6b640 10 monclient(hunting): authenticate will time out at 1477.444087s 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c04907640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c000a67c0 0x7f0c001ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57172/0 (socket says 172.21.15.1:57172) 2024-08-19T10:03:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.368+0000 7f0c04907640 1 -- 172.21.15.1:0/1449572314 learned_addr learned my addr 172.21.15.1:0/1449572314 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c04907640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6789/0 conn(0x7f0c000a67c0 legacy=0x7f0c001ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f0c000a67c0 2024-08-19T10:03:41.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c05108640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6793/0 conn(0x7f0c0013f5c0 legacy=0x7f0c001d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c05108640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6793/0 conn(0x7f0c0013f5c0 legacy=0x7f0c001d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c05108640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6793/0 conn(0x7f0c0013f5c0 legacy=0x7f0c001d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c05108640 1 --1- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6793/0 conn(0x7f0c0013f5c0 0x7f0c001d65b0 :-1 s=OPENED pgs=3574 cs=1 l=1).handle_message read tag failed 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c05108640 1 --1- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6793/0 conn(0x7f0c0013f5c0 0x7f0c001d65b0 :-1 s=OPENED pgs=3574 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f0c0013f5c0 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6793/0 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f0c000a6ba0 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfffff640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6789/0 conn(0x7f0c000a6ba0 legacy=0x7f0c001ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322986624 0 0) 0x7f0c001cf100 con 0x7f0c000a67c0 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 174663 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be0004ee0 con 0x7f0c000a67c0 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3353540605 0 0) 0x7f0c000aa140 con 0x7f0c000a6ba0 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 174556 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c001cf100 con 0x7f0c000a6ba0 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1543934983 0 0) 0x7f0be0004ee0 con 0x7f0c000a67c0 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfffff640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6789/0 conn(0x7f0c000a6ba0 legacy=0x7f0c001ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c000aa140 con 0x7f0c000a67c0 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfffff640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6789/0 conn(0x7f0c000a6ba0 legacy=0x7f0c001ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:03:41.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfffff640 1 --1- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6789/0 conn(0x7f0c000a6ba0 0x7f0c001ce9f0 :-1 s=OPENED pgs=3566 cs=1 l=1).handle_message read tag failed 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0bf8003610 con 0x7f0c000a67c0 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfffff640 1 --1- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6789/0 conn(0x7f0c000a6ba0 0x7f0c001ce9f0 :-1 s=OPENED pgs=3566 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): dump: 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.388 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6789/0 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3017021588 0 0) 0x7f0c000aa140 con 0x7f0c000a67c0 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6793/0 conn(0x7f0c0013f5c0 legacy=0x7f0c001d65b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6789/0 conn(0x7f0c000a6ba0 legacy=0x7f0c001ce9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 monclient: found mon.a 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0013e3e0 con 0x7f0c000a67c0 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0bfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 5 monclient: authenticate success, global_id 174663 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 10 monclient: _renew_subs 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0013bef0 con 0x7f0c000a67c0 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 10 client.174663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 10 monclient: _renew_subs 2024-08-19T10:03:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c000a28a0 con 0x7f0c000a67c0 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.370+0000 7f0c06b6b640 1 librados: init done 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bf8003bb0 con 0x7f0c000a67c0 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0bf80040e0 con 0x7f0c000a67c0 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfeffd640 4 set_mon_vals no callback set 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 10 monclient: dump: 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0c06b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0be40065c0 con 0x7f0be4002860 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.371+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0bf8004280 con 0x7f0c000a67c0 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.372+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(484..484 src has 299..484) ==== 4967+0+0 (unknown 3313500721 0 0) 0x7f0bf804e040 con 0x7f0c000a67c0 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.372+0000 7f0bfd7fa640 10 client.174663.objecter ms_dispatch 0x7f0c00000b90 osd_map(484..484 src has 299..484) 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.372+0000 7f0bfd7fa640 3 client.174663.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.372+0000 7f0bfd7fa640 3 client.174663.objecter handle_osd_map decoding full epoch 484 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.372+0000 7f0bfd7fa640 20 client.174663.objecter dump_active .. 0 homeless 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.372+0000 7f0c05108640 10 monclient: get_auth_request con 0x7f0be0041010 auth_method 0 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfffff640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6790/0 conn(0x7f0be4002860 legacy=0x7f0be4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 client.174663.objecter ms_handle_connect 0x7f0be4002860 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 client.174663.objecter resend_mon_ops 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366102805 0 0) 0x7f0be40065c0 con 0x7f0be4002860 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 184277 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c000aa140 con 0x7f0be4002860 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 524679298 0 0) 0x7f0c000aa140 con 0x7f0be4002860 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0be40065c0 con 0x7f0be4002860 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0bf0004c20 con 0x7f0be4002860 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient: dump: 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.395 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.395 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.395 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.395 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.395 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1941734331 0 0) 0x7f0be40065c0 con 0x7f0be4002860 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0be400a160 con 0x7f0be4002860 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 client.174663.objecter ms_handle_connect 0x7f0be0041010 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0bf0004e40 con 0x7f0be4002860 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6790/0 conn(0x7f0be4002860 legacy=0x7f0be4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.374+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6790/0 conn(0x7f0be4002860 legacy=0x7f0be4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.381+0000 7f0c06b6b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.381+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0be4009e70 con 0x7f0be4009130 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfffff640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6790/0 conn(0x7f0be4009130 legacy=0x7f0be40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 client.174663.objecter ms_handle_connect 0x7f0be4009130 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 client.174663.objecter resend_mon_ops 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526642300 0 0) 0x7f0be4009e70 con 0x7f0be4009130 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 184286 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be40065c0 con 0x7f0be4009130 2024-08-19T10:03:41.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1869746114 0 0) 0x7f0be40065c0 con 0x7f0be4009130 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0be4009e70 con 0x7f0be4009130 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0bf0005810 con 0x7f0be4009130 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 10 monclient: dump: 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.398 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.399 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 356368624 0 0) 0x7f0be4009e70 con 0x7f0be4009130 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.383+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0be400c410 con 0x7f0be4009130 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1410809258) 0x7f0bf0005940 con 0x7f0be4009130 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0bfd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0bfd7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6790/0 conn(0x7f0be4009130 legacy=0x7f0be40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0bfd7fa640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6790/0 conn(0x7f0be4009130 legacy=0x7f0be40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.400 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:41.401 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:41.401 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:41.401 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:41.401 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:41.401 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 138632, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.402 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.403 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.404 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:41.405 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.406 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.407 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.408 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:41.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.416 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:41.417 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.418 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:41.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.421 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:41.422 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 10 librados: watch_flush enter 2024-08-19T10:03:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 10 librados: watch_flush exit 2024-08-19T10:03:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 20 client.174663.objecter shutdown clearing up homeless session... 2024-08-19T10:03:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 10 client.174663.objecter successfully canceled tick 2024-08-19T10:03:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0be0041010 legacy=0x7f0be00434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 10 monclient: shutdown 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 >> v1:172.21.15.1:6789/0 conn(0x7f0c000a67c0 legacy=0x7f0c001ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c05108640 1 -- 172.21.15.1:0/1449572314 reap_dead start 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c05108640 1 -- 172.21.15.1:0/1449572314 reap_dead start 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 shutdown_connections 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 >> 172.21.15.1:0/1449572314 conn(0x7f0c000ac4b0 msgr2=0x7f0c00141ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 shutdown_connections 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 -- 172.21.15.1:0/1449572314 wait complete. 2024-08-19T10:03:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.384+0000 7f0c06b6b640 1 librados: shutdown 2024-08-19T10:03:41.425 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:41.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:03:41.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:03:41.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.523+0000 7f6fdd508640 10 monclient: get_monmap_and_config 2024-08-19T10:03:41.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.523+0000 7f6fdd508640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.523+0000 7f6fdd508640 10 monclient: monmap: 2024-08-19T10:03:41.527 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:41.524469+0000 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:41.524469+0000 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 1 Processor -- start 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 1 -- start start 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient: init 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient: _add_conns ranks=[8,1,0,7,6,4,2,3,5] 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient(hunting): picked mon.noname-e con 0x7f6fd80a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient(hunting): picked mon.noname-f con 0x7f6fd80a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd80a3b40 con 0x7f6fd813f5c0 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd80a8130 con 0x7f6fd80a6ba0 2024-08-19T10:03:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd806ede0 con 0x7f6fd80a67c0 2024-08-19T10:03:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fdd508640 10 monclient(hunting): authenticate will time out at 1477.600852s 2024-08-19T10:03:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fd6ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f6fd80a67c0 0x7f6fd80a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41726/0 (socket says 172.21.15.1:41726) 2024-08-19T10:03:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fd77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6fd813f5c0 0x7f6fd81419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57178/0 (socket says 172.21.15.1:57178) 2024-08-19T10:03:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.524+0000 7f6fd6ffd640 1 -- 172.21.15.1:0/2165099644 learned_addr learned my addr 172.21.15.1:0/2165099644 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd6ffd640 1 -- 172.21.15.1:0/2165099644 >> v1:172.21.15.1:6793/0 conn(0x7f6fd80a67c0 legacy=0x7f6fd80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd77fe640 1 -- 172.21.15.1:0/2165099644 >> v1:172.21.15.1:6789/0 conn(0x7f6fd813f5c0 legacy=0x7f6fd81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066949282 0 0) 0x7f6fd806ede0 con 0x7f6fd80a67c0 2024-08-19T10:03:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): my global_id is 184355 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fc4003520 con 0x7f6fd80a67c0 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148030626 0 0) 0x7f6fd80a3b40 con 0x7f6fd813f5c0 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient(hunting): my global_id is 174672 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd806ede0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fcffff640 1 -- 172.21.15.1:0/2165099644 >> v1:172.21.15.82:6789/0 conn(0x7f6fd80a6ba0 legacy=0x7f6fd80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 920014416 0 0) 0x7f6fd806ede0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 >> v1:172.21.15.1:6793/0 conn(0x7f6fd80a67c0 legacy=0x7f6fd80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 >> v1:172.21.15.82:6789/0 conn(0x7f6fd80a6ba0 legacy=0x7f6fd80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 monclient: found mon.noname-a 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd80a92a0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fdd508640 5 monclient: authenticate success, global_id 174672 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fd0001e40 con 0x7f6fd813f5c0 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fdd508640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:41.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: dump: 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.537 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fdd508640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fd0004b60 con 0x7f6fd813f5c0 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fdd508640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 1 -- 172.21.15.1:0/2165099644 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fd0005090 con 0x7f6fd813f5c0 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fdd508640 4 set_mon_vals no callback set 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.527+0000 7f6fd67fc640 10 monclient: dump: 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.539 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 10 monclient: shutdown 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 1 -- 172.21.15.1:0/2165099644 >> v1:172.21.15.1:6789/0 conn(0x7f6fd813f5c0 legacy=0x7f6fd81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 1 -- 172.21.15.1:0/2165099644 shutdown_connections 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 1 -- 172.21.15.1:0/2165099644 >> 172.21.15.1:0/2165099644 conn(0x7f6fd80ac4b0 msgr2=0x7f6fd80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 1 -- 172.21.15.1:0/2165099644 shutdown_connections 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 1 -- 172.21.15.1:0/2165099644 wait complete. 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.528+0000 7f6fdd508640 10 monclient: monmap: 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 librados: starting msgr at 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 librados: starting objecter 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 Processor -- start 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 -- start start 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 librados: setting wanted keys 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 librados: calling monclient init 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient: init 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient: _add_conns ranks=[5,3,0,8,2,1,6,4,7] 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient(hunting): picked mon.noname-h con 0x7f6fd80a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient(hunting): picked mon.noname-g con 0x7f6fd80a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd80a17b0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd8142c40 con 0x7f6fd80a6ba0 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd813a890 con 0x7f6fd80a67c0 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fdd508640 10 monclient(hunting): authenticate will time out at 1477.605734s 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fd77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6fd813f5c0 0x7f6fd81d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57184/0 (socket says 172.21.15.1:57184) 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.529+0000 7f6fd77fe640 1 -- 172.21.15.1:0/3816751036 learned_addr learned my addr 172.21.15.1:0/3816751036 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.531+0000 7f6fd77fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6789/0 conn(0x7f6fd813f5c0 legacy=0x7f6fd81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.531+0000 7f6fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f6fd813f5c0 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.531+0000 7f6fcf7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323752332 0 0) 0x7f6fd80a17b0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 10 monclient(hunting): my global_id is 174681 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fb8004eb0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcffff640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.82:6790/0 conn(0x7f6fd80a6ba0 legacy=0x7f6fd80a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:03:41.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.532+0000 7f6fd6ffd640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.82:6791/0 conn(0x7f6fd80a67c0 legacy=0x7f6fd809ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f6fd80a6ba0 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f6fd80a67c0 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790049202 0 0) 0x7f6fd8142c40 con 0x7f6fd80a6ba0 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): my global_id is 184323 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd80a17b0 con 0x7f6fd80a6ba0 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2992082003 0 0) 0x7f6fb8004eb0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd8142c40 con 0x7f6fd813f5c0 2024-08-19T10:03:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961443618 0 0) 0x7f6fd813a890 con 0x7f6fd80a67c0 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): my global_id is 27590 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fb8004eb0 con 0x7f6fd80a67c0 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fd0005050 con 0x7f6fd813f5c0 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): dump: 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2549075317 0 0) 0x7f6fd8142c40 con 0x7f6fd813f5c0 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.82:6791/0 conn(0x7f6fd80a67c0 legacy=0x7f6fd809ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.82:6790/0 conn(0x7f6fd80a6ba0 legacy=0x7f6fd80a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 monclient: found mon.a 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd80a27d0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 5 monclient: authenticate success, global_id 174681 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 10 monclient: _renew_subs 2024-08-19T10:03:41.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd80a2310 con 0x7f6fd813f5c0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 10 client.174681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 10 monclient: _renew_subs 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6fd8139c00 con 0x7f6fd813f5c0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fdd508640 1 librados: init done 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fd0004b60 con 0x7f6fd813f5c0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fd0001de0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fd5ffb640 4 set_mon_vals no callback set 2024-08-19T10:03:41.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.534+0000 7f6fcf7fe640 10 monclient: dump: 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.555 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6fd0011aa0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fdd508640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fbc0065c0 con 0x7f6fbc002860 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(484..484 src has 299..484) ==== 4967+0+0 (unknown 3313500721 0 0) 0x7f6fd004e2e0 con 0x7f6fd813f5c0 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcf7fe640 10 client.174681.objecter ms_dispatch 0x7f6fd8000b90 osd_map(484..484 src has 299..484) 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcf7fe640 3 client.174681.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcf7fe640 3 client.174681.objecter handle_osd_map decoding full epoch 484 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcf7fe640 20 client.174681.objecter dump_active .. 0 homeless 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.535+0000 7f6fcffff640 10 monclient: get_auth_request con 0x7f6fb8041230 auth_method 0 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fd6ffd640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6791/0 conn(0x7f6fbc002860 legacy=0x7f6fbc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 10 client.174681.objecter ms_handle_connect 0x7f6fbc002860 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 10 client.174681.objecter resend_mon_ops 2024-08-19T10:03:41.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 388159437 0 0) 0x7f6fbc0065c0 con 0x7f6fbc002860 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 10 monclient(hunting): my global_id is 194134 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd8142c40 con 0x7f6fbc002860 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.537+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4240661536 0 0) 0x7f6fd8142c40 con 0x7f6fbc002860 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fbc0065c0 con 0x7f6fbc002860 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8001320 con 0x7f6fbc002860 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 10 monclient: dump: 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.557 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.558 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 10 client.174681.objecter ms_handle_connect 0x7f6fb8041230 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2102936736 0 0) 0x7f6fbc0065c0 con 0x7f6fbc002860 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fbc00a160 con 0x7f6fbc002860 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6fc80049a0 con 0x7f6fbc002860 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6791/0 conn(0x7f6fbc002860 legacy=0x7f6fbc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.538+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6791/0 conn(0x7f6fbc002860 legacy=0x7f6fbc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.544+0000 7f6fdd508640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.544+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fbc009e70 con 0x7f6fbc009130 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fd6ffd640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6791/0 conn(0x7f6fbc009130 legacy=0x7f6fbc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 10 client.174681.objecter ms_handle_connect 0x7f6fbc009130 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 10 client.174681.objecter resend_mon_ops 2024-08-19T10:03:41.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3597389706 0 0) 0x7f6fbc009e70 con 0x7f6fbc009130 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 10 monclient(hunting): my global_id is 194143 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.546+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fbc0065c0 con 0x7f6fbc009130 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1732656724 0 0) 0x7f6fbc0065c0 con 0x7f6fbc009130 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fbc009e70 con 0x7f6fbc009130 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc80051d0 con 0x7f6fbc009130 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 10 monclient: dump: 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.560 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 945829753 0 0) 0x7f6fbc009e70 con 0x7f6fbc009130 2024-08-19T10:03:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fbc00c410 con 0x7f6fbc009130 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 4043055851) 0x7f6fc80074b0 con 0x7f6fbc009130 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6791/0 conn(0x7f6fbc009130 legacy=0x7f6fbc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fcf7fe640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6791/0 conn(0x7f6fbc009130 legacy=0x7f6fbc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:41.562 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9234, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:41.563 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.564 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.565 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:41.566 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.567 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:41.571 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:41.572 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.574 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:41.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.582 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:41.582 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:41.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 10 librados: watch_flush enter 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 10 librados: watch_flush exit 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 20 client.174681.objecter shutdown clearing up homeless session... 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 10 client.174681.objecter successfully canceled tick 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6fb8041230 legacy=0x7f6fb80436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 10 monclient: shutdown 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 >> v1:172.21.15.1:6789/0 conn(0x7f6fd813f5c0 legacy=0x7f6fd81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fd77fe640 1 -- 172.21.15.1:0/3816751036 reap_dead start 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.547+0000 7f6fd77fe640 1 -- 172.21.15.1:0/3816751036 reap_dead start 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.548+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 shutdown_connections 2024-08-19T10:03:41.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.548+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 >> 172.21.15.1:0/3816751036 conn(0x7f6fd80ac4b0 msgr2=0x7f6fd80a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.548+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 shutdown_connections 2024-08-19T10:03:41.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.548+0000 7f6fdd508640 1 -- 172.21.15.1:0/3816751036 wait complete. 2024-08-19T10:03:41.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.548+0000 7f6fdd508640 1 librados: shutdown 2024-08-19T10:03:41.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:41.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:03:41.585 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.683+0000 7f0c91314640 10 monclient: get_monmap_and_config 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.683+0000 7f0c91314640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.683+0000 7f0c91314640 10 monclient: monmap: 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:41.684718+0000 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:41.684718+0000 2024-08-19T10:03:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 1 Processor -- start 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 1 -- start start 2024-08-19T10:03:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient: init 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient: _add_conns ranks=[1,7,2,0,3,8,5,6,4] 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient(hunting): picked mon.noname-f con 0x7f0c8c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient(hunting): picked mon.noname-i con 0x7f0c8c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient(hunting): picked mon.noname-b con 0x7f0c8c13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c0a3b40 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c0a8130 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c06ede0 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.684+0000 7f0c91314640 10 monclient(hunting): authenticate will time out at 1477.760920s 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.685+0000 7f0c8b7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0c8c13f5c0 0x7f0c8c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45574/0 (socket says 172.21.15.1:45574) 2024-08-19T10:03:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.685+0000 7f0c8b7fe640 1 -- 172.21.15.1:0/3766533047 learned_addr learned my addr 172.21.15.1:0/3766533047 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c8b7fe640 1 -- 172.21.15.1:0/3766533047 >> v1:172.21.15.1:6790/0 conn(0x7f0c8c13f5c0 legacy=0x7f0c8c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806399903 0 0) 0x7f0c8c0a8130 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient(hunting): my global_id is 184295 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c74003520 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c8a7fc640 1 -- 172.21.15.1:0/3766533047 >> v1:172.21.15.82:6792/0 conn(0x7f0c8c0a6ba0 legacy=0x7f0c8c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c8affd640 1 -- 172.21.15.1:0/3766533047 >> v1:172.21.15.82:6789/0 conn(0x7f0c8c0a67c0 legacy=0x7f0c8c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2501609531 0 0) 0x7f0c74003520 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 >> v1:172.21.15.82:6792/0 conn(0x7f0c8c0a6ba0 legacy=0x7f0c8c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 >> v1:172.21.15.82:6789/0 conn(0x7f0c8c0a67c0 legacy=0x7f0c8c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 monclient: found mon.noname-b 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c0a92a0 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c91314640 5 monclient: authenticate success, global_id 184295 2024-08-19T10:03:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c84001e60 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c91314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: dump: 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c91314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c84004c20 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c91314640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 1 -- 172.21.15.1:0/3766533047 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c84005150 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c91314640 4 set_mon_vals no callback set 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.687+0000 7f0c89ffb640 10 monclient: dump: 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 10 monclient: shutdown 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 1 -- 172.21.15.1:0/3766533047 >> v1:172.21.15.1:6790/0 conn(0x7f0c8c13f5c0 legacy=0x7f0c8c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 1 -- 172.21.15.1:0/3766533047 shutdown_connections 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 1 -- 172.21.15.1:0/3766533047 >> 172.21.15.1:0/3766533047 conn(0x7f0c8c0ac4b0 msgr2=0x7f0c8c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 1 -- 172.21.15.1:0/3766533047 shutdown_connections 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 1 -- 172.21.15.1:0/3766533047 wait complete. 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.688+0000 7f0c91314640 10 monclient: monmap: 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 librados: starting msgr at 2024-08-19T10:03:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 librados: starting objecter 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 Processor -- start 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 -- start start 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 librados: setting wanted keys 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 librados: calling monclient init 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient: init 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient: _add_conns ranks=[2,8,4,5,1,7,0,6,3] 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient(hunting): picked mon.noname-b con 0x7f0c8c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient(hunting): picked mon.noname-e con 0x7f0c8c0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c13be00 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c1cf020 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c0a9c50 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c91314640 10 monclient(hunting): authenticate will time out at 1477.765908s 2024-08-19T10:03:41.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c8affd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0c8c0a67c0 0x7f0c8c1ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45580/0 (socket says 172.21.15.1:45580) 2024-08-19T10:03:41.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.689+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 learned_addr learned my addr 172.21.15.1:0/4186787367 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6790/0 conn(0x7f0c8c0a67c0 legacy=0x7f0c8c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 client.?.objecter ms_handle_connect 0x7f0c8c0a67c0 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c8b7fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6791/0 conn(0x7f0c8c13f5c0 legacy=0x7f0c8c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 client.?.objecter ms_handle_connect 0x7f0c8c13f5c0 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 client.?.objecter ms_handle_connect 0x7f0c8c0a6ba0 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c8a7fc640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6793/0 conn(0x7f0c8c0a6ba0 legacy=0x7f0c8c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3240883606 0 0) 0x7f0c8c13be00 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): my global_id is 184304 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c78004eb0 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797753060 0 0) 0x7f0c8c1cf020 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): my global_id is 194152 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c8c13be00 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710657807 0 0) 0x7f0c8c0a9c50 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): my global_id is 184364 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c8c1cf020 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2757066504 0 0) 0x7f0c8c13be00 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c8c0a9c50 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c840020b0 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): dump: 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.713 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2690827366 0 0) 0x7f0c78004eb0 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c8c13be00 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 556024106 0 0) 0x7f0c8c1cf020 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c78004eb0 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c7c003610 con 0x7f0c8c0a67c0 2024-08-19T10:03:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): dump: 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c80001c60 con 0x7f0c8c0a6ba0 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient(hunting): dump: 2024-08-19T10:03:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 284255112 0 0) 0x7f0c8c0a9c50 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6793/0 conn(0x7f0c8c0a6ba0 legacy=0x7f0c8c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6790/0 conn(0x7f0c8c0a67c0 legacy=0x7f0c8c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 monclient: found mon.c 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c13af90 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 5 monclient: authenticate success, global_id 194152 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 10 monclient: _renew_subs 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8c139aa0 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 10 client.194152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 10 monclient: _renew_subs 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c8c13a8a0 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c91314640 1 librados: init done 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c84004c20 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c84001900 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.692+0000 7f0c737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c897fa640 4 set_mon_vals no callback set 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c737fe640 10 monclient: dump: 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0c840119c0 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c91314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c6c0065c0 con 0x7f0c6c002860 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.693+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(484..484 src has 299..484) ==== 4967+0+0 (unknown 3313500721 0 0) 0x7f0c8404e300 con 0x7f0c8c13f5c0 2024-08-19T10:03:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.694+0000 7f0c737fe640 10 client.194152.objecter ms_dispatch 0x7f0c8c000b90 osd_map(484..484 src has 299..484) 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.694+0000 7f0c737fe640 3 client.194152.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.694+0000 7f0c737fe640 3 client.194152.objecter handle_osd_map decoding full epoch 484 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.694+0000 7f0c737fe640 20 client.194152.objecter dump_active .. 0 homeless 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.694+0000 7f0c8a7fc640 10 monclient: get_auth_request con 0x7f0c78041bf0 auth_method 0 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 client.194152.objecter ms_handle_connect 0x7f0c6c002860 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 client.194152.objecter resend_mon_ops 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c002860 legacy=0x7f0c6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776293939 0 0) 0x7f0c6c0065c0 con 0x7f0c6c002860 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient(hunting): my global_id is 194109 2024-08-19T10:03:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c8c0a9c50 con 0x7f0c6c002860 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1298769520 0 0) 0x7f0c8c0a9c50 con 0x7f0c6c002860 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6c0065c0 con 0x7f0c6c002860 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c7c003e10 con 0x7f0c6c002860 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 monclient: dump: 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 10 client.194152.objecter ms_handle_connect 0x7f0c78041bf0 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1109327425 0 0) 0x7f0c6c0065c0 con 0x7f0c6c002860 2024-08-19T10:03:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.696+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c6c00a160 con 0x7f0c6c002860 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.697+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0c7c0040d0 con 0x7f0c6c002860 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.697+0000 7f0c737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.697+0000 7f0c737fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.697+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c002860 legacy=0x7f0c6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.697+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c002860 legacy=0x7f0c6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.703+0000 7f0c91314640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.703+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c6c009e50 con 0x7f0c6c009110 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 legacy=0x7f0c6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 10 client.194152.objecter ms_handle_connect 0x7f0c6c009110 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 10 client.194152.objecter resend_mon_ops 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478552534 0 0) 0x7f0c6c009e50 con 0x7f0c6c009110 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 10 monclient(hunting): my global_id is 194118 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.705+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6c0065c0 con 0x7f0c6c009110 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1565986573 0 0) 0x7f0c6c0065c0 con 0x7f0c6c009110 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6c009e50 con 0x7f0c6c009110 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c7c005b00 con 0x7f0c6c009110 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 10 monclient: dump: 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 legacy=0x7f0c6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 legacy=0x7f0c6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c8affd640 1 --1- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 0x7f0c6c0095a0 :-1 s=OPENED pgs=3460 cs=1 l=1).handle_message read tag failed 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c8affd640 1 --1- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 0x7f0c6c0095a0 :-1 s=OPENED pgs=3460 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 legacy=0x7f0c6c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c6c009110 legacy=0x7f0c6c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.706+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c6c0065c0 con 0x7f0c7804b580 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c8affd640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c7804b580 legacy=0x7f0c78049fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 client.194152.objecter ms_handle_connect 0x7f0c7804b580 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 client.194152.objecter resend_mon_ops 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3129728520 0 0) 0x7f0c6c0065c0 con 0x7f0c7804b580 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient(hunting): my global_id is 194127 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c8c0a9c50 con 0x7f0c7804b580 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1457039986 0 0) 0x7f0c8c0a9c50 con 0x7f0c7804b580 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6c0065c0 con 0x7f0c7804b580 2024-08-19T10:03:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0c7c0062d0 con 0x7f0c7804b580 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 10 monclient: dump: 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1407168639 0 0) 0x7f0c6c0065c0 con 0x7f0c7804b580 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.708+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c7c0040d0 con 0x7f0c7804b580 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 2943631805) 0x7f0c7c006530 con 0x7f0c7804b580 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:41.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c737fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c7804b580 legacy=0x7f0c78049fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c8b7fe640 1 -- 172.21.15.1:0/4186787367 reap_dead start 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c737fe640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6792/0 conn(0x7f0c7804b580 legacy=0x7f0c78049fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9396, 2024-08-19T10:03:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:41.746 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:41.748 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:41.750 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.751 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.752 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:41.753 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.754 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.756 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:41.757 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:41.758 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:41.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 10 librados: watch_flush enter 2024-08-19T10:03:41.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 10 librados: watch_flush exit 2024-08-19T10:03:41.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 20 client.194152.objecter shutdown clearing up homeless session... 2024-08-19T10:03:41.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 10 client.194152.objecter successfully canceled tick 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0c78041bf0 legacy=0x7f0c780440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 10 monclient: shutdown 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 >> v1:172.21.15.1:6791/0 conn(0x7f0c8c13f5c0 legacy=0x7f0c8c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 shutdown_connections 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 >> 172.21.15.1:0/4186787367 conn(0x7f0c8c0ac4b0 msgr2=0x7f0c8c13eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 shutdown_connections 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 -- 172.21.15.1:0/4186787367 wait complete. 2024-08-19T10:03:41.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.709+0000 7f0c91314640 1 librados: shutdown 2024-08-19T10:03:41.760 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:41.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:03:41.761 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:03:41.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.860+0000 7fe54b8c5640 10 monclient: get_monmap_and_config 2024-08-19T10:03:41.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.860+0000 7fe54b8c5640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.860+0000 7fe54b8c5640 10 monclient: monmap: 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:41.861156+0000 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:41.861156+0000 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.862 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.863 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.860+0000 7fe54b8c5640 1 Processor -- start 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.860+0000 7fe54b8c5640 1 -- start start 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.860+0000 7fe54b8c5640 10 monclient: init 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient: _add_conns ranks=[0,4,2,8,1,7,6,5,3] 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient(hunting): picked mon.noname-a con 0x7fe5440a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient(hunting): picked mon.noname-c con 0x7fe5440694b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient(hunting): picked mon.noname-b con 0x7fe5440a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5440a3c70 con 0x7fe5440a6830 2024-08-19T10:03:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5440a33d0 con 0x7fe5440a46a0 2024-08-19T10:03:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5440a36b0 con 0x7fe5440694b0 2024-08-19T10:03:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe54b8c5640 10 monclient(hunting): authenticate will time out at 1477.937504s 2024-08-19T10:03:41.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe549661640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe5440a6830 0x7fe5440a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57198/0 (socket says 172.21.15.1:57198) 2024-08-19T10:03:41.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe549661640 1 -- 172.21.15.1:0/315442600 learned_addr learned my addr 172.21.15.1:0/315442600 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.861+0000 7fe549e62640 1 --1- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6790/0 conn(0x7fe5440a46a0 0x7fe544141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45594/0 (socket says 172.21.15.1:45594) 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe549661640 1 -- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6789/0 conn(0x7fe5440a6830 legacy=0x7fe5440a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe548e60640 1 -- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6791/0 conn(0x7fe5440694b0 legacy=0x7fe5440a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe549e62640 1 -- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6790/0 conn(0x7fe5440a46a0 legacy=0x7fe544141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736028871 0 0) 0x7fe5440a3c70 con 0x7fe5440a6830 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe543fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe543fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe543fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe543fff640 10 monclient(hunting): my global_id is 174690 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.863+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe52c003520 con 0x7fe5440a6830 2024-08-19T10:03:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 785015553 0 0) 0x7fe52c003520 con 0x7fe5440a6830 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6791/0 conn(0x7fe5440694b0 legacy=0x7fe5440a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6790/0 conn(0x7fe5440a46a0 legacy=0x7fe544141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 monclient: found mon.noname-a 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54406dbb0 con 0x7fe5440a6830 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 5 monclient: authenticate success, global_id 174690 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe53c001ae0 con 0x7fe5440a6830 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: dump: 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.875 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe53c0048a0 con 0x7fe5440a6830 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 1 -- 172.21.15.1:0/315442600 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe53c004dd0 con 0x7fe5440a6830 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 4 set_mon_vals no callback set 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe543fff640 10 monclient: dump: 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 10 monclient: shutdown 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 1 -- 172.21.15.1:0/315442600 >> v1:172.21.15.1:6789/0 conn(0x7fe5440a6830 legacy=0x7fe5440a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 1 -- 172.21.15.1:0/315442600 shutdown_connections 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.864+0000 7fe54b8c5640 1 -- 172.21.15.1:0/315442600 >> 172.21.15.1:0/315442600 conn(0x7fe5440a9020 msgr2=0x7fe544068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 -- 172.21.15.1:0/315442600 shutdown_connections 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 -- 172.21.15.1:0/315442600 wait complete. 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 10 monclient: build_initial_monmap 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 10 monclient: monmap: 2024-08-19T10:03:41.878 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:41.879 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 librados: starting msgr at 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 librados: starting objecter 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 Processor -- start 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 -- start start 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 librados: setting wanted keys 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 1 librados: calling monclient init 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.865+0000 7fe54b8c5640 10 monclient: init 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:41.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient: _add_conns ranks=[5,0,1,7,3,4,6,2,8] 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient(hunting): picked mon.noname-h con 0x7fe5440694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient(hunting): picked mon.noname-a con 0x7fe5440a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient(hunting): picked mon.noname-f con 0x7fe5440a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54413a480 con 0x7fe5440a46a0 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe544142cd0 con 0x7fe5440a6830 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5441382d0 con 0x7fe5440694b0 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe54b8c5640 10 monclient(hunting): authenticate will time out at 1477.942478s 2024-08-19T10:03:41.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe548e60640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe5440a46a0 0x7fe5441d3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57212/0 (socket says 172.21.15.1:57212) 2024-08-19T10:03:41.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.866+0000 7fe548e60640 1 -- 172.21.15.1:0/4023111031 learned_addr learned my addr 172.21.15.1:0/4023111031 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:41.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe548e60640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6789/0 conn(0x7fe5440a46a0 legacy=0x7fe5441d3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 client.?.objecter ms_handle_connect 0x7fe5440a46a0 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3484721459 0 0) 0x7fe54413a480 con 0x7fe5440a46a0 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 monclient(hunting): my global_id is 174699 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe530004eb0 con 0x7fe5440a46a0 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe549661640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.82:6791/0 conn(0x7fe5440694b0 legacy=0x7fe5441ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 client.?.objecter ms_handle_connect 0x7fe5440694b0 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 client.?.objecter ms_handle_connect 0x7fe5440a6830 2024-08-19T10:03:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe541ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.868+0000 7fe549e62640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.82:6789/0 conn(0x7fe5440a6830 legacy=0x7fe5441d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 287841739 0 0) 0x7fe530004eb0 con 0x7fe5440a46a0 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe54413a480 con 0x7fe5440a46a0 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe538003610 con 0x7fe5440a46a0 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient(hunting): dump: 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3248323012 0 0) 0x7fe54413a480 con 0x7fe5440a46a0 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.82:6791/0 conn(0x7fe5440694b0 legacy=0x7fe5441ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.82:6789/0 conn(0x7fe5440a6830 legacy=0x7fe5441d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 monclient: found mon.a 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54413ad00 con 0x7fe5440a46a0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: _finish_auth 0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 5 monclient: authenticate success, global_id 174699 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 10 monclient: _renew_subs 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe544138e30 con 0x7fe5440a46a0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe538003bd0 con 0x7fe5440a46a0 2024-08-19T10:03:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe538004100 con 0x7fe5440a46a0 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe541ffb640 10 monclient: dump: 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 10 client.174699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 10 monclient: _renew_subs 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe544137870 con 0x7fe5440a46a0 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe5437fe640 4 set_mon_vals no callback set 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.869+0000 7fe54b8c5640 1 librados: init done 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe5380042a0 con 0x7fe5440a46a0 2024-08-19T10:03:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe54b8c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe54413a6a0 con 0x7fe544143180 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(484..484 src has 299..484) ==== 4967+0+0 (unknown 3313500721 0 0) 0x7fe53804e100 con 0x7fe5440a46a0 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe541ffb640 10 client.174699.objecter ms_dispatch 0x7fe544000b90 osd_map(484..484 src has 299..484) 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe541ffb640 3 client.174699.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe541ffb640 3 client.174699.objecter handle_osd_map decoding full epoch 484 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe541ffb640 20 client.174699.objecter dump_active .. 0 homeless 2024-08-19T10:03:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.870+0000 7fe549e62640 10 monclient: get_auth_request con 0x7fe530040fd0 auth_method 0 2024-08-19T10:03:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.872+0000 7fe549661640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6793/0 conn(0x7fe544143180 legacy=0x7fe5440a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.872+0000 7fe541ffb640 10 client.174699.objecter ms_handle_connect 0x7fe544143180 2024-08-19T10:03:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.872+0000 7fe541ffb640 10 client.174699.objecter resend_mon_ops 2024-08-19T10:03:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:41.873+0000 7fe541ffb640 10 client.174699.objecter ms_handle_connect 0x7fe530040fd0 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1392516209 0 0) 0x7fe54413a6a0 con 0x7fe544143180 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient(hunting): my global_id is 184373 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe54413a480 con 0x7fe544143180 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 625629799 0 0) 0x7fe54413a480 con 0x7fe544143180 2024-08-19T10:03:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe54413a6a0 con 0x7fe544143180 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe53c001320 con 0x7fe544143180 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 10 monclient: dump: 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2893312036 0 0) 0x7fe54413a6a0 con 0x7fe544143180 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.329+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe54413ab00 con 0x7fe544143180 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.330+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe53c004b20 con 0x7fe544143180 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.330+0000 7fe541ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.330+0000 7fe541ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.330+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6793/0 conn(0x7fe544143180 legacy=0x7fe5440a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.330+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6793/0 conn(0x7fe544143180 legacy=0x7fe5440a0f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.337+0000 7fe54b8c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.337+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe544137bb0 con 0x7fe5441ced80 2024-08-19T10:03:42.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe549661640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6793/0 conn(0x7fe5441ced80 legacy=0x7fe5440a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 client.174699.objecter ms_handle_connect 0x7fe5441ced80 2024-08-19T10:03:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 client.174699.objecter resend_mon_ops 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3351922474 0 0) 0x7fe544137bb0 con 0x7fe5441ced80 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient(hunting): my global_id is 184382 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe54413a6a0 con 0x7fe5441ced80 2024-08-19T10:03:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1159183520 0 0) 0x7fe54413a6a0 con 0x7fe5441ced80 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe544137bb0 con 0x7fe5441ced80 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe53c001d30 con 0x7fe5441ced80 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.340+0000 7fe541ffb640 10 monclient: dump: 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2216874741 0 0) 0x7fe544137bb0 con 0x7fe5441ced80 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe544072fb0 con 0x7fe5441ced80 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 203980035) 0x7fe53c007400 con 0x7fe5441ced80 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6793/0 conn(0x7fe5441ced80 legacy=0x7fe5440a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe541ffb640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6793/0 conn(0x7fe5441ced80 legacy=0x7fe5440a1690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 139590, 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:42.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.346 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.347 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:42.348 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:42.349 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.350 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.351 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:42.365 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe54b8c5640 10 librados: watch_flush enter 2024-08-19T10:03:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe54b8c5640 10 librados: watch_flush exit 2024-08-19T10:03:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe54b8c5640 20 client.174699.objecter shutdown clearing up homeless session... 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.341+0000 7fe54b8c5640 10 client.174699.objecter successfully canceled tick 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe530040fd0 legacy=0x7fe530043490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 10 monclient: shutdown 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 >> v1:172.21.15.1:6789/0 conn(0x7fe5440a46a0 legacy=0x7fe5441d3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe549e62640 1 -- 172.21.15.1:0/4023111031 reap_dead start 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe549e62640 1 -- 172.21.15.1:0/4023111031 reap_dead start 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 shutdown_connections 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 >> 172.21.15.1:0/4023111031 conn(0x7fe5440a9020 msgr2=0x7fe5440a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 shutdown_connections 2024-08-19T10:03:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 -- 172.21.15.1:0/4023111031 wait complete. 2024-08-19T10:03:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.342+0000 7fe54b8c5640 1 librados: shutdown 2024-08-19T10:03:42.368 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:42.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:03:42.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.467+0000 7ff1176b3640 10 monclient: get_monmap_and_config 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.467+0000 7ff1176b3640 10 monclient: build_initial_monmap 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.467+0000 7ff1176b3640 10 monclient: monmap: 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:42.468661+0000 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:42.468661+0000 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 1 Processor -- start 2024-08-19T10:03:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 1 -- start start 2024-08-19T10:03:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 10 monclient: init 2024-08-19T10:03:42.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 10 monclient: _add_conns ranks=[0,4,8,3,1,7,6,2,5] 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 10 monclient(hunting): picked mon.noname-a con 0x7ff1100a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 10 monclient(hunting): picked mon.noname-c con 0x7ff1100a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 10 monclient(hunting): picked mon.noname-e con 0x7ff11013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.468+0000 7ff1176b3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1100a3b40 con 0x7ff1100a67c0 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff1176b3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1100a8130 con 0x7ff1100a6ba0 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff1176b3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff11006ede0 con 0x7ff11013f5c0 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff1176b3640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff1176b3640 10 monclient(hunting): authenticate will time out at 1478.545045s 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff115c50640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff11013f5c0 0x7ff1101419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41768/0 (socket says 172.21.15.1:41768) 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff115c50640 1 -- 172.21.15.1:0/2795830622 learned_addr learned my addr 172.21.15.1:0/2795830622 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.469+0000 7ff11544f640 1 --1- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6789/0 conn(0x7ff1100a67c0 0x7ff1100a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57216/0 (socket says 172.21.15.1:57216) 2024-08-19T10:03:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff115c50640 1 -- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6793/0 conn(0x7ff11013f5c0 legacy=0x7ff1101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff114c4e640 1 -- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6791/0 conn(0x7ff1100a6ba0 legacy=0x7ff1100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff11544f640 1 -- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6789/0 conn(0x7ff1100a67c0 legacy=0x7ff1100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594226046 0 0) 0x7ff11006ede0 con 0x7ff11013f5c0 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): my global_id is 184391 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0fc003520 con 0x7ff11013f5c0 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966605936 0 0) 0x7ff1100a3b40 con 0x7ff1100a67c0 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): my global_id is 174708 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff11006ede0 con 0x7ff1100a67c0 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569794052 0 0) 0x7ff1100a8130 con 0x7ff1100a6ba0 2024-08-19T10:03:42.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient(hunting): my global_id is 194161 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1100a3b40 con 0x7ff1100a6ba0 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2698831316 0 0) 0x7ff1100a3b40 con 0x7ff1100a6ba0 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6793/0 conn(0x7ff11013f5c0 legacy=0x7ff1101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6789/0 conn(0x7ff1100a67c0 legacy=0x7ff1100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: _finish_hunting 0 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 monclient: found mon.noname-c 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1100a92a0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: _finish_auth 0 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: _check_auth_tickets 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff100001a50 con 0x7ff1100a6ba0 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: dump: 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff1176b3640 5 monclient: authenticate success, global_id 194161 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff1176b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff100001ca0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 1 -- 172.21.15.1:0/2795830622 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff100003950 con 0x7ff1100a6ba0 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff107fff640 10 monclient: dump: 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.471+0000 7ff1176b3640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 4 set_mon_vals no callback set 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 10 monclient: shutdown 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 1 -- 172.21.15.1:0/2795830622 >> v1:172.21.15.1:6791/0 conn(0x7ff1100a6ba0 legacy=0x7ff1100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 1 -- 172.21.15.1:0/2795830622 shutdown_connections 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 1 -- 172.21.15.1:0/2795830622 >> 172.21.15.1:0/2795830622 conn(0x7ff1100ac4b0 msgr2=0x7ff1100a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 1 -- 172.21.15.1:0/2795830622 shutdown_connections 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.472+0000 7ff1176b3640 1 -- 172.21.15.1:0/2795830622 wait complete. 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient: build_initial_monmap 2024-08-19T10:03:42.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient: monmap: 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 librados: starting msgr at 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 librados: starting objecter 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 Processor -- start 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 -- start start 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 librados: setting wanted keys 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 librados: calling monclient init 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient: init 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient: _add_conns ranks=[6,5,3,0,2,4,1,8,7] 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient(hunting): picked mon.noname-d con 0x7ff1100a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:42.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient(hunting): picked mon.noname-h con 0x7ff1100a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient(hunting): picked mon.noname-g con 0x7ff11013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff11013f3f0 con 0x7ff11013f5c0 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1100a1620 con 0x7ff1100a6ba0 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff11013a1c0 con 0x7ff1100a67c0 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.473+0000 7ff1176b3640 10 monclient(hunting): authenticate will time out at 1478.549959s 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.474+0000 7ff11544f640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ff1100a67c0 0x7ff11013ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43782/0 (socket says 172.21.15.1:43782) 2024-08-19T10:03:42.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.474+0000 7ff11544f640 1 -- 172.21.15.1:0/2203138021 learned_addr learned my addr 172.21.15.1:0/2203138021 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff11544f640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.1:6792/0 conn(0x7ff1100a67c0 legacy=0x7ff11013ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 client.?.objecter ms_handle_connect 0x7ff1100a67c0 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 client.?.objecter ms_handle_connect 0x7ff11013f5c0 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff115c50640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6790/0 conn(0x7ff11013f5c0 legacy=0x7ff1101d6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 client.?.objecter ms_handle_connect 0x7ff1100a6ba0 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff114c4e640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6791/0 conn(0x7ff1100a6ba0 legacy=0x7ff1101d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585786108 0 0) 0x7ff1100a1620 con 0x7ff1100a6ba0 2024-08-19T10:03:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): my global_id is 27599 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f4004eb0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161450035 0 0) 0x7ff11013f3f0 con 0x7ff11013f5c0 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): my global_id is 184332 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1100a1620 con 0x7ff11013f5c0 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1754423856 0 0) 0x7ff0f4004eb0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff11013f3f0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 525414980 0 0) 0x7ff1100a1620 con 0x7ff11013f5c0 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0f4004eb0 con 0x7ff11013f5c0 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff100003950 con 0x7ff1100a6ba0 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.476+0000 7ff105ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient(hunting): dump: 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff1080020a0 con 0x7ff11013f5c0 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient(hunting): dump: 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2893709874 0 0) 0x7ff11013f3f0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.1:6792/0 conn(0x7ff1100a67c0 legacy=0x7ff11013ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6790/0 conn(0x7ff11013f5c0 legacy=0x7ff1101d6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 monclient: found mon.h 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff110139ee0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: _finish_auth 0 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 5 monclient: authenticate success, global_id 27599 2024-08-19T10:03:42.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 10 monclient: _renew_subs 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff110139ab0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff100003f80 con 0x7ff1100a6ba0 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff1000057c0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff105ffb640 10 monclient: dump: 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1077fe640 4 set_mon_vals no callback set 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 10 client.27599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 10 monclient: _renew_subs 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1100a2360 con 0x7ff1100a6ba0 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.477+0000 7ff1176b3640 1 librados: init done 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff100011c20 con 0x7ff1100a6ba0 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff1176b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff11013a3e0 con 0x7ff1101cb810 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(485..485 src has 299..485) ==== 5079+0+0 (unknown 223647928 0 0) 0x7ff10004dfa0 con 0x7ff1100a6ba0 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff105ffb640 10 client.27599.objecter ms_dispatch 0x7ff110000b90 osd_map(485..485 src has 299..485) 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff105ffb640 3 client.27599.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff105ffb640 3 client.27599.objecter handle_osd_map decoding full epoch 485 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff105ffb640 20 client.27599.objecter dump_active .. 0 homeless 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.478+0000 7ff115c50640 10 monclient: get_auth_request con 0x7ff0f4041500 auth_method 0 2024-08-19T10:03:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 10 client.27599.objecter ms_handle_connect 0x7ff1101cb810 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff11544f640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6789/0 conn(0x7ff1101cb810 legacy=0x7ff11013fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 10 client.27599.objecter resend_mon_ops 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863507509 0 0) 0x7ff11013a3e0 con 0x7ff1101cb810 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 10 monclient(hunting): my global_id is 174583 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.480+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff11013f3f0 con 0x7ff1101cb810 2024-08-19T10:03:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 10 client.27599.objecter ms_handle_connect 0x7ff0f4041500 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1173747573 0 0) 0x7ff11013f3f0 con 0x7ff1101cb810 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff11013a3e0 con 0x7ff1101cb810 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff10c001ab0 con 0x7ff1101cb810 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 10 monclient: dump: 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1966091189 0 0) 0x7ff11013a3e0 con 0x7ff1101cb810 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1100a10a0 con 0x7ff1101cb810 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff10c0049a0 con 0x7ff1101cb810 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:42.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:42.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6789/0 conn(0x7ff1101cb810 legacy=0x7ff11013fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.481+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6789/0 conn(0x7ff1101cb810 legacy=0x7ff11013fbe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:42.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.488+0000 7ff1176b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:42.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.488+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff110068f60 con 0x7ff11002e1d0 2024-08-19T10:03:42.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff11544f640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6789/0 conn(0x7ff11002e1d0 legacy=0x7ff1101402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 10 client.27599.objecter ms_handle_connect 0x7ff11002e1d0 2024-08-19T10:03:42.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 10 client.27599.objecter resend_mon_ops 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550595472 0 0) 0x7ff110068f60 con 0x7ff11002e1d0 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 10 monclient(hunting): my global_id is 174592 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff11013a3e0 con 0x7ff11002e1d0 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3712030264 0 0) 0x7ff11013a3e0 con 0x7ff11002e1d0 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.491+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff110068f60 con 0x7ff11002e1d0 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff10c005190 con 0x7ff11002e1d0 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:03:42.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 10 monclient: dump: 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.505 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2572139798 0 0) 0x7ff110068f60 con 0x7ff11002e1d0 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1101cbbf0 con 0x7ff11002e1d0 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2554219597) 0x7ff10c0074f0 con 0x7ff11002e1d0 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6789/0 conn(0x7ff11002e1d0 legacy=0x7ff1101402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff105ffb640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6789/0 conn(0x7ff11002e1d0 legacy=0x7ff1101402f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:42.506 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 272551, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:42.526 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff1176b3640 10 librados: watch_flush enter 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff1176b3640 10 librados: watch_flush exit 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff1176b3640 20 client.27599.objecter shutdown clearing up homeless session... 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.492+0000 7ff1176b3640 10 client.27599.objecter successfully canceled tick 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff0f4041500 legacy=0x7ff0f40439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 10 monclient: shutdown 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 >> v1:172.21.15.82:6791/0 conn(0x7ff1100a6ba0 legacy=0x7ff1101d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff115c50640 1 -- 172.21.15.1:0/2203138021 reap_dead start 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff115c50640 1 -- 172.21.15.1:0/2203138021 reap_dead start 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 shutdown_connections 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 >> 172.21.15.1:0/2203138021 conn(0x7ff1100ac4b0 msgr2=0x7ff1100a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 shutdown_connections 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 -- 172.21.15.1:0/2203138021 wait complete. 2024-08-19T10:03:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.493+0000 7ff1176b3640 1 librados: shutdown 2024-08-19T10:03:42.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:42.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:03:42.530 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:03:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.628+0000 7f6009f4c640 10 monclient: get_monmap_and_config 2024-08-19T10:03:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.628+0000 7f6009f4c640 10 monclient: build_initial_monmap 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.628+0000 7f6009f4c640 10 monclient: monmap: 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:42.629618+0000 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:42.629618+0000 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:42.631 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:42.631 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:42.631 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:42.631 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:42.631 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.629+0000 7f6009f4c640 1 Processor -- start 2024-08-19T10:03:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.629+0000 7f6009f4c640 1 -- start start 2024-08-19T10:03:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.629+0000 7f6009f4c640 10 monclient: init 2024-08-19T10:03:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.629+0000 7f6009f4c640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.629+0000 7f6009f4c640 10 monclient: _add_conns ranks=[2,7,4,3,8,5,6,1,0] 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 10 monclient(hunting): picked mon.noname-b con 0x7f60040a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 10 monclient(hunting): picked mon.noname-i con 0x7f60040a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 10 monclient(hunting): picked mon.noname-c con 0x7f600413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60040a3b40 con 0x7f60040a67c0 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60040a8130 con 0x7f600413f5c0 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600406ede0 con 0x7f60040a6ba0 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f6009f4c640 10 monclient(hunting): authenticate will time out at 1478.706116s 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f60037fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f60040a67c0 0x7f60040a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45608/0 (socket says 172.21.15.1:45608) 2024-08-19T10:03:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.630+0000 7f60037fe640 1 -- 172.21.15.1:0/1143953161 learned_addr learned my addr 172.21.15.1:0/1143953161 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:42.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6003fff640 1 -- 172.21.15.1:0/1143953161 >> v1:172.21.15.1:6791/0 conn(0x7f600413f5c0 legacy=0x7f60041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60037fe640 1 -- 172.21.15.1:0/1143953161 >> v1:172.21.15.1:6790/0 conn(0x7f60040a67c0 legacy=0x7f60040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6002ffd640 1 -- 172.21.15.1:0/1143953161 >> v1:172.21.15.82:6792/0 conn(0x7f60040a6ba0 legacy=0x7f60040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405062721 0 0) 0x7f60040a8130 con 0x7f600413f5c0 2024-08-19T10:03:42.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): my global_id is 194170 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec003520 con 0x7f600413f5c0 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586060853 0 0) 0x7f60040a3b40 con 0x7f60040a67c0 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): my global_id is 184313 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60040a8130 con 0x7f60040a67c0 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156082153 0 0) 0x7f600406ede0 con 0x7f60040a6ba0 2024-08-19T10:03:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient(hunting): my global_id is 145114 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60040a3b40 con 0x7f60040a6ba0 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 777578727 0 0) 0x7f60040a8130 con 0x7f60040a67c0 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 >> v1:172.21.15.82:6792/0 conn(0x7f60040a6ba0 legacy=0x7f60040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 >> v1:172.21.15.1:6791/0 conn(0x7f600413f5c0 legacy=0x7f60041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: _finish_hunting 0 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 monclient: found mon.noname-b 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60040a92a0 con 0x7f60040a67c0 2024-08-19T10:03:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: _finish_auth 0 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: _check_auth_tickets 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ff40019f0 con 0x7f60040a67c0 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6009f4c640 5 monclient: authenticate success, global_id 184313 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6009f4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: dump: 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6009f4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ff4001cc0 con 0x7f60040a67c0 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 1 -- 172.21.15.1:0/1143953161 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ff4004be0 con 0x7f60040a67c0 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6009f4c640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f6009f4c640 4 set_mon_vals no callback set 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.632+0000 7f60027fc640 10 monclient: dump: 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 10 monclient: shutdown 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 1 -- 172.21.15.1:0/1143953161 >> v1:172.21.15.1:6790/0 conn(0x7f60040a67c0 legacy=0x7f60040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 1 -- 172.21.15.1:0/1143953161 shutdown_connections 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 1 -- 172.21.15.1:0/1143953161 >> 172.21.15.1:0/1143953161 conn(0x7f60040ac4b0 msgr2=0x7f60040a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 1 -- 172.21.15.1:0/1143953161 shutdown_connections 2024-08-19T10:03:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.633+0000 7f6009f4c640 1 -- 172.21.15.1:0/1143953161 wait complete. 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient: build_initial_monmap 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient: monmap: 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 1 librados: starting msgr at 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 1 librados: starting objecter 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 1 Processor -- start 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 1 -- start start 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 1 librados: setting wanted keys 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 1 librados: calling monclient init 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient: init 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient: _add_conns ranks=[5,1,2,4,7,6,8,3,0] 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient(hunting): picked mon.noname-h con 0x7f60040a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient(hunting): picked mon.noname-f con 0x7f60040a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.634+0000 7f6009f4c640 10 monclient(hunting): picked mon.noname-b con 0x7f600413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6009f4c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60040a28b0 con 0x7f60040a6ba0 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6009f4c640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6004142c80 con 0x7f600413f5c0 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6009f4c640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60040a16e0 con 0x7f60040a67c0 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6009f4c640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6009f4c640 10 monclient(hunting): authenticate will time out at 1478.711049s 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6003fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f600413f5c0 0x7f60041d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45616/0 (socket says 172.21.15.1:45616) 2024-08-19T10:03:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.635+0000 7f6003fff640 1 -- 172.21.15.1:0/2753548792 learned_addr learned my addr 172.21.15.1:0/2753548792 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f6003fff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.1:6790/0 conn(0x7f600413f5c0 legacy=0x7f60041d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f600413f5c0 2024-08-19T10:03:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744046204 0 0) 0x7f6004142c80 con 0x7f600413f5c0 2024-08-19T10:03:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 10 monclient(hunting): my global_id is 184322 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff0004eb0 con 0x7f600413f5c0 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f6002ffd640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6789/0 conn(0x7f60040a6ba0 legacy=0x7f60040a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.637+0000 7f60037fe640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6791/0 conn(0x7f60040a67c0 legacy=0x7f600409fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f60040a6ba0 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f60040a67c0 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1102776348 0 0) 0x7f5ff0004eb0 con 0x7f600413f5c0 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6004142c80 con 0x7f600413f5c0 2024-08-19T10:03:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518398339 0 0) 0x7f60040a16e0 con 0x7f60040a67c0 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): my global_id is 27608 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff0004eb0 con 0x7f60040a67c0 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914394062 0 0) 0x7f60040a28b0 con 0x7f60040a6ba0 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): my global_id is 174601 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60040a16e0 con 0x7f60040a6ba0 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ffc003cd0 con 0x7f600413f5c0 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:42.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): dump: 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 221203123 0 0) 0x7f6004142c80 con 0x7f600413f5c0 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6791/0 conn(0x7f60040a67c0 legacy=0x7f600409fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6789/0 conn(0x7f60040a6ba0 legacy=0x7f60040a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: _finish_hunting 0 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 monclient: found mon.b 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600413a970 con 0x7f600413f5c0 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: _finish_auth 0 2024-08-19T10:03:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: _check_auth_tickets 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 5 monclient: authenticate success, global_id 184322 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 10 monclient: _renew_subs 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f600413a0d0 con 0x7f600413f5c0 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 10 client.184322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 10 monclient: _renew_subs 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f60040a2240 con 0x7f600413f5c0 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6009f4c640 1 librados: init done 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ffc003e70 con 0x7f600413f5c0 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ffc0043a0 con 0x7f600413f5c0 2024-08-19T10:03:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f6001ffb640 4 set_mon_vals no callback set 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.639+0000 7f5febfff640 10 monclient: dump: 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f6009f4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fe40065c0 con 0x7f5fe4002860 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5ffc004540 con 0x7f600413f5c0 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(485..485 src has 299..485) ==== 5079+0+0 (unknown 223647928 0 0) 0x7f5ffc04e3d0 con 0x7f600413f5c0 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f5febfff640 10 client.184322.objecter ms_dispatch 0x7f6004000b90 osd_map(485..485 src has 299..485) 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f5febfff640 3 client.184322.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.640+0000 7f5febfff640 3 client.184322.objecter handle_osd_map decoding full epoch 485 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.641+0000 7f5febfff640 20 client.184322.objecter dump_active .. 0 homeless 2024-08-19T10:03:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.641+0000 7f6002ffd640 10 monclient: get_auth_request con 0x7f5ff0041230 auth_method 0 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 client.184322.objecter ms_handle_connect 0x7f5fe4002860 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f60037fe640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6790/0 conn(0x7f5fe4002860 legacy=0x7f5fe4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 client.184322.objecter resend_mon_ops 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513106575 0 0) 0x7f5fe40065c0 con 0x7f5fe4002860 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient(hunting): my global_id is 184341 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6004142c80 con 0x7f5fe4002860 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2290519709 0 0) 0x7f6004142c80 con 0x7f5fe4002860 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fe40065c0 con 0x7f5fe4002860 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ff4004ba0 con 0x7f5fe4002860 2024-08-19T10:03:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 monclient: dump: 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 191444810 0 0) 0x7f5fe40065c0 con 0x7f5fe4002860 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fe400a160 con 0x7f5fe4002860 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.643+0000 7f5febfff640 10 client.184322.objecter ms_handle_connect 0x7f5ff0041230 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.644+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5ff4001560 con 0x7f5fe4002860 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.644+0000 7f5febfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.644+0000 7f5febfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.644+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6790/0 conn(0x7f5fe4002860 legacy=0x7f5fe4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.644+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6790/0 conn(0x7f5fe4002860 legacy=0x7f5fe4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.650+0000 7f6009f4c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.650+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fe4009e70 con 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f60037fe640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6790/0 conn(0x7f5fe4009130 legacy=0x7f5fe40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 10 client.184322.objecter ms_handle_connect 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 10 client.184322.objecter resend_mon_ops 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3742109961 0 0) 0x7f5fe4009e70 con 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 10 monclient(hunting): my global_id is 184350 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.652+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fe40065c0 con 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 146164974 0 0) 0x7f5fe40065c0 con 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fe4009e70 con 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5ff40019d0 con 0x7f5fe4009130 2024-08-19T10:03:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 10 monclient: dump: 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3922795836 0 0) 0x7f5fe4009e70 con 0x7f5fe4009130 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.653+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5fe400c410 con 0x7f5fe4009130 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 2628359742) 0x7f5ff4007520 con 0x7f5fe4009130 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f5febfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f5febfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6790/0 conn(0x7f5fe4009130 legacy=0x7f5fe40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f5febfff640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6790/0 conn(0x7f5fe4009130 legacy=0x7f5fe40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 139908, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:42.671 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:42.674 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:42.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.679 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:42.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:42.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.683 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:42.686 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:42.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 10 librados: watch_flush enter 2024-08-19T10:03:42.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 10 librados: watch_flush exit 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 20 client.184322.objecter shutdown clearing up homeless session... 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 10 client.184322.objecter successfully canceled tick 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5ff0041230 legacy=0x7f5ff00436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 10 monclient: shutdown 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 >> v1:172.21.15.1:6790/0 conn(0x7f600413f5c0 legacy=0x7f60041d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6003fff640 1 -- 172.21.15.1:0/2753548792 reap_dead start 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6003fff640 1 -- 172.21.15.1:0/2753548792 reap_dead start 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 shutdown_connections 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 >> 172.21.15.1:0/2753548792 conn(0x7f60040ac4b0 msgr2=0x7f60040a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 shutdown_connections 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 -- 172.21.15.1:0/2753548792 wait complete. 2024-08-19T10:03:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.654+0000 7f6009f4c640 1 librados: shutdown 2024-08-19T10:03:42.690 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:42.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:03:42.690 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:03:42.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.789+0000 7f5c229cf640 10 monclient: get_monmap_and_config 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.789+0000 7f5c229cf640 10 monclient: build_initial_monmap 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.789+0000 7f5c229cf640 10 monclient: monmap: 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:42.790324+0000 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:42.790324+0000 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:42.792 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 1 Processor -- start 2024-08-19T10:03:42.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 1 -- start start 2024-08-19T10:03:42.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient: init 2024-08-19T10:03:42.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:42.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient: _add_conns ranks=[6,7,1,5,3,0,8,2,4] 2024-08-19T10:03:42.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient(hunting): picked mon.noname-d con 0x7f5c1c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient(hunting): picked mon.noname-i con 0x7f5c1c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient(hunting): picked mon.noname-f con 0x7f5c1c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c0a3b40 con 0x7f5c1c13f5c0 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c0a8130 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c06ede0 con 0x7f5c1c0a6ba0 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c229cf640 10 monclient(hunting): authenticate will time out at 1478.866656s 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c1bfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f5c1c0a67c0 0x7f5c1c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43786/0 (socket says 172.21.15.1:43786) 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.790+0000 7f5c1bfff640 1 -- 172.21.15.1:0/3978128091 learned_addr learned my addr 172.21.15.1:0/3978128091 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.792+0000 7f5c1bfff640 1 -- 172.21.15.1:0/3978128091 >> v1:172.21.15.1:6792/0 conn(0x7f5c1c0a67c0 legacy=0x7f5c1c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.792+0000 7f5c1b7fe640 1 -- 172.21.15.1:0/3978128091 >> v1:172.21.15.82:6792/0 conn(0x7f5c1c0a6ba0 legacy=0x7f5c1c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c20f6c640 1 -- 172.21.15.1:0/3978128091 >> v1:172.21.15.82:6789/0 conn(0x7f5c1c13f5c0 legacy=0x7f5c1c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255256833 0 0) 0x7f5c1c0a8130 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient(hunting): my global_id is 194136 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c04003520 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 589422781 0 0) 0x7f5c04003520 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 >> v1:172.21.15.82:6792/0 conn(0x7f5c1c0a6ba0 legacy=0x7f5c1c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 >> v1:172.21.15.82:6789/0 conn(0x7f5c1c13f5c0 legacy=0x7f5c1c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 monclient: found mon.noname-d 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:03:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1c0a92a0 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: _finish_auth 0 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c0c001ae0 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 5 monclient: authenticate success, global_id 194136 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: dump: 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c0c0048a0 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 1 -- 172.21.15.1:0/3978128091 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c0c004dd0 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 4 set_mon_vals no callback set 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c1affd640 10 monclient: dump: 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 10 monclient: shutdown 2024-08-19T10:03:42.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 1 -- 172.21.15.1:0/3978128091 >> v1:172.21.15.1:6792/0 conn(0x7f5c1c0a67c0 legacy=0x7f5c1c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 1 -- 172.21.15.1:0/3978128091 shutdown_connections 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.793+0000 7f5c229cf640 1 -- 172.21.15.1:0/3978128091 >> 172.21.15.1:0/3978128091 conn(0x7f5c1c0ac4b0 msgr2=0x7f5c1c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 1 -- 172.21.15.1:0/3978128091 shutdown_connections 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 1 -- 172.21.15.1:0/3978128091 wait complete. 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 10 monclient: build_initial_monmap 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 10 monclient: monmap: 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:42.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 1 librados: starting msgr at 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 1 librados: starting objecter 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.794+0000 7f5c229cf640 1 Processor -- start 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 1 -- start start 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 1 librados: setting wanted keys 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 1 librados: calling monclient init 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient: init 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient: _add_conns ranks=[3,1,4,6,5,2,7,8,0] 2024-08-19T10:03:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient(hunting): picked mon.noname-g con 0x7f5c1c0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient(hunting): picked mon.noname-f con 0x7f5c1c0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5c1c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c13bb70 con 0x7f5c1c0a6ba0 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c13d170 con 0x7f5c1c0a67c0 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1c0aa0f0 con 0x7f5c1c13f5c0 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c229cf640 10 monclient(hunting): authenticate will time out at 1478.871648s 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c20f6c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f5c1c13f5c0 0x7f5c1c1d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52542/0 (socket says 172.21.15.1:52542) 2024-08-19T10:03:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.795+0000 7f5c20f6c640 1 -- 172.21.15.1:0/1286530888 learned_addr learned my addr 172.21.15.1:0/1286530888 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c20f6c640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.1:6791/0 conn(0x7f5c1c13f5c0 legacy=0x7f5c1c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c18ff9640 10 client.?.objecter ms_handle_connect 0x7f5c1c13f5c0 2024-08-19T10:03:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c18ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c18ff9640 10 client.?.objecter ms_handle_connect 0x7f5c1c0a6ba0 2024-08-19T10:03:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c18ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c1b7fe640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6789/0 conn(0x7f5c1c0a6ba0 legacy=0x7f5c1c13ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c18ff9640 10 client.?.objecter ms_handle_connect 0x7f5c1c0a67c0 2024-08-19T10:03:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c18ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:42.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:42.797+0000 7f5c1bfff640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6790/0 conn(0x7f5c1c0a67c0 legacy=0x7f5c1c13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619546705 0 0) 0x7f5c1c13bb70 con 0x7f5c1c0a6ba0 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): my global_id is 174610 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c08004eb0 con 0x7f5c1c0a6ba0 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113911139 0 0) 0x7f5c1c0aa0f0 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 10 monclient(hunting): my global_id is 194179 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.122+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c1c13bb70 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.123+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723803679 0 0) 0x7f5c1c13d170 con 0x7f5c1c0a67c0 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.123+0000 7f5c18ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.123+0000 7f5c18ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.123+0000 7f5c18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.123+0000 7f5c18ff9640 10 monclient(hunting): my global_id is 184359 2024-08-19T10:03:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.123+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c1c0aa0f0 con 0x7f5c1c0a67c0 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.126+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1954101731 0 0) 0x7f5c1c13bb70 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.126+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c1c13d170 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.126+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c14002050 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient(hunting): dump: 2024-08-19T10:03:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.130 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2422948381 0 0) 0x7f5c1c13d170 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6790/0 conn(0x7f5c1c0a67c0 legacy=0x7f5c1c13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6789/0 conn(0x7f5c1c0a6ba0 legacy=0x7f5c1c13ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 1 monclient: found mon.c 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1c0a9cc0 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient: _finish_auth 0 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c229cf640 5 monclient: authenticate success, global_id 194179 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c229cf640 10 monclient: _renew_subs 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c229cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:43.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.127+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1c13b890 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c14003e00 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c14003990 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c1a7fc640 4 set_mon_vals no callback set 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 10 monclient: dump: 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.134 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.135 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c229cf640 10 client.194179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5c140043f0 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c229cf640 10 monclient: _renew_subs 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.128+0000 7f5c229cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c1c13a920 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c229cf640 1 librados: init done 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(485..485 src has 299..485) ==== 5079+0+0 (unknown 223647928 0 0) 0x7f5c1404e400 con 0x7f5c1c13f5c0 2024-08-19T10:03:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c18ff9640 10 client.194179.objecter ms_dispatch 0x7f5c1c000b90 osd_map(485..485 src has 299..485) 2024-08-19T10:03:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c18ff9640 3 client.194179.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T10:03:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c18ff9640 3 client.194179.objecter handle_osd_map decoding full epoch 485 2024-08-19T10:03:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.129+0000 7f5c18ff9640 20 client.194179.objecter dump_active .. 0 homeless 2024-08-19T10:03:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.130+0000 7f5c1bfff640 10 monclient: get_auth_request con 0x7f5c08041240 auth_method 0 2024-08-19T10:03:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.130+0000 7f5c229cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.130+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c1c0a55a0 con 0x7f5c1c09da50 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.132+0000 7f5c18ff9640 10 client.194179.objecter ms_handle_connect 0x7f5c08041240 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 client.194179.objecter ms_handle_connect 0x7f5c1c09da50 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 client.194179.objecter resend_mon_ops 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c1b7fe640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6791/0 conn(0x7f5c1c09da50 legacy=0x7f5c1c09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732189300 0 0) 0x7f5c1c0a55a0 con 0x7f5c1c09da50 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient(hunting): my global_id is 27617 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c1c13d170 con 0x7f5c1c09da50 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2358971609 0 0) 0x7f5c1c13d170 con 0x7f5c1c09da50 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c1c0a55a0 con 0x7f5c1c09da50 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c10001c60 con 0x7f5c1c09da50 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.133+0000 7f5c18ff9640 10 monclient: dump: 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4266581987 0 0) 0x7f5c1c0a55a0 con 0x7f5c1c09da50 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c1c0008d0 con 0x7f5c1c09da50 2024-08-19T10:03:43.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5c100057c0 con 0x7f5c1c09da50 2024-08-19T10:03:43.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:43.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6791/0 conn(0x7f5c1c09da50 legacy=0x7f5c1c09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.134+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6791/0 conn(0x7f5c1c09da50 legacy=0x7f5c1c09de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.146+0000 7f5c229cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.146+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c1c09e920 con 0x7f5c1c09e540 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c1b7fe640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6791/0 conn(0x7f5c1c09e540 legacy=0x7f5c1c1dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 client.194179.objecter ms_handle_connect 0x7f5c1c09e540 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 client.194179.objecter resend_mon_ops 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666726295 0 0) 0x7f5c1c09e920 con 0x7f5c1c09e540 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient(hunting): my global_id is 27626 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c1c0a55a0 con 0x7f5c1c09e540 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4183823871 0 0) 0x7f5c1c0a55a0 con 0x7f5c1c09e540 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c1c09e920 con 0x7f5c1c09e540 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5c10003ce0 con 0x7f5c1c09e540 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 10 monclient: dump: 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2182448736 0 0) 0x7f5c1c09e920 con 0x7f5c1c09e540 2024-08-19T10:03:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.149+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c1c13a330 con 0x7f5c1c09e540 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 1809424888) 0x7f5c10005a30 con 0x7f5c1c09e540 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c18ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6791/0 conn(0x7f5c1c09e540 legacy=0x7f5c1c1dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c18ff9640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6791/0 conn(0x7f5c1c09e540 legacy=0x7f5c1c1dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 829415, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:43.175 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 10 librados: watch_flush enter 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 10 librados: watch_flush exit 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 20 client.194179.objecter shutdown clearing up homeless session... 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 10 client.194179.objecter successfully canceled tick 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5c08041240 legacy=0x7f5c08043700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 10 monclient: shutdown 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:43.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 >> v1:172.21.15.1:6791/0 conn(0x7f5c1c13f5c0 legacy=0x7f5c1c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c20f6c640 1 -- 172.21.15.1:0/1286530888 reap_dead start 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c20f6c640 1 -- 172.21.15.1:0/1286530888 reap_dead start 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 shutdown_connections 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 >> 172.21.15.1:0/1286530888 conn(0x7f5c1c0ac4b0 msgr2=0x7f5c1c13ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 shutdown_connections 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.150+0000 7f5c229cf640 1 -- 172.21.15.1:0/1286530888 wait complete. 2024-08-19T10:03:43.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.151+0000 7f5c229cf640 1 librados: shutdown 2024-08-19T10:03:43.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:43.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:03:43.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.277+0000 7f55d82af640 10 monclient: get_monmap_and_config 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.277+0000 7f55d82af640 10 monclient: build_initial_monmap 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.277+0000 7f55d82af640 10 monclient: monmap: 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:43.278807+0000 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:43.278807+0000 2024-08-19T10:03:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.278+0000 7f55d82af640 1 Processor -- start 2024-08-19T10:03:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.278+0000 7f55d82af640 1 -- start start 2024-08-19T10:03:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.278+0000 7f55d82af640 10 monclient: init 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient: _add_conns ranks=[8,1,5,2,6,7,0,4,3] 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient(hunting): picked mon.noname-e con 0x7f55d009eb60 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient(hunting): picked mon.noname-f con 0x7f55d009fc00 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient(hunting): picked mon.noname-h con 0x7f55d009f650 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d00a4070 con 0x7f55d009fc00 2024-08-19T10:03:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d00a3df0 con 0x7f55d009f650 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d00a0760 con 0x7f55d009eb60 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d82af640 10 monclient(hunting): authenticate will time out at 1479.355155s 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d604b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f55d009eb60 0x7f55d009ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41780/0 (socket says 172.21.15.1:41780) 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.279+0000 7f55d604b640 1 -- 172.21.15.1:0/183510833 learned_addr learned my addr 172.21.15.1:0/183510833 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d604b640 1 -- 172.21.15.1:0/183510833 >> v1:172.21.15.1:6793/0 conn(0x7f55d009eb60 legacy=0x7f55d009ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d584a640 1 -- 172.21.15.1:0/183510833 >> v1:172.21.15.82:6789/0 conn(0x7f55d009fc00 legacy=0x7f55d013b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450508943 0 0) 0x7f55d00a0760 con 0x7f55d009eb60 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): my global_id is 184400 2024-08-19T10:03:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b8003520 con 0x7f55d009eb60 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d684c640 1 -- 172.21.15.1:0/183510833 >> v1:172.21.15.82:6791/0 conn(0x7f55d009f650 legacy=0x7f55d013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941915671 0 0) 0x7f55d00a4070 con 0x7f55d009fc00 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): my global_id is 174619 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d00a0760 con 0x7f55d009fc00 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264168640 0 0) 0x7f55d00a3df0 con 0x7f55d009f650 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient(hunting): my global_id is 27635 2024-08-19T10:03:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d00a4070 con 0x7f55d009f650 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3736025720 0 0) 0x7f55b8003520 con 0x7f55d009eb60 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 >> v1:172.21.15.82:6791/0 conn(0x7f55d009f650 legacy=0x7f55d013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 >> v1:172.21.15.82:6789/0 conn(0x7f55d009fc00 legacy=0x7f55d013b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: _finish_hunting 0 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 monclient: found mon.noname-e 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d00a0a40 con 0x7f55d009eb60 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: _finish_auth 0 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: _check_auth_tickets 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c8001ae0 con 0x7f55d009eb60 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d82af640 5 monclient: authenticate success, global_id 184400 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d82af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 10 monclient: dump: 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d82af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.281+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55c80048a0 con 0x7f55d009eb60 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d5049640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d5049640 1 -- 172.21.15.1:0/183510833 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c8004e10 con 0x7f55d009eb60 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 4 set_mon_vals no callback set 2024-08-19T10:03:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d5049640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d5049640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d5049640 10 monclient: dump: 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 10 monclient: shutdown 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 1 -- 172.21.15.1:0/183510833 >> v1:172.21.15.1:6793/0 conn(0x7f55d009eb60 legacy=0x7f55d009ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 1 -- 172.21.15.1:0/183510833 shutdown_connections 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 1 -- 172.21.15.1:0/183510833 >> 172.21.15.1:0/183510833 conn(0x7f55d0068650 msgr2=0x7f55d0068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 1 -- 172.21.15.1:0/183510833 shutdown_connections 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.282+0000 7f55d82af640 1 -- 172.21.15.1:0/183510833 wait complete. 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient: build_initial_monmap 2024-08-19T10:03:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient: monmap: 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 1 librados: starting msgr at 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 1 librados: starting objecter 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 1 Processor -- start 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 1 -- start start 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 1 librados: setting wanted keys 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 1 librados: calling monclient init 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient: init 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient: _add_conns ranks=[7,3,8,5,1,0,4,2,6] 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient(hunting): picked mon.noname-i con 0x7f55d009eb60 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.283+0000 7f55d82af640 10 monclient(hunting): picked mon.noname-g con 0x7f55d009f650 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d82af640 10 monclient(hunting): picked mon.noname-e con 0x7f55d009fc00 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d82af640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d01ca900 con 0x7f55d009f650 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d82af640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d0073060 con 0x7f55d009eb60 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d82af640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d0123970 con 0x7f55d009fc00 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d82af640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d82af640 10 monclient(hunting): authenticate will time out at 1479.360058s 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d684c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f55d009fc00 0x7f55d01d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41786/0 (socket says 172.21.15.1:41786) 2024-08-19T10:03:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.284+0000 7f55d684c640 1 -- 172.21.15.1:0/1109753905 learned_addr learned my addr 172.21.15.1:0/1109753905 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d684c640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.1:6793/0 conn(0x7f55d009fc00 legacy=0x7f55d01d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 client.?.objecter ms_handle_connect 0x7f55d009fc00 2024-08-19T10:03:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 client.?.objecter ms_handle_connect 0x7f55d009eb60 2024-08-19T10:03:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d604b640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55d009eb60 legacy=0x7f55d0136570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 client.?.objecter ms_handle_connect 0x7f55d009f650 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d584a640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6790/0 conn(0x7f55d009f650 legacy=0x7f55d01ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3643949581 0 0) 0x7f55d0123970 con 0x7f55d009fc00 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): my global_id is 184409 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55bc004eb0 con 0x7f55d009fc00 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213549491 0 0) 0x7f55d01ca900 con 0x7f55d009f650 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): my global_id is 184368 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d0123970 con 0x7f55d009f650 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818687584 0 0) 0x7f55d00a0a40 con 0x7f55d009eb60 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): my global_id is 145123 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d01ca900 con 0x7f55d009eb60 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1836297935 0 0) 0x7f55bc004eb0 con 0x7f55d009fc00 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55d00a0a40 con 0x7f55d009fc00 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c0003990 con 0x7f55d009fc00 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): dump: 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1185754896 0 0) 0x7f55d0123970 con 0x7f55d009f650 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55bc004eb0 con 0x7f55d009f650 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c4001cf0 con 0x7f55d009f650 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:43.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient(hunting): dump: 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2710647841 0 0) 0x7f55d00a0a40 con 0x7f55d009fc00 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55d009eb60 legacy=0x7f55d0136570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6790/0 conn(0x7f55d009f650 legacy=0x7f55d01ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 monclient: found mon.e 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d0123c50 con 0x7f55d009fc00 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient: _finish_auth 0 2024-08-19T10:03:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 5 monclient: authenticate success, global_id 184409 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 10 monclient: _renew_subs 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55d0121bd0 con 0x7f55d009fc00 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 10 client.184409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 10 monclient: _renew_subs 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f55d01c9c80 con 0x7f55d009fc00 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.286+0000 7f55d82af640 1 librados: init done 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55c0003ed0 con 0x7f55d009fc00 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:43.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c0004440 con 0x7f55d009fc00 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55d4848640 4 set_mon_vals no callback set 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 10 monclient: dump: 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.309 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55d82af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f55b00065c0 con 0x7f55b0002860 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.287+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f55c00045e0 con 0x7f55d009fc00 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.288+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(485..485 src has 299..485) ==== 5079+0+0 (unknown 223647928 0 0) 0x7f55c004e3d0 con 0x7f55d009fc00 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.288+0000 7f55ceffd640 10 client.184409.objecter ms_dispatch 0x7f55d0000b90 osd_map(485..485 src has 299..485) 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.288+0000 7f55ceffd640 3 client.184409.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T10:03:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.288+0000 7f55ceffd640 3 client.184409.objecter handle_osd_map decoding full epoch 485 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.288+0000 7f55ceffd640 20 client.184409.objecter dump_active .. 0 homeless 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.288+0000 7f55d584a640 10 monclient: get_auth_request con 0x7f55bc0414d0 auth_method 0 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 client.184409.objecter ms_handle_connect 0x7f55b0002860 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55d604b640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55b0002860 legacy=0x7f55b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 client.184409.objecter resend_mon_ops 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3825471669 0 0) 0x7f55b00065c0 con 0x7f55b0002860 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient(hunting): my global_id is 145132 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d00a0a40 con 0x7f55b0002860 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3597023109 0 0) 0x7f55d00a0a40 con 0x7f55b0002860 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55b00065c0 con 0x7f55b0002860 2024-08-19T10:03:43.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c8001d30 con 0x7f55b0002860 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient: dump: 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.312 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 client.184409.objecter ms_handle_connect 0x7f55bc0414d0 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4210859126 0 0) 0x7f55b00065c0 con 0x7f55b0002860 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55b000a160 con 0x7f55b0002860 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f55c8001320 con 0x7f55b0002860 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55b0002860 legacy=0x7f55b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.290+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55b0002860 legacy=0x7f55b0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:43.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.296+0000 7f55d82af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.297+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f55b0009e70 con 0x7f55b0009130 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55d604b640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55b0009130 legacy=0x7f55b00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 client.184409.objecter ms_handle_connect 0x7f55b0009130 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 client.184409.objecter resend_mon_ops 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169053506 0 0) 0x7f55b0009e70 con 0x7f55b0009130 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient(hunting): my global_id is 145141 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b00065c0 con 0x7f55b0009130 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3034669809 0 0) 0x7f55b00065c0 con 0x7f55b0009130 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55b0009e70 con 0x7f55b0009130 2024-08-19T10:03:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f55c8005b00 con 0x7f55b0009130 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 10 monclient: dump: 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3783691536 0 0) 0x7f55b0009e70 con 0x7f55b0009130 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.299+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f55b000c410 con 0x7f55b0009130 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3453925967) 0x7f55c8006ff0 con 0x7f55b0009130 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55ceffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55b0009130 legacy=0x7f55b00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55ceffd640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6792/0 conn(0x7f55b0009130 legacy=0x7f55b00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 547680, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:43.317 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:43.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.320 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:43.321 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:43.322 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.323 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:43.326 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.327 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:43.330 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.333 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:43.333 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:43.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:43.335 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.336 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:43.337 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:43.338 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 10 librados: watch_flush enter 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 10 librados: watch_flush exit 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 20 client.184409.objecter shutdown clearing up homeless session... 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 10 client.184409.objecter successfully canceled tick 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.82:6812/2549104415 conn(0x7f55bc0414d0 legacy=0x7f55bc043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 10 monclient: shutdown 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 >> v1:172.21.15.1:6793/0 conn(0x7f55d009fc00 legacy=0x7f55d01d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d684c640 1 -- 172.21.15.1:0/1109753905 reap_dead start 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d684c640 1 -- 172.21.15.1:0/1109753905 reap_dead start 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 shutdown_connections 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 >> 172.21.15.1:0/1109753905 conn(0x7f55d0068650 msgr2=0x7f55d0073610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 shutdown_connections 2024-08-19T10:03:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 -- 172.21.15.1:0/1109753905 wait complete. 2024-08-19T10:03:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.300+0000 7f55d82af640 1 librados: shutdown 2024-08-19T10:03:43.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:43.341 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T10:03:43.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.439+0000 7f49b886e640 10 monclient: get_monmap_and_config 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.439+0000 7f49b886e640 10 monclient: build_initial_monmap 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.439+0000 7f49b886e640 10 monclient: monmap: 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:43.440000+0000 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:43.440000+0000 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:43.441 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.439+0000 7f49b886e640 1 Processor -- start 2024-08-19T10:03:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.439+0000 7f49b886e640 1 -- start start 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.439+0000 7f49b886e640 10 monclient: init 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient: _add_conns ranks=[6,5,3,2,0,7,1,4,8] 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient(hunting): picked mon.noname-d con 0x7f49b40a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient(hunting): picked mon.noname-h con 0x7f49b40694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient(hunting): picked mon.noname-g con 0x7f49b40a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b40a3c70 con 0x7f49b40a46a0 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b40a33d0 con 0x7f49b40694b0 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b40a36b0 con 0x7f49b40a6830 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b886e640 10 monclient(hunting): authenticate will time out at 1479.516424s 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b259c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f49b40a6830 0x7f49b40a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43790/0 (socket says 172.21.15.1:43790) 2024-08-19T10:03:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.440+0000 7f49b259c640 1 -- 172.21.15.1:0/4219858292 learned_addr learned my addr 172.21.15.1:0/4219858292 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.442+0000 7f49b259c640 1 -- 172.21.15.1:0/4219858292 >> v1:172.21.15.1:6792/0 conn(0x7f49b40a6830 legacy=0x7f49b40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.442+0000 7f49b2d9d640 1 -- 172.21.15.1:0/4219858292 >> v1:172.21.15.82:6790/0 conn(0x7f49b40a46a0 legacy=0x7f49b4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.442+0000 7f49b1d9b640 1 -- 172.21.15.1:0/4219858292 >> v1:172.21.15.82:6791/0 conn(0x7f49b40694b0 legacy=0x7f49b40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4082289344 0 0) 0x7f49b40a36b0 con 0x7f49b40a6830 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): my global_id is 194145 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49a0003520 con 0x7f49b40a6830 2024-08-19T10:03:43.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3766498504 0 0) 0x7f49b40a3c70 con 0x7f49b40a46a0 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): my global_id is 184377 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b40a36b0 con 0x7f49b40a46a0 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1854479935 0 0) 0x7f49b40a33d0 con 0x7f49b40694b0 2024-08-19T10:03:43.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient(hunting): my global_id is 27644 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b40a3c70 con 0x7f49b40694b0 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1975113975 0 0) 0x7f49a0003520 con 0x7f49b40a6830 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 >> v1:172.21.15.82:6791/0 conn(0x7f49b40694b0 legacy=0x7f49b40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 >> v1:172.21.15.82:6790/0 conn(0x7f49b40a46a0 legacy=0x7f49b4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient: _finish_hunting 0 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 monclient: found mon.noname-d 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b406dbb0 con 0x7f49b40a6830 2024-08-19T10:03:43.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient: _finish_auth 0 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 10 monclient: _check_auth_tickets 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f49a4001f20 con 0x7f49b40a6830 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b886e640 5 monclient: authenticate success, global_id 194145 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.715+0000 7f49b886e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: dump: 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.722 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49a4004c20 con 0x7f49b40a6830 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 1 -- 172.21.15.1:0/4219858292 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f49a40020c0 con 0x7f49b40a6830 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b159a640 10 monclient: dump: 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 4 set_mon_vals no callback set 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 10 monclient: shutdown 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 1 -- 172.21.15.1:0/4219858292 >> v1:172.21.15.1:6792/0 conn(0x7f49b40a6830 legacy=0x7f49b40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.716+0000 7f49b886e640 1 -- 172.21.15.1:0/4219858292 shutdown_connections 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.717+0000 7f49b886e640 1 -- 172.21.15.1:0/4219858292 >> 172.21.15.1:0/4219858292 conn(0x7f49b40a9020 msgr2=0x7f49b4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.717+0000 7f49b886e640 1 -- 172.21.15.1:0/4219858292 shutdown_connections 2024-08-19T10:03:43.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.717+0000 7f49b886e640 1 -- 172.21.15.1:0/4219858292 wait complete. 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.717+0000 7f49b886e640 10 monclient: build_initial_monmap 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.717+0000 7f49b886e640 10 monclient: monmap: 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 librados: starting msgr at 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 librados: starting objecter 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 Processor -- start 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 -- start start 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 librados: setting wanted keys 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 librados: calling monclient init 2024-08-19T10:03:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient: init 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient: _add_conns ranks=[2,7,5,0,4,1,3,8,6] 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient(hunting): picked mon.noname-b con 0x7f49b40694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient(hunting): picked mon.noname-i con 0x7f49b40a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient(hunting): picked mon.noname-h con 0x7f49b40a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b4138c60 con 0x7f49b40694b0 2024-08-19T10:03:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b4142d30 con 0x7f49b40a6830 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b4136900 con 0x7f49b40a46a0 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b886e640 10 monclient(hunting): authenticate will time out at 1479.794745s 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b259c640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f49b40694b0 0x7f49b4134fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45632/0 (socket says 172.21.15.1:45632) 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.718+0000 7f49b259c640 1 -- 172.21.15.1:0/21394891 learned_addr learned my addr 172.21.15.1:0/21394891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.720+0000 7f49b259c640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.1:6790/0 conn(0x7f49b40694b0 legacy=0x7f49b4134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.720+0000 7f49aaffd640 10 client.?.objecter ms_handle_connect 0x7f49b40694b0 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.720+0000 7f49aaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 client.?.objecter ms_handle_connect 0x7f49b40a46a0 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b1d9b640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.82:6792/0 conn(0x7f49b40a46a0 legacy=0x7f49b41356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b259c640 1 --1- 172.21.15.1:0/21394891 >> v1:172.21.15.1:6790/0 conn(0x7f49b40694b0 0x7f49b4134fc0 :-1 s=OPENED pgs=3421 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f49b4138c60 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:03:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 client.?.objecter ms_handle_connect 0x7f49b40a6830 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b2d9d640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.82:6791/0 conn(0x7f49b40a6830 legacy=0x7f49b41d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247484393 0 0) 0x7f49b4136900 con 0x7f49b40a46a0 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): my global_id is 145150 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4994004eb0 con 0x7f49b40a46a0 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410726725 0 0) 0x7f49b4142d30 con 0x7f49b40a6830 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): my global_id is 27653 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b4136900 con 0x7f49b40a6830 2024-08-19T10:03:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3590609399 0 0) 0x7f4994004eb0 con 0x7f49b40a46a0 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49b4142d30 con 0x7f49b40a46a0 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f499c001c20 con 0x7f49b40a46a0 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): dump: 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1618615608 0 0) 0x7f49b4136900 con 0x7f49b40a6830 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4994004eb0 con 0x7f49b40a6830 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f49ac0036b0 con 0x7f49b40a6830 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient(hunting): dump: 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.740 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3243790596 0 0) 0x7f49b4142d30 con 0x7f49b40a46a0 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.82:6791/0 conn(0x7f49b40a6830 legacy=0x7f49b41d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.1:6790/0 conn(0x7f49b40694b0 legacy=0x7f49b4134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 monclient: found mon.i 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b4135de0 con 0x7f49b40a46a0 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient: _finish_auth 0 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1643015317 0 0) 0x7f49b4138c60 con 0x7f49b40694b0 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49aaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b886e640 5 monclient: authenticate success, global_id 145150 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b886e640 10 monclient: _renew_subs 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b886e640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.721+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49b41383c0 con 0x7f49b40a46a0 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49b886e640 10 client.145150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49b886e640 10 monclient: _renew_subs 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49b886e640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f49b4137460 con 0x7f49b40a46a0 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49b886e640 1 librados: init done 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f499c0039a0 con 0x7f49b40a46a0 2024-08-19T10:03:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49aaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f499c003e90 con 0x7f49b40a46a0 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49b0d99640 4 set_mon_vals no callback set 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49aaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.722+0000 7f49aaffd640 10 monclient: dump: 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49b886e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49b886e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49b886e640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4998005080 con 0x7f49b40a46a0 2024-08-19T10:03:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f499c0040f0 con 0x7f49b40a46a0 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(486..486 src has 299..486) ==== 5502+0+0 (unknown 1760802290 0 0) 0x7f499c04e140 con 0x7f49b40a46a0 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49aaffd640 10 client.145150.objecter ms_dispatch 0x7f49b4000b90 osd_map(486..486 src has 299..486) 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49aaffd640 3 client.145150.objecter handle_osd_map got epochs [486,486] > 0 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49aaffd640 3 client.145150.objecter handle_osd_map decoding full epoch 486 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49aaffd640 20 client.145150.objecter dump_active .. 0 homeless 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.723+0000 7f49b259c640 10 monclient: get_auth_request con 0x7f4994041580 auth_method 0 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.725+0000 7f49aaffd640 10 client.145150.objecter ms_handle_connect 0x7f4994041580 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.727+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f499c0174d0 con 0x7f49b40a46a0 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.727+0000 7f49aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.727+0000 7f49aaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.982+0000 7f49b886e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T10:03:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.982+0000 7f49b886e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T10:03:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.982+0000 7f49b886e640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:03:43.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.982+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f4998005c40 con 0x7f49b40a46a0 2024-08-19T10:03:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.984+0000 7f49aaffd640 1 -- 172.21.15.1:0/21394891 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f499c016dd0 con 0x7f49b40a46a0 2024-08-19T10:03:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.984+0000 7f49aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T10:03:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.984+0000 7f49aaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 10 librados: watch_flush enter 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 10 librados: watch_flush exit 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 20 client.145150.objecter shutdown clearing up homeless session... 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 10 client.145150.objecter successfully canceled tick 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4994041580 legacy=0x7f4994043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 10 monclient: shutdown 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.986+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 >> v1:172.21.15.82:6792/0 conn(0x7f49b40a46a0 legacy=0x7f49b41356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.987+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 shutdown_connections 2024-08-19T10:03:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.987+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 >> 172.21.15.1:0/21394891 conn(0x7f49b40a9020 msgr2=0x7f49b40a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.987+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 shutdown_connections 2024-08-19T10:03:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.987+0000 7f49b886e640 1 -- 172.21.15.1:0/21394891 wait complete. 2024-08-19T10:03:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:43.987+0000 7f49b886e640 1 librados: shutdown 2024-08-19T10:03:43.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=861161512,config=2586702516,health=3726805051,kv=1455257465,logm=3756268999}) 2024-08-19T10:03:43.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=3603950281,config=2586702516,health=3726805051,kv=1505562194,logm=1117180766}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=1159839690,config=2586702516,health=1097993572,kv=375198362,logm=1626911232}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4194655047,config=2586702516,health=418924436,kv=1850857368,logm=1618789304}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=2292528720,config=2586702516,health=418924436,kv=3798231245,logm=3444739735}) 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=1042126162,config=2586702516,health=4213877817,kv=2401233588,logm=966871934}) 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=3058826016,config=2586702516,health=3726805051,kv=3098392845,logm=1321610272}) 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1313840506,config=2586702516,health=77634962,kv=1063813278,logm=3390988051}) 2024-08-19T10:03:43.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.991+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=3421234652,config=2586702516,health=2825370583,kv=2401233588,logm=571426621}) 2024-08-19T10:03:43.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187488978}) 2024-08-19T10:03:43.996 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2933191192}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.993+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167875390}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3034328654}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=225494470}) 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.997 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1206654067}) 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=509827218}) 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289312149}) 2024-08-19T10:03:43.998 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.994+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1093405123}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688086472}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3024125901}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1203412227}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1960726237}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=117914295}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=283315567}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2979362318}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890666686}) 2024-08-19T10:03:44.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.996+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4050116509}) 2024-08-19T10:03:44.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T10:03:44.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2910730057}) 2024-08-19T10:03:44.017 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1530273011}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1183419555}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=445998171}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.018 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=722452167}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1323474498}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4048916513}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268806048}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:43.999+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4620088}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1439850302}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748635846}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2578216513}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=715882042}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1220753947}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=583950582}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2038658867}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777821916}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.001+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3168340900}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2920627196}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968953892}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4215841711}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2668888312}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=751482352}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=715990226}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3388130341}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545034720}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.003+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176471298}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1220897566}) 2024-08-19T10:03:44.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3720714546}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929617041}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459845435}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3198431006}) 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1990486613}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3448394269}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130633747}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.004+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1497529219}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3811663884}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212037053}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535598144}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=153814187}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3198933145}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2913900911}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517121388}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.005+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.006+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209097794}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.006+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2266123592}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727311482}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554491193}) 2024-08-19T10:03:44.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3360877485}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2266482099}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=330782720}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1927608941}) 2024-08-19T10:03:44.028 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2589348327}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747536793}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.007+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2712891839}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1634092176}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9646270}) 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.029 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=143411112}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2271235499}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2687003454}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2100338322}) 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.030 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1727462832}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341798547}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.008+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1474553673}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783129001}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386572878}) 2024-08-19T10:03:44.031 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1719762722}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=363506905}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3703529110}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=702649001}) 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.032 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2522831706}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252425047}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.009+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2298126136}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412958353}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713714887}) 2024-08-19T10:03:44.033 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=718414424}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3311979869}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=752536172}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1782821491}) 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2285828987}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515259495}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.011+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1397335352}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301735889}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052639719}) 2024-08-19T10:03:44.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1457501483}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=842391784}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2971472771}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2202579988}) 2024-08-19T10:03:44.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1448660196}) 2024-08-19T10:03:44.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908450236}) 2024-08-19T10:03:44.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.012+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2210334564}) 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416328474}) 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352940082}) 2024-08-19T10:03:44.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=32722176}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3015811666}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=457138477}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1255737449}) 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=248294054}) 2024-08-19T10:03:44.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218390207}) 2024-08-19T10:03:44.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.120+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3751398305}) 2024-08-19T10:03:44.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367669146}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2858566131}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1485230785}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=707675460}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1980972327}) 2024-08-19T10:03:44.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1517710350}) 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2829996849}) 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759031231}) 2024-08-19T10:03:44.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.124+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3981073407}) 2024-08-19T10:03:44.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500922254}) 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3818964809}) 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.328+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=36211225}) 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=121684929}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=674531776}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3885121056}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2669964274}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263500442}) 2024-08-19T10:03:44.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.329+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3957648364}) 2024-08-19T10:03:44.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129580553}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134421313}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1230190026}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2431370965}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3867960837}) 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2963690838}) 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3284812272}) 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682228844}) 2024-08-19T10:03:44.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.334+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848237988}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676286279}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522277335}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4055851456}) 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4221390172}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2354666379}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4224642427}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=93998867}) 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832854314}) 2024-08-19T10:03:44.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.342+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3559223913}) 2024-08-19T10:03:44.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532450854}) 2024-08-19T10:03:44.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=482661962}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=353152688}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=446080274}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1178528492}) 2024-08-19T10:03:44.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=539782570}) 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.716+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.717+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1374197106}) 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.717+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.717+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211561632}) 2024-08-19T10:03:44.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.717+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2600319959}) 2024-08-19T10:03:44.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3563816968}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946674529}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.733+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3822227443}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3688823115}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2186101957}) 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3782892401}) 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137948505}) 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475304761}) 2024-08-19T10:03:44.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:44.734+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3141305495}) 2024-08-19T10:03:45.003 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:03:45.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:03:45.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.101 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4264318986}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509345161}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1646464120}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3847100094}) 2024-08-19T10:03:45.102 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=11147361}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=144827180}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1303146088}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159987643}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.094+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=826556254}) 2024-08-19T10:03:45.103 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285481825}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944807390}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3750985156}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=216431958}) 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.104 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3258687081}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3597650281}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1279548689}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767333098}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.096+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3281179550}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2051449382}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928765329}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4234345592}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=796447983}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2734834056}) 2024-08-19T10:03:45.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3254241694}) 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2545840972}) 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558229662}) 2024-08-19T10:03:45.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.097+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3782150088}) 2024-08-19T10:03:45.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.113+0000 7fbfb1ecc640 10 monclient: get_monmap_and_config 2024-08-19T10:03:45.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.113+0000 7fbfb1ecc640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.113+0000 7fbfb1ecc640 10 monclient: monmap: 2024-08-19T10:03:45.114 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.114 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.114 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:45.114467+0000 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:45.114467+0000 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.115 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 1 Processor -- start 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 1 -- start start 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient: init 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient: _add_conns ranks=[3,8,5,1,2,6,4,7,0] 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient(hunting): picked mon.noname-g con 0x7fbfac0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient(hunting): picked mon.noname-e con 0x7fbfac0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient(hunting): picked mon.noname-h con 0x7fbfac13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac0a3b40 con 0x7fbfac0a67c0 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac0a8130 con 0x7fbfac13f5c0 2024-08-19T10:03:45.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac06ede0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfb1ecc640 10 monclient(hunting): authenticate will time out at 1481.190850s 2024-08-19T10:03:45.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfaaffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbfac0a6ba0 0x7fbfac0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41802/0 (socket says 172.21.15.1:41802) 2024-08-19T10:03:45.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.114+0000 7fbfaaffd640 1 -- 172.21.15.1:0/1208806863 learned_addr learned my addr 172.21.15.1:0/1208806863 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaaffd640 1 -- 172.21.15.1:0/1208806863 >> v1:172.21.15.1:6793/0 conn(0x7fbfac0a6ba0 legacy=0x7fbfac0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4193274507 0 0) 0x7fbfac06ede0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient(hunting): my global_id is 184436 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf94003520 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfab7fe640 1 -- 172.21.15.1:0/1208806863 >> v1:172.21.15.82:6790/0 conn(0x7fbfac0a67c0 legacy=0x7fbfac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfabfff640 1 -- 172.21.15.1:0/1208806863 >> v1:172.21.15.82:6791/0 conn(0x7fbfac13f5c0 legacy=0x7fbfac1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1141571659 0 0) 0x7fbf94003520 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 >> v1:172.21.15.82:6791/0 conn(0x7fbfac13f5c0 legacy=0x7fbfac1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 >> v1:172.21.15.82:6790/0 conn(0x7fbfac0a67c0 legacy=0x7fbfac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 monclient: found mon.noname-e 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac0a92a0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbfa0001ad0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfb1ecc640 5 monclient: authenticate success, global_id 184436 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfb1ecc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: dump: 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfb1ecc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfa00048a0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 1 -- 172.21.15.1:0/1208806863 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbfa0004e10 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfb1ecc640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfb1ecc640 4 set_mon_vals no callback set 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.117+0000 7fbfaa7fc640 10 monclient: dump: 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.126 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 10 monclient: shutdown 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1208806863 >> v1:172.21.15.1:6793/0 conn(0x7fbfac0a6ba0 legacy=0x7fbfac0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1208806863 shutdown_connections 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1208806863 >> 172.21.15.1:0/1208806863 conn(0x7fbfac0ac4b0 msgr2=0x7fbfac0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1208806863 shutdown_connections 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1208806863 wait complete. 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.118+0000 7fbfb1ecc640 10 monclient: monmap: 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.128 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 librados: starting msgr at 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 librados: starting objecter 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 Processor -- start 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 -- start start 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 librados: setting wanted keys 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 librados: calling monclient init 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient: init 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient: _add_conns ranks=[4,2,5,0,1,7,8,3,6] 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient(hunting): picked mon.noname-c con 0x7fbfac0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient(hunting): picked mon.noname-b con 0x7fbfac0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient(hunting): picked mon.noname-h con 0x7fbfac13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac1cf020 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac139a70 con 0x7fbfac0a67c0 2024-08-19T10:03:45.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfac0aa1d0 con 0x7fbfac13f5c0 2024-08-19T10:03:45.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfb1ecc640 10 monclient(hunting): authenticate will time out at 1481.195860s 2024-08-19T10:03:45.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfab7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fbfac0a67c0 0x7fbfac1ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52554/0 (socket says 172.21.15.1:52554) 2024-08-19T10:03:45.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.119+0000 7fbfab7fe640 1 -- 172.21.15.1:0/1979693079 learned_addr learned my addr 172.21.15.1:0/1979693079 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfaaffd640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.1:6790/0 conn(0x7fbfac0a6ba0 legacy=0x7fbfac1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 client.?.objecter ms_handle_connect 0x7fbfac0a6ba0 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfab7fe640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.1:6791/0 conn(0x7fbfac0a67c0 legacy=0x7fbfac1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 client.?.objecter ms_handle_connect 0x7fbfac0a67c0 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfabfff640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.82:6791/0 conn(0x7fbfac13f5c0 legacy=0x7fbfac1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 client.?.objecter ms_handle_connect 0x7fbfac13f5c0 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719862974 0 0) 0x7fbfac1cf020 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): my global_id is 184340 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf98004eb0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732466279 0 0) 0x7fbfac139a70 con 0x7fbfac0a67c0 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): my global_id is 194188 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfac1cf020 con 0x7fbfac0a67c0 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328833554 0 0) 0x7fbfac0aa1d0 con 0x7fbfac13f5c0 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): my global_id is 27680 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfac139a70 con 0x7fbfac13f5c0 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 519480488 0 0) 0x7fbf98004eb0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfac0aa1d0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbfa0004d90 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): dump: 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2303913257 0 0) 0x7fbfac1cf020 con 0x7fbfac0a67c0 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf98004eb0 con 0x7fbfac0a67c0 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbf9c003610 con 0x7fbfac0a67c0 2024-08-19T10:03:45.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient(hunting): dump: 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2480864021 0 0) 0x7fbfac0aa1d0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.82:6791/0 conn(0x7fbfac13f5c0 legacy=0x7fbfac1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.1:6791/0 conn(0x7fbfac0a67c0 legacy=0x7fbfac1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 monclient: found mon.b 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac13b510 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 5 monclient: authenticate success, global_id 184340 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 10 monclient: _renew_subs 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbfac13e5b0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 10 client.184340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 10 monclient: _renew_subs 2024-08-19T10:03:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbfac13d1e0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfb1ecc640 1 librados: init done 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfa00048a0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbfa0001d20 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbfa9ffb640 4 set_mon_vals no callback set 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.122+0000 7fbf93fff640 10 monclient: dump: 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.141 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.142 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbfa0011a00 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbfb1ecc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(487..487 src has 299..487) ==== 5502+0+0 (unknown 648762363 0 0) 0x7fbfa004e220 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbf93fff640 10 client.184340.objecter ms_dispatch 0x7fbfac000b90 osd_map(487..487 src has 299..487) 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbf93fff640 3 client.184340.objecter handle_osd_map got epochs [487,487] > 0 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbf93fff640 3 client.184340.objecter handle_osd_map decoding full epoch 487 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbfb1ecc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbfb1ecc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.123+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c005080 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.124+0000 7fbf93fff640 20 client.184340.objecter dump_active .. 0 homeless 2024-08-19T10:03:45.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.124+0000 7fbfab7fe640 10 monclient: get_auth_request con 0x7fbf980414f0 auth_method 0 2024-08-19T10:03:45.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.126+0000 7fbf93fff640 10 client.184340.objecter ms_handle_connect 0x7fbf980414f0 2024-08-19T10:03:45.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.128+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fbfa00175f0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.128+0000 7fbf93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.128+0000 7fbf93fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381390640}) 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076318719}) 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3554651393}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=14089115}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1350960545}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=750211009}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1015816811}) 2024-08-19T10:03:45.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041991876}) 2024-08-19T10:03:45.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.201+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1108502001}) 2024-08-19T10:03:45.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382034160}) 2024-08-19T10:03:45.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3719673092}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4118997953}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=694403270}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4116593297}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3991960515}) 2024-08-19T10:03:45.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2581744483}) 2024-08-19T10:03:45.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297497568}) 2024-08-19T10:03:45.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.332+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2241272275}) 2024-08-19T10:03:45.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426295605}) 2024-08-19T10:03:45.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700529837}) 2024-08-19T10:03:45.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3674953198}) 2024-08-19T10:03:45.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=571446761}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3484391735}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3108379605}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2972649726}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487310807}) 2024-08-19T10:03:45.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.337+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3145290866}) 2024-08-19T10:03:45.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbfb1ecc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbfb1ecc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbfb1ecc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbf8c005370 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbf93fff640 1 -- 172.21.15.1:0/1979693079 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 84028604) 0x7fbfa0016ef0 con 0x7fbfac0a6ba0 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbf93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.393+0000 7fbf93fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:45.395 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":118,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 10 librados: watch_flush enter 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 10 librados: watch_flush exit 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 20 client.184340.objecter shutdown clearing up homeless session... 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 10 client.184340.objecter successfully canceled tick 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbf980414f0 legacy=0x7fbf980439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 10 monclient: shutdown 2024-08-19T10:03:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 >> v1:172.21.15.1:6790/0 conn(0x7fbfac0a6ba0 legacy=0x7fbfac1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 shutdown_connections 2024-08-19T10:03:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.395+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 >> 172.21.15.1:0/1979693079 conn(0x7fbfac0ac4b0 msgr2=0x7fbfac13ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.396+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 shutdown_connections 2024-08-19T10:03:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.396+0000 7fbfb1ecc640 1 -- 172.21.15.1:0/1979693079 wait complete. 2024-08-19T10:03:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.396+0000 7fbfb1ecc640 1 librados: shutdown 2024-08-19T10:03:45.407 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:03:45.407 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T10:03:45.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:03:45.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.506+0000 7ff394627640 10 monclient: get_monmap_and_config 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.506+0000 7ff394627640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.506+0000 7ff394627640 10 monclient: monmap: 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:45.507032+0000 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:45.507032+0000 2024-08-19T10:03:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.508 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.506+0000 7ff394627640 1 Processor -- start 2024-08-19T10:03:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.506+0000 7ff394627640 1 -- start start 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.506+0000 7ff394627640 10 monclient: init 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient: _add_conns ranks=[0,5,2,8,6,3,4,7,1] 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient(hunting): picked mon.noname-a con 0x7ff38c0a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient(hunting): picked mon.noname-h con 0x7ff38c0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient(hunting): picked mon.noname-b con 0x7ff38c0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c0a3c70 con 0x7ff38c0a6830 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c0a33d0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c0a36b0 con 0x7ff38c0694b0 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff394627640 10 monclient(hunting): authenticate will time out at 1481.583420s 2024-08-19T10:03:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff3923c3640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff38c0a6830 0x7ff38c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57234/0 (socket says 172.21.15.1:57234) 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.507+0000 7ff3923c3640 1 -- 172.21.15.1:0/3255919737 learned_addr learned my addr 172.21.15.1:0/3255919737 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff392bc4640 1 -- 172.21.15.1:0/3255919737 >> v1:172.21.15.1:6790/0 conn(0x7ff38c0a46a0 legacy=0x7ff38c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3923c3640 1 -- 172.21.15.1:0/3255919737 >> v1:172.21.15.1:6789/0 conn(0x7ff38c0a6830 legacy=0x7ff38c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161953614 0 0) 0x7ff38c0a33d0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): my global_id is 184349 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff374003520 con 0x7ff38c0a46a0 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784512998 0 0) 0x7ff38c0a3c70 con 0x7ff38c0a6830 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 10 monclient(hunting): my global_id is 174726 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff38c0a33d0 con 0x7ff38c0a6830 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.509+0000 7ff391bc2640 1 -- 172.21.15.1:0/3255919737 >> v1:172.21.15.82:6791/0 conn(0x7ff38c0694b0 legacy=0x7ff38c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1942828248 0 0) 0x7ff374003520 con 0x7ff38c0a46a0 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 >> v1:172.21.15.82:6791/0 conn(0x7ff38c0694b0 legacy=0x7ff38c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 >> v1:172.21.15.1:6789/0 conn(0x7ff38c0a6830 legacy=0x7ff38c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 monclient: found mon.noname-b 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c06dbb0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff384001b30 con 0x7ff38c0a46a0 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 5 monclient: authenticate success, global_id 184349 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: dump: 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff384003610 con 0x7ff38c0a46a0 2024-08-19T10:03:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 4 set_mon_vals no callback set 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 1 -- 172.21.15.1:0/3255919737 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff384003bc0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff3913c1640 10 monclient: dump: 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 10 monclient: shutdown 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 1 -- 172.21.15.1:0/3255919737 >> v1:172.21.15.1:6790/0 conn(0x7ff38c0a46a0 legacy=0x7ff38c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 1 -- 172.21.15.1:0/3255919737 shutdown_connections 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 1 -- 172.21.15.1:0/3255919737 >> 172.21.15.1:0/3255919737 conn(0x7ff38c0a9020 msgr2=0x7ff38c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 1 -- 172.21.15.1:0/3255919737 shutdown_connections 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.510+0000 7ff394627640 1 -- 172.21.15.1:0/3255919737 wait complete. 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 10 monclient: monmap: 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 1 librados: starting msgr at 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 1 librados: starting objecter 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 1 Processor -- start 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 1 -- start start 2024-08-19T10:03:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 1 librados: setting wanted keys 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 1 librados: calling monclient init 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.511+0000 7ff394627640 10 monclient: init 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient: _add_conns ranks=[8,2,6,1,7,0,5,3,4] 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient(hunting): picked mon.noname-e con 0x7ff38c0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient(hunting): picked mon.noname-b con 0x7ff38c0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient(hunting): picked mon.noname-d con 0x7ff38c0a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c138910 con 0x7ff38c0a46a0 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c138bf0 con 0x7ff38c0a6830 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c137d80 con 0x7ff38c0694b0 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff394627640 10 monclient(hunting): authenticate will time out at 1481.588371s 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff3923c3640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff38c0694b0 0x7ff38c135010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41814/0 (socket says 172.21.15.1:41814) 2024-08-19T10:03:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.512+0000 7ff3923c3640 1 -- 172.21.15.1:0/2970873802 learned_addr learned my addr 172.21.15.1:0/2970873802 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff391bc2640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6790/0 conn(0x7ff38c0a46a0 legacy=0x7ff38c135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff3923c3640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6793/0 conn(0x7ff38c0694b0 legacy=0x7ff38c135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 client.?.objecter ms_handle_connect 0x7ff38c0a46a0 2024-08-19T10:03:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 client.?.objecter ms_handle_connect 0x7ff38c0694b0 2024-08-19T10:03:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 client.?.objecter ms_handle_connect 0x7ff38c0a6830 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff392bc4640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6792/0 conn(0x7ff38c0a6830 legacy=0x7ff38c1d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657525644 0 0) 0x7ff38c138910 con 0x7ff38c0a46a0 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): my global_id is 184358 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff378004eb0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922872447 0 0) 0x7ff38c137d80 con 0x7ff38c0694b0 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): my global_id is 184445 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff38c138910 con 0x7ff38c0694b0 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2445110356 0 0) 0x7ff38c138bf0 con 0x7ff38c0a6830 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 10 monclient(hunting): my global_id is 194154 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.514+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff38c137d80 con 0x7ff38c0a6830 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1964198701 0 0) 0x7ff378004eb0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff38c138bf0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff380003610 con 0x7ff38c0a46a0 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:45.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): dump: 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1764387215 0 0) 0x7ff38c138910 con 0x7ff38c0694b0 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff378004eb0 con 0x7ff38c0694b0 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff37c0020a0 con 0x7ff38c0694b0 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): dump: 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.535 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2474711710 0 0) 0x7ff38c137d80 con 0x7ff38c0a6830 2024-08-19T10:03:45.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff38c138910 con 0x7ff38c0a6830 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff384001ab0 con 0x7ff38c0a6830 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient(hunting): dump: 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.537 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3543299411 0 0) 0x7ff38c138bf0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6793/0 conn(0x7ff38c0694b0 legacy=0x7ff38c135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6792/0 conn(0x7ff38c0a6830 legacy=0x7ff38c1d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 monclient: found mon.b 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c135e30 con 0x7ff38c0a46a0 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 5 monclient: authenticate success, global_id 184358 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 10 monclient: _renew_subs 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff38c138350 con 0x7ff38c0a46a0 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 10 client.184358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 10 monclient: _renew_subs 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff38c1373f0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff394627640 1 librados: init done 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff380003b90 con 0x7ff38c0a46a0 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff3800040c0 con 0x7ff38c0a46a0 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff390bc0640 4 set_mon_vals no callback set 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.515+0000 7ff38affd640 10 monclient: dump: 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.540 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff380004260 con 0x7ff38c0a46a0 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff394627640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff36c0065c0 con 0x7ff36c002860 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(488..488 src has 299..488) ==== 5875+0+0 (unknown 3303444826 0 0) 0x7ff38004e370 con 0x7ff38c0a46a0 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff38affd640 10 client.184358.objecter ms_dispatch 0x7ff38c000b90 osd_map(488..488 src has 299..488) 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff38affd640 3 client.184358.objecter handle_osd_map got epochs [488,488] > 0 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff38affd640 3 client.184358.objecter handle_osd_map decoding full epoch 488 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff38affd640 20 client.184358.objecter dump_active .. 0 homeless 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.516+0000 7ff392bc4640 10 monclient: get_auth_request con 0x7ff378041850 auth_method 0 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 10 client.184358.objecter ms_handle_connect 0x7ff36c002860 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 10 client.184358.objecter resend_mon_ops 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff3923c3640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6789/0 conn(0x7ff36c002860 legacy=0x7ff36c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241451920 0 0) 0x7ff36c0065c0 con 0x7ff36c002860 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.518+0000 7ff38affd640 10 monclient(hunting): my global_id is 174735 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff38c138bf0 con 0x7ff36c002860 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 10 client.184358.objecter ms_handle_connect 0x7ff378041850 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1664556992 0 0) 0x7ff38c138bf0 con 0x7ff36c002860 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff36c0065c0 con 0x7ff36c002860 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff37c005540 con 0x7ff36c002860 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 10 monclient: dump: 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.543 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3896220525 0 0) 0x7ff36c0065c0 con 0x7ff36c002860 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff36c00a160 con 0x7ff36c002860 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff37c007720 con 0x7ff36c002860 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6789/0 conn(0x7ff36c002860 legacy=0x7ff36c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.519+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6789/0 conn(0x7ff36c002860 legacy=0x7ff36c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.525+0000 7ff394627640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.525+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff36c009e70 con 0x7ff36c009130 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff3923c3640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6789/0 conn(0x7ff36c009130 legacy=0x7ff36c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 client.184358.objecter ms_handle_connect 0x7ff36c009130 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 client.184358.objecter resend_mon_ops 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2730432939 0 0) 0x7ff36c009e70 con 0x7ff36c009130 2024-08-19T10:03:45.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient(hunting): my global_id is 174744 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff36c0065c0 con 0x7ff36c009130 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1043663214 0 0) 0x7ff36c0065c0 con 0x7ff36c009130 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff36c009e70 con 0x7ff36c009130 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff37c005de0 con 0x7ff36c009130 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 10 monclient: dump: 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.546 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 567526315 0 0) 0x7ff36c009e70 con 0x7ff36c009130 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.528+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff36c00c3f0 con 0x7ff36c009130 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7687 (unknown 0 0 4045473283) 0x7ff37c0073b0 con 0x7ff36c009130 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff38affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff38affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6789/0 conn(0x7ff36c009130 legacy=0x7ff36c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff38affd640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6789/0 conn(0x7ff36c009130 legacy=0x7ff36c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:45.548 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 275569, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:45.549 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.550 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.551 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:45.552 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.553 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:45.557 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.558 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:45.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.562 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.565 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.566 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.567 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:45.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.569 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:45.569 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 10 librados: watch_flush enter 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 10 librados: watch_flush exit 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 20 client.184358.objecter shutdown clearing up homeless session... 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 10 client.184358.objecter successfully canceled tick 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff378041850 legacy=0x7ff378043d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 10 monclient: shutdown 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 >> v1:172.21.15.1:6790/0 conn(0x7ff38c0a46a0 legacy=0x7ff38c135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff392bc4640 1 -- 172.21.15.1:0/2970873802 reap_dead start 2024-08-19T10:03:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff392bc4640 1 -- 172.21.15.1:0/2970873802 reap_dead start 2024-08-19T10:03:45.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 shutdown_connections 2024-08-19T10:03:45.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 >> 172.21.15.1:0/2970873802 conn(0x7ff38c0a9020 msgr2=0x7ff38c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 shutdown_connections 2024-08-19T10:03:45.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 -- 172.21.15.1:0/2970873802 wait complete. 2024-08-19T10:03:45.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.529+0000 7ff394627640 1 librados: shutdown 2024-08-19T10:03:45.572 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:45.572 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:03:45.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:03:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.671+0000 7fa1b6cb3640 10 monclient: get_monmap_and_config 2024-08-19T10:03:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.671+0000 7fa1b6cb3640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.671+0000 7fa1b6cb3640 10 monclient: monmap: 2024-08-19T10:03:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:45.672401+0000 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:45.672401+0000 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 1 Processor -- start 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 1 -- start start 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient: init 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient: _add_conns ranks=[3,8,7,0,6,2,4,5,1] 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient(hunting): picked mon.noname-g con 0x7fa1b00a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient(hunting): picked mon.noname-e con 0x7fa1b00a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient(hunting): picked mon.noname-i con 0x7fa1b013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b00a3b40 con 0x7fa1b00a67c0 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b00a8130 con 0x7fa1b013f5c0 2024-08-19T10:03:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b006ede0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1b6cb3640 10 monclient(hunting): authenticate will time out at 1481.748743s 2024-08-19T10:03:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1affff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fa1b00a6ba0 0x7fa1b00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41826/0 (socket says 172.21.15.1:41826) 2024-08-19T10:03:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.672+0000 7fa1affff640 1 -- 172.21.15.1:0/2257691361 learned_addr learned my addr 172.21.15.1:0/2257691361 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1affff640 1 -- 172.21.15.1:0/2257691361 >> v1:172.21.15.1:6793/0 conn(0x7fa1b00a6ba0 legacy=0x7fa1b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b4a4f640 1 -- 172.21.15.1:0/2257691361 >> v1:172.21.15.82:6790/0 conn(0x7fa1b00a67c0 legacy=0x7fa1b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83960583 0 0) 0x7fa1b006ede0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): my global_id is 184454 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b5250640 1 -- 172.21.15.1:0/2257691361 >> v1:172.21.15.82:6792/0 conn(0x7fa1b013f5c0 legacy=0x7fa1b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a0003520 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429678397 0 0) 0x7fa1b00a3b40 con 0x7fa1b00a67c0 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): my global_id is 184404 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b006ede0 con 0x7fa1b00a67c0 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728208428 0 0) 0x7fa1b00a8130 con 0x7fa1b013f5c0 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient(hunting): my global_id is 145168 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b00a3b40 con 0x7fa1b013f5c0 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1391699316 0 0) 0x7fa1a0003520 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 >> v1:172.21.15.82:6792/0 conn(0x7fa1b013f5c0 legacy=0x7fa1b01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 >> v1:172.21.15.82:6790/0 conn(0x7fa1b00a67c0 legacy=0x7fa1b00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 monclient: found mon.noname-e 2024-08-19T10:03:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b00a92a0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa19c001ad0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b6cb3640 5 monclient: authenticate success, global_id 184454 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b6cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: dump: 2024-08-19T10:03:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b6cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa19c0048a0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 1 -- 172.21.15.1:0/2257691361 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa19c004e10 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b6cb3640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1b6cb3640 4 set_mon_vals no callback set 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.675+0000 7fa1af7fe640 10 monclient: dump: 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 10 monclient: shutdown 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/2257691361 >> v1:172.21.15.1:6793/0 conn(0x7fa1b00a6ba0 legacy=0x7fa1b00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/2257691361 shutdown_connections 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/2257691361 >> 172.21.15.1:0/2257691361 conn(0x7fa1b00ac4b0 msgr2=0x7fa1b00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/2257691361 shutdown_connections 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/2257691361 wait complete. 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 10 monclient: monmap: 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.687 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 librados: starting msgr at 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.676+0000 7fa1b6cb3640 1 librados: starting objecter 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 Processor -- start 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 -- start start 2024-08-19T10:03:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 librados: setting wanted keys 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 librados: calling monclient init 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient: init 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient: _add_conns ranks=[4,2,8,7,1,5,3,6,0] 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b00a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b00a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient(hunting): picked mon.noname-e con 0x7fa1b013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b0139a70 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b00a9cc0 con 0x7fa1b00a67c0 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b013bdb0 con 0x7fa1b013f5c0 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b6cb3640 10 monclient(hunting): authenticate will time out at 1481.753742s 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b4a4f640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fa1b00a67c0 0x7fa1b01ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52564/0 (socket says 172.21.15.1:52564) 2024-08-19T10:03:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1affff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa1b00a6ba0 0x7fa1b01ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45660/0 (socket says 172.21.15.1:45660) 2024-08-19T10:03:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b5250640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fa1b013f5c0 0x7fa1b01d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41830/0 (socket says 172.21.15.1:41830) 2024-08-19T10:03:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.677+0000 7fa1b4a4f640 1 -- 172.21.15.1:0/4104417308 learned_addr learned my addr 172.21.15.1:0/4104417308 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.679+0000 7fa1b4a4f640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6791/0 conn(0x7fa1b00a67c0 legacy=0x7fa1b01ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.679+0000 7fa1affff640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa1b00a6ba0 legacy=0x7fa1b01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.679+0000 7fa1b5250640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6793/0 conn(0x7fa1b013f5c0 legacy=0x7fa1b01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.679+0000 7fa1ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa1b00a67c0 2024-08-19T10:03:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa1b013f5c0 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa1b00a6ba0 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131285073 0 0) 0x7fa1b013bdb0 con 0x7fa1b013f5c0 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): my global_id is 184463 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa194004eb0 con 0x7fa1b013f5c0 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509650081 0 0) 0x7fa1b006ede0 con 0x7fa1b00a67c0 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): my global_id is 194197 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b013bdb0 con 0x7fa1b00a67c0 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976451632 0 0) 0x7fa1b0139a70 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): my global_id is 184367 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b006ede0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2439980957 0 0) 0x7fa1b013bdb0 con 0x7fa1b00a67c0 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1b0139a70 con 0x7fa1b00a67c0 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa1a4003610 con 0x7fa1b00a67c0 2024-08-19T10:03:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): dump: 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 707342579 0 0) 0x7fa1b006ede0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1b013bdb0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1340852381 0 0) 0x7fa194004eb0 con 0x7fa1b013f5c0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1b006ede0 con 0x7fa1b013f5c0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa19c004d90 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): dump: 2024-08-19T10:03:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa1a8001fa0 con 0x7fa1b013f5c0 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient(hunting): dump: 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.702 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 153505751 0 0) 0x7fa1b013bdb0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6793/0 conn(0x7fa1b013f5c0 legacy=0x7fa1b01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6791/0 conn(0x7fa1b00a67c0 legacy=0x7fa1b01ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 monclient: found mon.b 2024-08-19T10:03:45.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b01cf020 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 5 monclient: authenticate success, global_id 184367 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 10 monclient: _renew_subs 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b013af40 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 10 client.184367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 10 monclient: _renew_subs 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1b013a880 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1b6cb3640 1 librados: init done 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa19c001d20 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa19c0048a0 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1aeffd640 4 set_mon_vals no callback set 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.680+0000 7fa1ad7fa640 10 monclient: dump: 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.705 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.706 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.681+0000 7fa1b6cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.681+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa1900065c0 con 0x7fa190002860 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.681+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa19c011a00 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.682+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(488..488 src has 299..488) ==== 5875+0+0 (unknown 3303444826 0 0) 0x7fa19c04e050 con 0x7fa1b00a6ba0 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.682+0000 7fa1ad7fa640 10 client.184367.objecter ms_dispatch 0x7fa1b0000b90 osd_map(488..488 src has 299..488) 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.682+0000 7fa1ad7fa640 3 client.184367.objecter handle_osd_map got epochs [488,488] > 0 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.682+0000 7fa1ad7fa640 3 client.184367.objecter handle_osd_map decoding full epoch 488 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.682+0000 7fa1ad7fa640 20 client.184367.objecter dump_active .. 0 homeless 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.682+0000 7fa1b5250640 10 monclient: get_auth_request con 0x7fa194041ba0 auth_method 0 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 10 client.184367.objecter ms_handle_connect 0x7fa190002860 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 10 client.184367.objecter resend_mon_ops 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1b4a4f640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa190002860 legacy=0x7fa190004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298307442 0 0) 0x7fa1900065c0 con 0x7fa190002860 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 10 monclient(hunting): my global_id is 184376 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.683+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b013bdb0 con 0x7fa190002860 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 726119501 0 0) 0x7fa1b013bdb0 con 0x7fa190002860 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1900065c0 con 0x7fa190002860 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa1a4004170 con 0x7fa190002860 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 10 monclient: dump: 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.708 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.709 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3959273081 0 0) 0x7fa1900065c0 con 0x7fa190002860 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa19000a160 con 0x7fa190002860 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 10 client.184367.objecter ms_handle_connect 0x7fa194041ba0 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa1a40057c0 con 0x7fa190002860 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa190002860 legacy=0x7fa190004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.684+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa190002860 legacy=0x7fa190004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:45.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.690+0000 7fa1b6cb3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.690+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa190009e50 con 0x7fa190009110 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1b4a4f640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa190009110 legacy=0x7fa1900095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 client.184367.objecter ms_handle_connect 0x7fa190009110 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 client.184367.objecter resend_mon_ops 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 674839759 0 0) 0x7fa190009e50 con 0x7fa190009110 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient(hunting): my global_id is 184385 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1900065c0 con 0x7fa190009110 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1556959814 0 0) 0x7fa1900065c0 con 0x7fa190009110 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa190009e50 con 0x7fa190009110 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa1a4005ea0 con 0x7fa190009110 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient: dump: 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3629851670 0 0) 0x7fa190009e50 con 0x7fa190009110 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa19000c3f0 con 0x7fa190009110 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3149287095) 0x7fa1a400ad70 con 0x7fa190009110 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa190009110 legacy=0x7fa1900095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.693+0000 7fa1ad7fa640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa190009110 legacy=0x7fa1900095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:45.714 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 142942, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:45.715 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.716 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.717 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:45.718 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:45.719 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.720 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:45.722 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:45.726 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.730 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.731 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.733 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:45.734 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 10 librados: watch_flush enter 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 10 librados: watch_flush exit 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 20 client.184367.objecter shutdown clearing up homeless session... 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 10 client.184367.objecter successfully canceled tick 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa194041ba0 legacy=0x7fa194044060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 10 monclient: shutdown 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 >> v1:172.21.15.1:6790/0 conn(0x7fa1b00a6ba0 legacy=0x7fa1b01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b5250640 1 -- 172.21.15.1:0/4104417308 reap_dead start 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b5250640 1 -- 172.21.15.1:0/4104417308 reap_dead start 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 shutdown_connections 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 >> 172.21.15.1:0/4104417308 conn(0x7fa1b00ac4b0 msgr2=0x7fa1b01421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 shutdown_connections 2024-08-19T10:03:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 -- 172.21.15.1:0/4104417308 wait complete. 2024-08-19T10:03:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.694+0000 7fa1b6cb3640 1 librados: shutdown 2024-08-19T10:03:45.738 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:45.738 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:03:45.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:03:45.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766024171}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3323408032}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929288570}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=640611083}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2234164835}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1870471407}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1428508487}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779551404}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.720+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2286743373}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1103086850}) 2024-08-19T10:03:45.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150613788}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2215383261}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2952179472}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2198858966}) 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3606914567}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=188071927}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.724+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.725+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614704229}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.725+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1786875110}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835089705}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095732517}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3651261628}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=394536405}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2114195603}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3301349091}) 2024-08-19T10:03:45.748 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.749 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.749 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=16272461}) 2024-08-19T10:03:45.749 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.749 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158594181}) 2024-08-19T10:03:45.749 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.737+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1516187667}) 2024-08-19T10:03:45.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.847+0000 7f1ef7e84640 10 monclient: get_monmap_and_config 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.847+0000 7f1ef7e84640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.847+0000 7f1ef7e84640 10 monclient: monmap: 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:45.848258+0000 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:45.848258+0000 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.849 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.847+0000 7f1ef7e84640 1 Processor -- start 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 1 -- start start 2024-08-19T10:03:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient: init 2024-08-19T10:03:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient: _add_conns ranks=[1,4,6,2,5,0,7,3,8] 2024-08-19T10:03:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient(hunting): picked mon.noname-f con 0x7f1ef00a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient(hunting): picked mon.noname-c con 0x7f1ef00a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient(hunting): picked mon.noname-d con 0x7f1ef013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef00a3b40 con 0x7f1ef00a67c0 2024-08-19T10:03:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef00a8130 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef006ede0 con 0x7f1ef013f5c0 2024-08-19T10:03:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef7e84640 10 monclient(hunting): authenticate will time out at 1481.924596s 2024-08-19T10:03:45.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef541f640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f1ef00a6ba0 0x7f1ef00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52580/0 (socket says 172.21.15.1:52580) 2024-08-19T10:03:45.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef541f640 1 -- 172.21.15.1:0/120421778 learned_addr learned my addr 172.21.15.1:0/120421778 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.848+0000 7f1ef6421640 1 --1- 172.21.15.1:0/120421778 >> v1:172.21.15.1:6792/0 conn(0x7f1ef013f5c0 0x7f1ef01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43812/0 (socket says 172.21.15.1:43812) 2024-08-19T10:03:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.850+0000 7f1ef541f640 1 -- 172.21.15.1:0/120421778 >> v1:172.21.15.1:6791/0 conn(0x7f1ef00a6ba0 legacy=0x7f1ef00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef6421640 1 -- 172.21.15.1:0/120421778 >> v1:172.21.15.1:6792/0 conn(0x7f1ef013f5c0 legacy=0x7f1ef01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef5c20640 1 -- 172.21.15.1:0/120421778 >> v1:172.21.15.82:6789/0 conn(0x7f1ef00a67c0 legacy=0x7f1ef00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253281581 0 0) 0x7f1ef00a8130 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): my global_id is 194206 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1edc003520 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842150864 0 0) 0x7f1ef00a3b40 con 0x7f1ef00a67c0 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient(hunting): my global_id is 174628 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ef00a8130 con 0x7f1ef00a67c0 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2991861334 0 0) 0x7f1edc003520 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 >> v1:172.21.15.1:6792/0 conn(0x7f1ef013f5c0 legacy=0x7f1ef01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 >> v1:172.21.15.82:6789/0 conn(0x7f1ef00a67c0 legacy=0x7f1ef00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 monclient: found mon.noname-c 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef00a92a0 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1ee0001ae0 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: dump: 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef7e84640 5 monclient: authenticate success, global_id 194206 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef7e84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ee00048a0 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 1 -- 172.21.15.1:0/120421778 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1ee0004e10 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef7e84640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef7e84640 4 set_mon_vals no callback set 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f1ef4c1e640 10 monclient: dump: 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.860 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.861 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=804059328}) 2024-08-19T10:03:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907512475}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.851+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1277380524}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1479476295}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1716559824}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3006854301}) 2024-08-19T10:03:45.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3829353322}) 2024-08-19T10:03:45.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266496481}) 2024-08-19T10:03:45.866 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1037320058}) 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 10 monclient: shutdown 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 1 -- 172.21.15.1:0/120421778 >> v1:172.21.15.1:6791/0 conn(0x7f1ef00a6ba0 legacy=0x7f1ef00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 1 -- 172.21.15.1:0/120421778 shutdown_connections 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 1 -- 172.21.15.1:0/120421778 >> 172.21.15.1:0/120421778 conn(0x7f1ef00ac4b0 msgr2=0x7f1ef00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 1 -- 172.21.15.1:0/120421778 shutdown_connections 2024-08-19T10:03:45.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.852+0000 7f1ef7e84640 1 -- 172.21.15.1:0/120421778 wait complete. 2024-08-19T10:03:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070456326}) 2024-08-19T10:03:45.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f1ef7e84640 10 monclient: build_initial_monmap 2024-08-19T10:03:45.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008178440}) 2024-08-19T10:03:45.869 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f1ef7e84640 10 monclient: monmap: 2024-08-19T10:03:45.869 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:45.869 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:45.869 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:45.869 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:45.870 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038015110}) 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=774083802}) 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.872 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=9196591}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4034833716}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4092947056}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848591886}) 2024-08-19T10:03:45.873 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=66550718}) 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f1ef7e84640 1 librados: starting msgr at 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.853+0000 7f1ef7e84640 1 librados: starting objecter 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 1 Processor -- start 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 1 -- start start 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 1 librados: setting wanted keys 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 1 librados: calling monclient init 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 10 monclient: init 2024-08-19T10:03:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 10 monclient: _add_conns ranks=[4,6,7,5,1,8,2,3,0] 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 10 monclient(hunting): picked mon.noname-c con 0x7f1ef00a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 10 monclient(hunting): picked mon.noname-d con 0x7f1ef00a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 10 monclient(hunting): picked mon.noname-i con 0x7f1ef013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef013ae40 con 0x7f1ef00a67c0 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.854+0000 7f1ef7e84640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef013b410 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.855+0000 7f1ef7e84640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ef0139950 con 0x7f1ef013f5c0 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.855+0000 7f1ef7e84640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.855+0000 7f1ef5c20640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f1ef00a67c0 0x7f1ef013c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52596/0 (socket says 172.21.15.1:52596) 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.855+0000 7f1ef7e84640 10 monclient(hunting): authenticate will time out at 1481.931032s 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.855+0000 7f1ef5c20640 1 -- 172.21.15.1:0/1737175862 learned_addr learned my addr 172.21.15.1:0/1737175862 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:45.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef5c20640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ef00a67c0 legacy=0x7f1ef013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 client.?.objecter ms_handle_connect 0x7f1ef00a67c0 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 client.?.objecter ms_handle_connect 0x7f1ef00a6ba0 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef541f640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6792/0 conn(0x7f1ef00a6ba0 legacy=0x7f1ef013c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 client.?.objecter ms_handle_connect 0x7f1ef013f5c0 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef6421640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.82:6792/0 conn(0x7f1ef013f5c0 legacy=0x7f1ef01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989009941 0 0) 0x7f1ef013ae40 con 0x7f1ef00a67c0 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 194215 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed4004ee0 con 0x7f1ef00a67c0 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929805417 0 0) 0x7f1ef00a92a0 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 194163 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ef013ae40 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926292270 0 0) 0x7f1ef0139950 con 0x7f1ef013f5c0 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 145177 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ef00a92a0 con 0x7f1ef013f5c0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2053327135 0 0) 0x7f1ef013ae40 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ef0139950 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1637590014 0 0) 0x7f1ed4004ee0 con 0x7f1ef00a67c0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ef013ae40 con 0x7f1ef00a67c0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1ee0004d90 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): dump: 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:45.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1eec003610 con 0x7f1ef00a67c0 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient(hunting): dump: 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.886 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2475848887 0 0) 0x7f1ef0139950 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.82:6792/0 conn(0x7f1ef013f5c0 legacy=0x7f1ef01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ef00a67c0 legacy=0x7f1ef013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 monclient: found mon.d 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:45.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef013bcb0 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient: _finish_auth 0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 5 monclient: authenticate success, global_id 194163 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 10 monclient: _renew_subs 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef013e520 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 10 client.194163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 10 monclient: _renew_subs 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ef013cfd0 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.857+0000 7f1ef7e84640 1 librados: init done 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ee0001d30 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1ee0001580 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee7fff640 4 set_mon_vals no callback set 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ee67fc640 10 monclient: dump: 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.889 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.858+0000 7f1ef7e84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1ed80065c0 con 0x7f1ed8002860 2024-08-19T10:03:45.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1ee0011a00 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(488..488 src has 299..488) ==== 5875+0+0 (unknown 3303444826 0 0) 0x7f1ee004e340 con 0x7f1ef00a6ba0 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ee67fc640 10 client.194163.objecter ms_dispatch 0x7f1ef0000b90 osd_map(488..488 src has 299..488) 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ee67fc640 3 client.194163.objecter handle_osd_map got epochs [488,488] > 0 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ee67fc640 3 client.194163.objecter handle_osd_map decoding full epoch 488 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ee67fc640 20 client.194163.objecter dump_active .. 0 homeless 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.859+0000 7f1ef6421640 10 monclient: get_auth_request con 0x7f1ed40414d0 auth_method 0 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ef5c20640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ed8002860 legacy=0x7f1ed8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 client.194163.objecter ms_handle_connect 0x7f1ed8002860 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 client.194163.objecter resend_mon_ops 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684100512 0 0) 0x7f1ed80065c0 con 0x7f1ed8002860 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 194224 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ef0139950 con 0x7f1ed8002860 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 597509291 0 0) 0x7f1ef0139950 con 0x7f1ed8002860 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ed80065c0 con 0x7f1ed8002860 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1eec003f20 con 0x7f1ed8002860 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 10 monclient: dump: 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1377785289 0 0) 0x7f1ed80065c0 con 0x7f1ed8002860 2024-08-19T10:03:45.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.861+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ed800a160 con 0x7f1ed8002860 2024-08-19T10:03:45.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.862+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1eec0041f0 con 0x7f1ed8002860 2024-08-19T10:03:45.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.862+0000 7f1ee67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:45.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.862+0000 7f1ee67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:45.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.862+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ed8002860 legacy=0x7f1ed8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.862+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ed8002860 legacy=0x7f1ed8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:45.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.862+0000 7f1ee67fc640 10 client.194163.objecter ms_handle_connect 0x7f1ed40414d0 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3670484865}) 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817545453}) 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=283514625}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522062186}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3049532267}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3798295638}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1472291506}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994674697}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.863+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3891884810}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819310318}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662714603}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=59958186}) 2024-08-19T10:03:45.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=492778347}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3573905125}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3402342019}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3721238215}) 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476130979}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.864+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2299191334}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018704807}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052341557}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2765403017}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1810200609}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3478328583}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3067712606}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607905534}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919633943}) 2024-08-19T10:03:45.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.865+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2838093406}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311459577}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685459579}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2912336251}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2442442562}) 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2840095017}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3603093918}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4193267471}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739193600}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.867+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2682905276}) 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.909 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979982858}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570817246}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2979562875}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=801360461}) 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=8543256}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4088566355}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4263858127}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509788637}) 2024-08-19T10:03:45.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:45.868+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=879793574}) 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.871+0000 7f1ef7e84640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.871+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1ed8009e50 con 0x7f1ed8009110 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ef5c20640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ed8009110 legacy=0x7f1ed80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 client.194163.objecter ms_handle_connect 0x7f1ed8009110 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 client.194163.objecter resend_mon_ops 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793288687 0 0) 0x7f1ed8009e50 con 0x7f1ed8009110 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 194233 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed80065c0 con 0x7f1ed8009110 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4195569739 0 0) 0x7f1ed80065c0 con 0x7f1ed8009110 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ed8009e50 con 0x7f1ed8009110 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1eec005b00 con 0x7f1ed8009110 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.873+0000 7f1ee67fc640 10 monclient: dump: 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3595031701 0 0) 0x7f1ed8009e50 con 0x7f1ed8009110 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ed800c3f0 con 0x7f1ed8009110 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 195270469) 0x7f1eec005e70 con 0x7f1ed8009110 2024-08-19T10:03:45.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ed8009110 legacy=0x7f1ed80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ee67fc640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6791/0 conn(0x7f1ed8009110 legacy=0x7f1ed80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13561, 2024-08-19T10:03:45.916 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:03:45.917 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:45.924 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:45.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:45.937 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:45.937 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 10 librados: watch_flush enter 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 10 librados: watch_flush exit 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 20 client.194163.objecter shutdown clearing up homeless session... 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 10 client.194163.objecter successfully canceled tick 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1ed40414d0 legacy=0x7f1ed4043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 10 monclient: shutdown 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 >> v1:172.21.15.1:6792/0 conn(0x7f1ef00a6ba0 legacy=0x7f1ef013c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef6421640 1 -- 172.21.15.1:0/1737175862 reap_dead start 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef6421640 1 -- 172.21.15.1:0/1737175862 reap_dead start 2024-08-19T10:03:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 shutdown_connections 2024-08-19T10:03:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 >> 172.21.15.1:0/1737175862 conn(0x7f1ef00ac4b0 msgr2=0x7f1ef013ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 shutdown_connections 2024-08-19T10:03:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 -- 172.21.15.1:0/1737175862 wait complete. 2024-08-19T10:03:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:45.874+0000 7f1ef7e84640 1 librados: shutdown 2024-08-19T10:03:45.940 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:45.940 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:03:45.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.039+0000 7f95e0921640 10 monclient: get_monmap_and_config 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.039+0000 7f95e0921640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.039+0000 7f95e0921640 10 monclient: monmap: 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:46.040796+0000 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:46.040796+0000 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.041 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.042 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.040+0000 7f95e0921640 1 Processor -- start 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.040+0000 7f95e0921640 1 -- start start 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.040+0000 7f95e0921640 10 monclient: init 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient: _add_conns ranks=[8,0,2,4,3,5,1,7,6] 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient(hunting): picked mon.noname-e con 0x7f95dc0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient(hunting): picked mon.noname-a con 0x7f95dc0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient(hunting): picked mon.noname-b con 0x7f95dc13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc0a3b40 con 0x7f95dc0a6ba0 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc0a8130 con 0x7f95dc13f5c0 2024-08-19T10:03:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc06ede0 con 0x7f95dc0a67c0 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95e0921640 10 monclient(hunting): authenticate will time out at 1482.117359s 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95da59c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f95dc0a67c0 0x7f95dc0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41836/0 (socket says 172.21.15.1:41836) 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.041+0000 7f95da59c640 1 -- 172.21.15.1:0/2083829394 learned_addr learned my addr 172.21.15.1:0/2083829394 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d9d9b640 1 -- 172.21.15.1:0/2083829394 >> v1:172.21.15.1:6789/0 conn(0x7f95dc0a6ba0 legacy=0x7f95dc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95dad9d640 1 -- 172.21.15.1:0/2083829394 >> v1:172.21.15.1:6790/0 conn(0x7f95dc13f5c0 legacy=0x7f95dc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95da59c640 1 -- 172.21.15.1:0/2083829394 >> v1:172.21.15.1:6793/0 conn(0x7f95dc0a67c0 legacy=0x7f95dc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3948032283 0 0) 0x7f95dc0a3b40 con 0x7f95dc0a6ba0 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): my global_id is 174753 2024-08-19T10:03:46.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95c8003520 con 0x7f95dc0a6ba0 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893753123 0 0) 0x7f95dc0a8130 con 0x7f95dc13f5c0 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): my global_id is 184394 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95dc0a3b40 con 0x7f95dc13f5c0 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2400888643 0 0) 0x7f95dc06ede0 con 0x7f95dc0a67c0 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 10 monclient(hunting): my global_id is 184472 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.043+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95dc0a8130 con 0x7f95dc0a67c0 2024-08-19T10:03:46.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 382828793 0 0) 0x7f95dc0a8130 con 0x7f95dc0a67c0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 >> v1:172.21.15.1:6790/0 conn(0x7f95dc13f5c0 legacy=0x7f95dc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 >> v1:172.21.15.1:6789/0 conn(0x7f95dc0a6ba0 legacy=0x7f95dc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 monclient: found mon.noname-e 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc0a92a0 con 0x7f95dc0a67c0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95cc001cf0 con 0x7f95dc0a67c0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: dump: 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.052 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 5 monclient: authenticate success, global_id 184472 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95cc001fc0 con 0x7f95dc0a67c0 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 1 -- 172.21.15.1:0/2083829394 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95cc004f60 con 0x7f95dc0a67c0 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95d959a640 10 monclient: dump: 2024-08-19T10:03:46.053 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.054 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 4 set_mon_vals no callback set 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 10 monclient: shutdown 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 1 -- 172.21.15.1:0/2083829394 >> v1:172.21.15.1:6793/0 conn(0x7f95dc0a67c0 legacy=0x7f95dc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 1 -- 172.21.15.1:0/2083829394 shutdown_connections 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 1 -- 172.21.15.1:0/2083829394 >> 172.21.15.1:0/2083829394 conn(0x7f95dc0ac4b0 msgr2=0x7f95dc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 1 -- 172.21.15.1:0/2083829394 shutdown_connections 2024-08-19T10:03:46.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.044+0000 7f95e0921640 1 -- 172.21.15.1:0/2083829394 wait complete. 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 10 monclient: monmap: 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 1 librados: starting msgr at 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 1 librados: starting objecter 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 1 Processor -- start 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 1 -- start start 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 1 librados: setting wanted keys 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 1 librados: calling monclient init 2024-08-19T10:03:46.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.045+0000 7f95e0921640 10 monclient: init 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient: _add_conns ranks=[1,3,6,4,0,7,8,2,5] 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient(hunting): picked mon.noname-f con 0x7f95dc0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient(hunting): picked mon.noname-g con 0x7f95dc0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient(hunting): picked mon.noname-d con 0x7f95dc13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc13b380 con 0x7f95dc0a67c0 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc142e40 con 0x7f95dc0a6ba0 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc13d6e0 con 0x7f95dc13f5c0 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95e0921640 10 monclient(hunting): authenticate will time out at 1482.122497s 2024-08-19T10:03:46.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95dad9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f95dc13f5c0 0x7f95dc1d6260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43836/0 (socket says 172.21.15.1:43836) 2024-08-19T10:03:46.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.046+0000 7f95dad9d640 1 -- 172.21.15.1:0/1611222745 learned_addr learned my addr 172.21.15.1:0/1611222745 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.048+0000 7f95dad9d640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95dc13f5c0 legacy=0x7f95dc1d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.048+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.048+0000 7f95da59c640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.82:6789/0 conn(0x7f95dc0a67c0 legacy=0x7f95dc13ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.048+0000 7f95d9d9b640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.82:6790/0 conn(0x7f95dc0a6ba0 legacy=0x7f95dc1d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 client.?.objecter ms_handle_connect 0x7f95dc13f5c0 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 client.?.objecter ms_handle_connect 0x7f95dc0a67c0 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 client.?.objecter ms_handle_connect 0x7f95dc0a6ba0 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630604526 0 0) 0x7f95dc13d6e0 con 0x7f95dc13f5c0 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): my global_id is 194172 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95bc004eb0 con 0x7f95dc13f5c0 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471049763 0 0) 0x7f95dc13b380 con 0x7f95dc0a67c0 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): my global_id is 174637 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95dc13d6e0 con 0x7f95dc0a67c0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 320831746 0 0) 0x7f95c4001630 con 0x7f95dc0a6ba0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 10 monclient(hunting): my global_id is 184413 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.050+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95dc13b380 con 0x7f95dc0a6ba0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1343739492 0 0) 0x7f95bc004eb0 con 0x7f95dc13f5c0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95c4001630 con 0x7f95dc13f5c0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95d4003610 con 0x7f95dc13f5c0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient(hunting): dump: 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2531697282 0 0) 0x7f95c4001630 con 0x7f95dc13f5c0 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.82:6790/0 conn(0x7f95dc0a6ba0 legacy=0x7f95dc1d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.82:6789/0 conn(0x7f95dc0a67c0 legacy=0x7f95dc13ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 monclient: found mon.d 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc1429d0 con 0x7f95dc13f5c0 2024-08-19T10:03:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 5 monclient: authenticate success, global_id 194172 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 10 monclient: _renew_subs 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f95dc13d400 con 0x7f95dc13f5c0 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 10 client.194172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 10 monclient: _renew_subs 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f95dc13c120 con 0x7f95dc13f5c0 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95e0921640 1 librados: init done 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95d4003b40 con 0x7f95dc13f5c0 2024-08-19T10:03:46.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95d4004030 con 0x7f95dc13f5c0 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d8d99640 4 set_mon_vals no callback set 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.051+0000 7f95d2ffd640 10 monclient: dump: 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.069 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f95d4004300 con 0x7f95dc13f5c0 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95e0921640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95c00065c0 con 0x7f95c0002860 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(488..488 src has 299..488) ==== 5875+0+0 (unknown 3303444826 0 0) 0x7f95d404e4e0 con 0x7f95dc13f5c0 2024-08-19T10:03:46.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d2ffd640 10 client.194172.objecter ms_dispatch 0x7f95dc000b90 osd_map(488..488 src has 299..488) 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d2ffd640 3 client.194172.objecter handle_osd_map got epochs [488,488] > 0 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d2ffd640 3 client.194172.objecter handle_osd_map decoding full epoch 488 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d2ffd640 20 client.194172.objecter dump_active .. 0 homeless 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.052+0000 7f95d9d9b640 10 monclient: get_auth_request con 0x7f95bc040dd0 auth_method 0 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95da59c640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95c0002860 legacy=0x7f95c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 client.194172.objecter ms_handle_connect 0x7f95c0002860 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 client.194172.objecter resend_mon_ops 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369170329 0 0) 0x7f95c00065c0 con 0x7f95c0002860 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient(hunting): my global_id is 194181 2024-08-19T10:03:46.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95c4001630 con 0x7f95c0002860 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3226593972 0 0) 0x7f95c4001630 con 0x7f95c0002860 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95c00065c0 con 0x7f95c0002860 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 client.194172.objecter ms_handle_connect 0x7f95bc040dd0 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95cc005040 con 0x7f95c0002860 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.054+0000 7f95d2ffd640 10 monclient: dump: 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.072 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 183883520 0 0) 0x7f95c00065c0 con 0x7f95c0002860 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95c000a160 con 0x7f95c0002860 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f95cc0055e0 con 0x7f95c0002860 2024-08-19T10:03:46.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:46.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95c0002860 legacy=0x7f95c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.055+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95c0002860 legacy=0x7f95c0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.061+0000 7f95e0921640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:46.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.061+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95c0009e70 con 0x7f95c0009130 2024-08-19T10:03:46.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.063+0000 7f95da59c640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95c0009130 legacy=0x7f95c00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 client.194172.objecter ms_handle_connect 0x7f95c0009130 2024-08-19T10:03:46.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 client.194172.objecter resend_mon_ops 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160543774 0 0) 0x7f95c0009e70 con 0x7f95c0009130 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient(hunting): my global_id is 194190 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95c00065c0 con 0x7f95c0009130 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1062751103 0 0) 0x7f95c00065c0 con 0x7f95c0009130 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95c0009e70 con 0x7f95c0009130 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f95cc005da0 con 0x7f95c0009130 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient: dump: 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.077 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.078 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3697387699 0 0) 0x7f95c0009e70 con 0x7f95c0009130 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f95c000c410 con 0x7f95c0009130 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 450845825) 0x7f95cc006130 con 0x7f95c0009130 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95c0009130 legacy=0x7f95c00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.064+0000 7f95d2ffd640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95c0009130 legacy=0x7f95c00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.079 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13751, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:46.080 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.081 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.082 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:46.083 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:46.084 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.085 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:46.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.089 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.091 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.092 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.093 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:46.096 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.097 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.098 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:46.099 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:46.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 10 librados: watch_flush enter 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 10 librados: watch_flush exit 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 20 client.194172.objecter shutdown clearing up homeless session... 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 10 client.194172.objecter successfully canceled tick 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.82:6812/2549104415 conn(0x7f95bc040dd0 legacy=0x7f95bc043290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 10 monclient: shutdown 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 >> v1:172.21.15.1:6792/0 conn(0x7f95dc13f5c0 legacy=0x7f95dc1d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95dad9d640 1 -- 172.21.15.1:0/1611222745 reap_dead start 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95dad9d640 1 -- 172.21.15.1:0/1611222745 reap_dead start 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 shutdown_connections 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 >> 172.21.15.1:0/1611222745 conn(0x7f95dc0ac4b0 msgr2=0x7f95dc1419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 shutdown_connections 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 -- 172.21.15.1:0/1611222745 wait complete. 2024-08-19T10:03:46.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.065+0000 7f95e0921640 1 librados: shutdown 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=207036029}) 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4120744356}) 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.105 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1509337899}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=157215711}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3529495371}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1308683089}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.106 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3574118729}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597348432}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.072+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2102348376}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126995479}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450352408}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=729939741}) 2024-08-19T10:03:46.107 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2994575323}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=784532249}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3475902120}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=307643293}) 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.108 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151166017}) 2024-08-19T10:03:46.109 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.074+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1517040486}) 2024-08-19T10:03:46.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:46.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:03:46.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:03:46.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2760257052}) 2024-08-19T10:03:46.121 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218765929}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=95084449}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4096157804}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1601422558}) 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.122 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3186721374}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3052805790}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308700844}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.075+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=533296445}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3944688261}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.123 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=188436072}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154321342}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3834860663}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3963128266}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.076+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=252546506}) 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.124 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=420752025}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625076406}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.077+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4058645658}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=304015993}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1929917300}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.125 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1645548045}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2872422798}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2869549847}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3042176656}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4243696529}) 2024-08-19T10:03:46.126 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984567517}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.078+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1038432850}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200825841}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684682264}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902021501}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.127 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=563874093}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4282674511}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2663573968}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3206399743}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396194327}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.079+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1288572848}) 2024-08-19T10:03:46.128 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3235962025}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283370042}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1865710599}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1156250787}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.129 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4075844714}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3888983560}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3623263815}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222189502}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.080+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=594322262}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301284885}) 2024-08-19T10:03:46.130 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2071179071}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3231706850}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2270274025}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3247123323}) 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.131 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.081+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1406164619}) 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3663827819}) 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257210002}) 2024-08-19T10:03:46.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.082+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3988647270}) 2024-08-19T10:03:46.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.231+0000 7f33ebf98640 10 monclient: get_monmap_and_config 2024-08-19T10:03:46.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.231+0000 7f33ebf98640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.231+0000 7f33ebf98640 10 monclient: monmap: 2024-08-19T10:03:46.232 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:46.232283+0000 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:46.232283+0000 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.233 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.234 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.234 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.234 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.234 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.231+0000 7f33ebf98640 1 Processor -- start 2024-08-19T10:03:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 1 -- start start 2024-08-19T10:03:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient: init 2024-08-19T10:03:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient: _add_conns ranks=[0,6,2,5,1,8,4,7,3] 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient(hunting): picked mon.noname-a con 0x7f33e40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient(hunting): picked mon.noname-d con 0x7f33e40a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient(hunting): picked mon.noname-b con 0x7f33e413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e40a3b40 con 0x7f33e40a67c0 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e40a8130 con 0x7f33e413f5c0 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e406ede0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ebf98640 10 monclient(hunting): authenticate will time out at 1482.308670s 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33e9d34640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33e40a67c0 0x7f33e40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57280/0 (socket says 172.21.15.1:57280) 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33e9533640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f33e40a6ba0 0x7f33e40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43852/0 (socket says 172.21.15.1:43852) 2024-08-19T10:03:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33e9d34640 1 -- 172.21.15.1:0/2447939409 learned_addr learned my addr 172.21.15.1:0/2447939409 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.232+0000 7f33ea535640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f33e413f5c0 0x7f33e41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45696/0 (socket says 172.21.15.1:45696) 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.234+0000 7f33e9d34640 1 -- 172.21.15.1:0/2447939409 >> v1:172.21.15.1:6789/0 conn(0x7f33e40a67c0 legacy=0x7f33e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.234+0000 7f33e9533640 1 -- 172.21.15.1:0/2447939409 >> v1:172.21.15.1:6792/0 conn(0x7f33e40a6ba0 legacy=0x7f33e40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33ea535640 1 -- 172.21.15.1:0/2447939409 >> v1:172.21.15.1:6790/0 conn(0x7f33e413f5c0 legacy=0x7f33e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3292783614 0 0) 0x7f33e40a3b40 con 0x7f33e40a67c0 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): my global_id is 174762 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33d0003520 con 0x7f33e40a67c0 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300472473 0 0) 0x7f33e406ede0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): my global_id is 194199 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33e40a3b40 con 0x7f33e40a6ba0 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315017796 0 0) 0x7f33e40a8130 con 0x7f33e413f5c0 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient(hunting): my global_id is 184403 2024-08-19T10:03:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33e406ede0 con 0x7f33e413f5c0 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3947204358 0 0) 0x7f33e40a3b40 con 0x7f33e40a6ba0 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 >> v1:172.21.15.1:6790/0 conn(0x7f33e413f5c0 legacy=0x7f33e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 >> v1:172.21.15.1:6789/0 conn(0x7f33e40a67c0 legacy=0x7f33e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 monclient: found mon.noname-d 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e40a92a0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f33d40019d0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: dump: 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.244 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33ebf98640 5 monclient: authenticate success, global_id 194199 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33ebf98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33d4001ca0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 1 -- 172.21.15.1:0/2447939409 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f33d4004b20 con 0x7f33e40a6ba0 2024-08-19T10:03:46.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33ebf98640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33ebf98640 4 set_mon_vals no callback set 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.235+0000 7f33e8d32640 10 monclient: dump: 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 10 monclient: shutdown 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 -- 172.21.15.1:0/2447939409 >> v1:172.21.15.1:6792/0 conn(0x7f33e40a6ba0 legacy=0x7f33e40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 -- 172.21.15.1:0/2447939409 shutdown_connections 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 -- 172.21.15.1:0/2447939409 >> 172.21.15.1:0/2447939409 conn(0x7f33e40ac4b0 msgr2=0x7f33e40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 -- 172.21.15.1:0/2447939409 shutdown_connections 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 -- 172.21.15.1:0/2447939409 wait complete. 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 10 monclient: monmap: 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.248 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 librados: starting msgr at 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.236+0000 7f33ebf98640 1 librados: starting objecter 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 Processor -- start 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 -- start start 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 librados: setting wanted keys 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 librados: calling monclient init 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient: init 2024-08-19T10:03:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient: _add_conns ranks=[3,5,0,8,7,6,4,2,1] 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient(hunting): picked mon.noname-g con 0x7f33e40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient(hunting): picked mon.noname-h con 0x7f33e40a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient(hunting): picked mon.noname-a con 0x7f33e413f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e413b380 con 0x7f33e413f5c0 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e4142e40 con 0x7f33e40a67c0 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e413d6e0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ebf98640 10 monclient(hunting): authenticate will time out at 1482.313606s 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ea535640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33e413f5c0 0x7f33e41d6260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57282/0 (socket says 172.21.15.1:57282) 2024-08-19T10:03:46.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.237+0000 7f33ea535640 1 -- 172.21.15.1:0/682324895 learned_addr learned my addr 172.21.15.1:0/682324895 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33ea535640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6789/0 conn(0x7f33e413f5c0 legacy=0x7f33e41d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 10 client.?.objecter ms_handle_connect 0x7f33e413f5c0 2024-08-19T10:03:46.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360876691 0 0) 0x7f33e413b380 con 0x7f33e413f5c0 2024-08-19T10:03:46.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 10 monclient(hunting): my global_id is 174771 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.239+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33c8004eb0 con 0x7f33e413f5c0 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 client.?.objecter ms_handle_connect 0x7f33e40a67c0 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33e9d34640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.82:6790/0 conn(0x7f33e40a67c0 legacy=0x7f33e413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33e9533640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.82:6791/0 conn(0x7f33e40a6ba0 legacy=0x7f33e41d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 client.?.objecter ms_handle_connect 0x7f33e40a6ba0 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3291565481 0 0) 0x7f33c8004eb0 con 0x7f33e413f5c0 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33e413b380 con 0x7f33e413f5c0 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f33dc002050 con 0x7f33e413f5c0 2024-08-19T10:03:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): dump: 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 945221552 0 0) 0x7f33e00015e0 con 0x7f33e40a67c0 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): my global_id is 184422 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33c8004eb0 con 0x7f33e40a67c0 2024-08-19T10:03:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1301676322 0 0) 0x7f33e413d6e0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient(hunting): my global_id is 27698 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33e00015e0 con 0x7f33e40a6ba0 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1443056385 0 0) 0x7f33e413b380 con 0x7f33e413f5c0 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.82:6791/0 conn(0x7f33e40a6ba0 legacy=0x7f33e41d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.82:6790/0 conn(0x7f33e40a67c0 legacy=0x7f33e413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 monclient: found mon.a 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e41429d0 con 0x7f33e413f5c0 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 5 monclient: authenticate success, global_id 174771 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 10 monclient: _renew_subs 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f33e413d400 con 0x7f33e413f5c0 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 10 client.174771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 10 monclient: _renew_subs 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f33e413c120 con 0x7f33e413f5c0 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33ebf98640 1 librados: init done 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33dc004e90 con 0x7f33e413f5c0 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f33dc005380 con 0x7f33e413f5c0 2024-08-19T10:03:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33dbfff640 4 set_mon_vals no callback set 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.240+0000 7f33da7fc640 10 monclient: dump: 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33ebf98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f33cc0065c0 con 0x7f33cc002860 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f33dc0055e0 con 0x7f33e413f5c0 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(488..488 src has 299..488) ==== 5875+0+0 (unknown 3303444826 0 0) 0x7f33dc04e730 con 0x7f33e413f5c0 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33da7fc640 10 client.174771.objecter ms_dispatch 0x7f33e4000b90 osd_map(488..488 src has 299..488) 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33da7fc640 3 client.174771.objecter handle_osd_map got epochs [488,488] > 0 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33da7fc640 3 client.174771.objecter handle_osd_map decoding full epoch 488 2024-08-19T10:03:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33da7fc640 20 client.174771.objecter dump_active .. 0 homeless 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.241+0000 7f33e9533640 10 monclient: get_auth_request con 0x7f33c80411b0 auth_method 0 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 client.174771.objecter ms_handle_connect 0x7f33cc002860 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33e9d34640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6793/0 conn(0x7f33cc002860 legacy=0x7f33cc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 client.174771.objecter resend_mon_ops 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824011495 0 0) 0x7f33cc0065c0 con 0x7f33cc002860 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient(hunting): my global_id is 184481 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33e413b380 con 0x7f33cc002860 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2214491660 0 0) 0x7f33e413b380 con 0x7f33cc002860 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33cc0065c0 con 0x7f33cc002860 2024-08-19T10:03:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f33e0001a20 con 0x7f33cc002860 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 10 monclient: dump: 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3509106111 0 0) 0x7f33cc0065c0 con 0x7f33cc002860 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.243+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33cc00a160 con 0x7f33cc002860 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.244+0000 7f33da7fc640 10 client.174771.objecter ms_handle_connect 0x7f33c80411b0 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.244+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f33e0001b50 con 0x7f33cc002860 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.244+0000 7f33da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.244+0000 7f33da7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.244+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6793/0 conn(0x7f33cc002860 legacy=0x7f33cc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.244+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6793/0 conn(0x7f33cc002860 legacy=0x7f33cc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312064751}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586763673}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1220692690}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3543466786}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=988050719}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4131010686}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3647754343}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058668722}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.245+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1819408412}) 2024-08-19T10:03:46.276 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263242874}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=328183571}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=386446651}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1413529}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.277 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=689863199}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1574328495}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1148404672}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787765159}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.247+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3587052749}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386210077}) 2024-08-19T10:03:46.278 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448229870}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2513231981}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=981897363}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1630318424}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.279 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3642487962}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3034292473}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637845758}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.248+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3366117008}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161173595}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342629926}) 2024-08-19T10:03:46.280 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3358163707}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1902718243}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.249+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1934633598}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=500300902}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.281 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=294834609}) 2024-08-19T10:03:46.282 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.282 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536203676}) 2024-08-19T10:03:46.282 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.250+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=189189153}) 2024-08-19T10:03:46.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.252+0000 7f33ebf98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:46.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.252+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f33cc009e50 con 0x7f33cc009110 2024-08-19T10:03:46.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33e9d34640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6793/0 conn(0x7f33cc009110 legacy=0x7f33cc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 client.174771.objecter ms_handle_connect 0x7f33cc009110 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 client.174771.objecter resend_mon_ops 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177107600 0 0) 0x7f33cc009e50 con 0x7f33cc009110 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient(hunting): my global_id is 184490 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33cc0065c0 con 0x7f33cc009110 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 646797889 0 0) 0x7f33cc0065c0 con 0x7f33cc009110 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33cc009e50 con 0x7f33cc009110 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f33e0005390 con 0x7f33cc009110 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:46.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient: dump: 2024-08-19T10:03:46.285 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.285 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.285 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.285 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2847190232 0 0) 0x7f33cc009e50 con 0x7f33cc009110 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f33cc00c3a0 con 0x7f33cc009110 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3137823490) 0x7f33e00076b0 con 0x7f33cc009110 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6793/0 conn(0x7f33cc009110 legacy=0x7f33cc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.255+0000 7f33da7fc640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6793/0 conn(0x7f33cc009110 legacy=0x7f33cc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 143504, 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:46.289 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:46.290 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:46.292 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.293 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:46.294 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.297 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.301 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:46.302 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:46.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.308 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.308 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:46.308 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:46.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 10 librados: watch_flush enter 2024-08-19T10:03:46.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 10 librados: watch_flush exit 2024-08-19T10:03:46.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 20 client.174771.objecter shutdown clearing up homeless session... 2024-08-19T10:03:46.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 10 client.174771.objecter successfully canceled tick 2024-08-19T10:03:46.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.82:6812/2549104415 conn(0x7f33c80411b0 legacy=0x7f33c8043670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 10 monclient: shutdown 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 >> v1:172.21.15.1:6789/0 conn(0x7f33e413f5c0 legacy=0x7f33e41d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ea535640 1 -- 172.21.15.1:0/682324895 reap_dead start 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ea535640 1 -- 172.21.15.1:0/682324895 reap_dead start 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 shutdown_connections 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 >> 172.21.15.1:0/682324895 conn(0x7f33e40ac4b0 msgr2=0x7f33e41419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 shutdown_connections 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 -- 172.21.15.1:0/682324895 wait complete. 2024-08-19T10:03:46.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.256+0000 7f33ebf98640 1 librados: shutdown 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14748732}) 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3307289580}) 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=605057617}) 2024-08-19T10:03:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=283674566}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=428565068}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2286776911}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2254366577}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521147727}) 2024-08-19T10:03:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.262+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2879469911}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.263+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.263+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.263+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685247886}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.263+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3851546754}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3052824362}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3754109700}) 2024-08-19T10:03:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=443857526}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2531842794}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2232638175}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179183684}) 2024-08-19T10:03:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.264+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=67393128}) 2024-08-19T10:03:46.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:46.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:03:46.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:03:46.330 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.330 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941295100}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252776326}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859108471}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1652561517}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.331 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3544890536}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=306083527}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2253486608}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383429601}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.273+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3124362163}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=659844965}) 2024-08-19T10:03:46.332 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955558663}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=16407610}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.274+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3454407314}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2359112021}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.333 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=391402304}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2391645335}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667670134}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.275+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3673424603}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285834542}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1077173562}) 2024-08-19T10:03:46.334 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1624016481}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=540833199}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2679764901}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1951236653}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.335 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3839471314}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822467956}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.276+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3547502058}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699905906}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609031888}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2658199247}) 2024-08-19T10:03:46.336 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2246833874}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3877267533}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=713206094}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494932718}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308940871}) 2024-08-19T10:03:46.337 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.277+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1231042198}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.278+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.278+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984971240}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659766129}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1721695293}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2923550909}) 2024-08-19T10:03:46.338 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3819933872}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=794902746}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=576797065}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357907806}) 2024-08-19T10:03:46.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.279+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1313197193}) 2024-08-19T10:03:46.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769245438}) 2024-08-19T10:03:46.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2539043623}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3895087989}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1944426356}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1366754271}) 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3335056495}) 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=395032116}) 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57924579}) 2024-08-19T10:03:46.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.335+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2994594951}) 2024-08-19T10:03:46.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835553619}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950905038}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3257553308}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=847583633}) 2024-08-19T10:03:46.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2044876222}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3455646634}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803086235}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6386087}) 2024-08-19T10:03:46.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.341+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3888507296}) 2024-08-19T10:03:46.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263887969}) 2024-08-19T10:03:46.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2405800328}) 2024-08-19T10:03:46.358 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4282152636}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086934997}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=620939561}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.356+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1573854193}) 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.359 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1760346188}) 2024-08-19T10:03:46.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058901276}) 2024-08-19T10:03:46.360 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.357+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3113525557}) 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2531146711}) 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959769522}) 2024-08-19T10:03:46.362 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2806231652}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4020840614}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=780517634}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=404228679}) 2024-08-19T10:03:46.363 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3310520580}) 2024-08-19T10:03:46.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.359+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225314848}) 2024-08-19T10:03:46.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.360+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1437559536}) 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.446+0000 7fd517786640 10 monclient: get_monmap_and_config 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.446+0000 7fd517786640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.446+0000 7fd517786640 10 monclient: monmap: 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:46.447848+0000 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:46.447848+0000 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.449 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.447+0000 7fd517786640 1 Processor -- start 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.447+0000 7fd517786640 1 -- start start 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.447+0000 7fd517786640 10 monclient: init 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient: _add_conns ranks=[8,3,0,5,7,1,4,6,2] 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient(hunting): picked mon.noname-e con 0x7fd5100a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient(hunting): picked mon.noname-g con 0x7fd5100a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient(hunting): picked mon.noname-a con 0x7fd51013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5100a3b40 con 0x7fd51013f5c0 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5100a8130 con 0x7fd5100a6ba0 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51006ede0 con 0x7fd5100a67c0 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd517786640 10 monclient(hunting): authenticate will time out at 1482.524268s 2024-08-19T10:03:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd515d23640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd51013f5c0 0x7fd5101419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57296/0 (socket says 172.21.15.1:57296) 2024-08-19T10:03:46.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd515d23640 1 -- 172.21.15.1:0/3410394628 learned_addr learned my addr 172.21.15.1:0/3410394628 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.448+0000 7fd515522640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd5100a67c0 0x7fd5100a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:41870/0 (socket says 172.21.15.1:41870) 2024-08-19T10:03:46.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd515522640 1 -- 172.21.15.1:0/3410394628 >> v1:172.21.15.1:6793/0 conn(0x7fd5100a67c0 legacy=0x7fd5100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd515d23640 1 -- 172.21.15.1:0/3410394628 >> v1:172.21.15.1:6789/0 conn(0x7fd51013f5c0 legacy=0x7fd5101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd514d21640 1 -- 172.21.15.1:0/3410394628 >> v1:172.21.15.82:6790/0 conn(0x7fd5100a6ba0 legacy=0x7fd5100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 590862672 0 0) 0x7fd51006ede0 con 0x7fd5100a67c0 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd50ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd50ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.450+0000 7fd50ffff640 10 monclient(hunting): my global_id is 184499 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4f8003520 con 0x7fd5100a67c0 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012176241 0 0) 0x7fd5100a3b40 con 0x7fd51013f5c0 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): my global_id is 174780 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd51006ede0 con 0x7fd51013f5c0 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 572498645 0 0) 0x7fd5100a8130 con 0x7fd5100a6ba0 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient(hunting): my global_id is 184431 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5100a3b40 con 0x7fd5100a6ba0 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 490746910 0 0) 0x7fd4f8003520 con 0x7fd5100a67c0 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 >> v1:172.21.15.82:6790/0 conn(0x7fd5100a6ba0 legacy=0x7fd5100a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 >> v1:172.21.15.1:6789/0 conn(0x7fd51013f5c0 legacy=0x7fd5101419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 monclient: found mon.noname-e 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5100a92a0 con 0x7fd5100a67c0 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd500001e50 con 0x7fd5100a67c0 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 5 monclient: authenticate success, global_id 184499 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: dump: 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd500004b60 con 0x7fd5100a67c0 2024-08-19T10:03:46.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 1 -- 172.21.15.1:0/3410394628 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd500005090 con 0x7fd5100a67c0 2024-08-19T10:03:46.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:46.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 4 set_mon_vals no callback set 2024-08-19T10:03:46.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd50ffff640 10 monclient: dump: 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.462 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 10 monclient: shutdown 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 1 -- 172.21.15.1:0/3410394628 >> v1:172.21.15.1:6793/0 conn(0x7fd5100a67c0 legacy=0x7fd5100a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 1 -- 172.21.15.1:0/3410394628 shutdown_connections 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 1 -- 172.21.15.1:0/3410394628 >> 172.21.15.1:0/3410394628 conn(0x7fd5100ac4b0 msgr2=0x7fd5100a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 1 -- 172.21.15.1:0/3410394628 shutdown_connections 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.451+0000 7fd517786640 1 -- 172.21.15.1:0/3410394628 wait complete. 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 10 monclient: monmap: 2024-08-19T10:03:46.463 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.464 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 1 librados: starting msgr at 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 1 librados: starting objecter 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 1 Processor -- start 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 1 -- start start 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 1 librados: setting wanted keys 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 1 librados: calling monclient init 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.452+0000 7fd517786640 10 monclient: init 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient: _add_conns ranks=[1,4,6,0,3,5,2,7,8] 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient(hunting): picked mon.noname-f con 0x7fd5100a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient(hunting): picked mon.noname-c con 0x7fd5100a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:46.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient(hunting): picked mon.noname-d con 0x7fd51013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5101cf020 con 0x7fd5100a67c0 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51013e400 con 0x7fd5100a6ba0 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd510139bb0 con 0x7fd51013f5c0 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd517786640 10 monclient(hunting): authenticate will time out at 1482.529321s 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd514d21640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd5100a6ba0 0x7fd5101ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52614/0 (socket says 172.21.15.1:52614) 2024-08-19T10:03:46.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.453+0000 7fd514d21640 1 -- 172.21.15.1:0/790425723 learned_addr learned my addr 172.21.15.1:0/790425723 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd515d23640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.1:6792/0 conn(0x7fd51013f5c0 legacy=0x7fd5101d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 client.?.objecter ms_handle_connect 0x7fd51013f5c0 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 client.?.objecter ms_handle_connect 0x7fd5100a6ba0 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd514d21640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.1:6791/0 conn(0x7fd5100a6ba0 legacy=0x7fd5101ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd515522640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd5100a67c0 legacy=0x7fd5101ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 client.?.objecter ms_handle_connect 0x7fd5100a67c0 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1937187415 0 0) 0x7fd510139bb0 con 0x7fd51013f5c0 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): my global_id is 194208 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4fc004eb0 con 0x7fd51013f5c0 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 544058407 0 0) 0x7fd51013e400 con 0x7fd5100a6ba0 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): my global_id is 194242 2024-08-19T10:03:46.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd510139bb0 con 0x7fd5100a6ba0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353562129 0 0) 0x7fd5101cf020 con 0x7fd5100a67c0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 10 monclient(hunting): my global_id is 174646 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd51013e400 con 0x7fd5100a67c0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3543877964 0 0) 0x7fd4fc004eb0 con 0x7fd51013f5c0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.455+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5101cf020 con 0x7fd51013f5c0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd508001c80 con 0x7fd51013f5c0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): dump: 2024-08-19T10:03:46.475 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3634455176 0 0) 0x7fd510139bb0 con 0x7fd5100a6ba0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4fc004eb0 con 0x7fd5100a6ba0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 254775980 0 0) 0x7fd51013e400 con 0x7fd5100a67c0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd510139bb0 con 0x7fd5100a67c0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd504003610 con 0x7fd5100a6ba0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): dump: 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd500005050 con 0x7fd5100a67c0 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient(hunting): dump: 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2130799687 0 0) 0x7fd5101cf020 con 0x7fd51013f5c0 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.1:6791/0 conn(0x7fd5100a6ba0 legacy=0x7fd5101ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd5100a67c0 legacy=0x7fd5101ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 monclient: found mon.d 2024-08-19T10:03:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51013bf10 con 0x7fd51013f5c0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 5 monclient: authenticate success, global_id 194208 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 10 monclient: _renew_subs 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd51013b670 con 0x7fd51013f5c0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd508004a90 con 0x7fd51013f5c0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd508004f80 con 0x7fd51013f5c0 2024-08-19T10:03:46.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50dffb640 10 monclient: dump: 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.482 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 10 client.194208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 10 monclient: _renew_subs 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5100a2890 con 0x7fd51013f5c0 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd50f7fe640 4 set_mon_vals no callback set 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.456+0000 7fd517786640 1 librados: init done 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd5080114e0 con 0x7fd51013f5c0 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd517786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd510143070 con 0x7fd51009da50 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(489..489 src has 299..489) ==== 5888+0+0 (unknown 3091791536 0 0) 0x7fd50804e1f0 con 0x7fd51013f5c0 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd50dffb640 10 client.194208.objecter ms_dispatch 0x7fd510000b90 osd_map(489..489 src has 299..489) 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd50dffb640 3 client.194208.objecter handle_osd_map got epochs [489,489] > 0 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd50dffb640 3 client.194208.objecter handle_osd_map decoding full epoch 489 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd50dffb640 20 client.194208.objecter dump_active .. 0 homeless 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.457+0000 7fd514d21640 10 monclient: get_auth_request con 0x7fd4fc0419e0 auth_method 0 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 10 client.194208.objecter ms_handle_connect 0x7fd51009da50 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 10 client.194208.objecter resend_mon_ops 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd515522640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd51009da50 legacy=0x7fd51009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1070804939 0 0) 0x7fd510139bb0 con 0x7fd51009da50 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 10 monclient(hunting): my global_id is 174655 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.459+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5101cf020 con 0x7fd51009da50 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 10 client.194208.objecter ms_handle_connect 0x7fd4fc0419e0 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3595927019 0 0) 0x7fd5101cf020 con 0x7fd51009da50 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd510139bb0 con 0x7fd51009da50 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd500004b60 con 0x7fd51009da50 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 10 monclient: dump: 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.485 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2968684715 0 0) 0x7fd510139bb0 con 0x7fd51009da50 2024-08-19T10:03:46.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5101427b0 con 0x7fd51009da50 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd500005710 con 0x7fd51009da50 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd51009da50 legacy=0x7fd51009de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.460+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd51009da50 legacy=0x7fd51009de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.466+0000 7fd517786640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.466+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd51013bac0 con 0x7fd51009e540 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd515522640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd51009e540 legacy=0x7fd5101dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 client.194208.objecter ms_handle_connect 0x7fd51009e540 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 client.194208.objecter resend_mon_ops 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909147503 0 0) 0x7fd51013bac0 con 0x7fd51009e540 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient(hunting): my global_id is 174664 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd510139bb0 con 0x7fd51009e540 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 224913514 0 0) 0x7fd510139bb0 con 0x7fd51009e540 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd51013bac0 con 0x7fd51009e540 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd500005f50 con 0x7fd51009e540 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 10 monclient: dump: 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2399813323 0 0) 0x7fd51013bac0 con 0x7fd51009e540 2024-08-19T10:03:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.469+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd510068f60 con 0x7fd51009e540 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 232303309) 0x7fd500008270 con 0x7fd51009e540 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd50dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd50dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd51009e540 legacy=0x7fd5101dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd50dffb640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6789/0 conn(0x7fd51009e540 legacy=0x7fd5101dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:46.490 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 276529, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:46.491 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.492 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.493 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.494 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:46.496 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.497 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.498 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.500 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.501 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.503 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.504 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.505 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.508 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.509 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:46.510 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 10 librados: watch_flush enter 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 10 librados: watch_flush exit 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 20 client.194208.objecter shutdown clearing up homeless session... 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 10 client.194208.objecter successfully canceled tick 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd4fc0419e0 legacy=0x7fd4fc043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 10 monclient: shutdown 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 >> v1:172.21.15.1:6792/0 conn(0x7fd51013f5c0 legacy=0x7fd5101d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd515d23640 1 -- 172.21.15.1:0/790425723 reap_dead start 2024-08-19T10:03:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd515d23640 1 -- 172.21.15.1:0/790425723 reap_dead start 2024-08-19T10:03:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 shutdown_connections 2024-08-19T10:03:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 >> 172.21.15.1:0/790425723 conn(0x7fd5100ac4b0 msgr2=0x7fd5101419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 shutdown_connections 2024-08-19T10:03:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 -- 172.21.15.1:0/790425723 wait complete. 2024-08-19T10:03:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.470+0000 7fd517786640 1 librados: shutdown 2024-08-19T10:03:46.513 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:46.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:03:46.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:03:46.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.651+0000 7fb4a0e27640 10 monclient: get_monmap_and_config 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.651+0000 7fb4a0e27640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.651+0000 7fb4a0e27640 10 monclient: monmap: 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:46.652683+0000 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:46.652683+0000 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.652+0000 7fb4a0e27640 1 Processor -- start 2024-08-19T10:03:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.652+0000 7fb4a0e27640 1 -- start start 2024-08-19T10:03:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.652+0000 7fb4a0e27640 10 monclient: init 2024-08-19T10:03:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.652+0000 7fb4a0e27640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 10 monclient: _add_conns ranks=[1,0,4,8,5,7,2,6,3] 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 10 monclient(hunting): picked mon.noname-f con 0x7fb49c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 10 monclient(hunting): picked mon.noname-a con 0x7fb49c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 10 monclient(hunting): picked mon.noname-c con 0x7fb49c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c0a3b40 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c0a8130 con 0x7fb49c0a67c0 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c06ede0 con 0x7fb49c13f5c0 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb4a0e27640 10 monclient(hunting): authenticate will time out at 1482.729115s 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb493fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb49c0a6ba0 0x7fb49c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57302/0 (socket says 172.21.15.1:57302) 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb493fff640 1 -- 172.21.15.1:0/746573692 learned_addr learned my addr 172.21.15.1:0/746573692 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.653+0000 7fb49ad9d640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fb49c13f5c0 0x7fb49c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:52626/0 (socket says 172.21.15.1:52626) 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb493fff640 1 -- 172.21.15.1:0/746573692 >> v1:172.21.15.1:6789/0 conn(0x7fb49c0a6ba0 legacy=0x7fb49c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb49ad9d640 1 -- 172.21.15.1:0/746573692 >> v1:172.21.15.1:6791/0 conn(0x7fb49c13f5c0 legacy=0x7fb49c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb49a59c640 1 -- 172.21.15.1:0/746573692 >> v1:172.21.15.82:6789/0 conn(0x7fb49c0a67c0 legacy=0x7fb49c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484007586 0 0) 0x7fb49c0a3b40 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): my global_id is 174789 2024-08-19T10:03:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb488003520 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856242698 0 0) 0x7fb49c06ede0 con 0x7fb49c13f5c0 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): my global_id is 194251 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb49c0a3b40 con 0x7fb49c13f5c0 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3937346639 0 0) 0x7fb49c0a8130 con 0x7fb49c0a67c0 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient(hunting): my global_id is 174673 2024-08-19T10:03:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb49c06ede0 con 0x7fb49c0a67c0 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 356181148 0 0) 0x7fb488003520 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 >> v1:172.21.15.1:6791/0 conn(0x7fb49c13f5c0 legacy=0x7fb49c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 >> v1:172.21.15.82:6789/0 conn(0x7fb49c0a67c0 legacy=0x7fb49c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 monclient: found mon.noname-a 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c0a92a0 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb4a0e27640 5 monclient: authenticate success, global_id 174789 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb484001ae0 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: dump: 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb4a0e27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4840048a0 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 1 -- 172.21.15.1:0/746573692 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb484004e10 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb4a0e27640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb4a0e27640 4 set_mon_vals no callback set 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.655+0000 7fb499d9b640 10 monclient: dump: 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 10 monclient: shutdown 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 1 -- 172.21.15.1:0/746573692 >> v1:172.21.15.1:6789/0 conn(0x7fb49c0a6ba0 legacy=0x7fb49c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 1 -- 172.21.15.1:0/746573692 shutdown_connections 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 1 -- 172.21.15.1:0/746573692 >> 172.21.15.1:0/746573692 conn(0x7fb49c0ac4b0 msgr2=0x7fb49c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 1 -- 172.21.15.1:0/746573692 shutdown_connections 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.656+0000 7fb4a0e27640 1 -- 172.21.15.1:0/746573692 wait complete. 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 10 monclient: monmap: 2024-08-19T10:03:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 1 librados: starting msgr at 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 1 librados: starting objecter 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 1 Processor -- start 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 1 -- start start 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 1 librados: setting wanted keys 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 1 librados: calling monclient init 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 10 monclient: init 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.657+0000 7fb4a0e27640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 10 monclient: _add_conns ranks=[6,3,5,1,0,7,2,8,4] 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 10 monclient(hunting): picked mon.noname-d con 0x7fb49c0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 10 monclient(hunting): picked mon.noname-g con 0x7fb49c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 10 monclient(hunting): picked mon.noname-h con 0x7fb49c13f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c13bdd0 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c13af60 con 0x7fb49c13f5c0 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c139a70 con 0x7fb49c0a67c0 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb4a0e27640 10 monclient(hunting): authenticate will time out at 1482.734120s 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb49a59c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fb49c0a67c0 0x7fb49c13c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:43856/0 (socket says 172.21.15.1:43856) 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.658+0000 7fb49a59c640 1 -- 172.21.15.1:0/1759174668 learned_addr learned my addr 172.21.15.1:0/1759174668 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb49a59c640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.1:6792/0 conn(0x7fb49c0a67c0 legacy=0x7fb49c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 client.?.objecter ms_handle_connect 0x7fb49c0a67c0 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500302969 0 0) 0x7fb49c139a70 con 0x7fb49c0a67c0 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): my global_id is 194217 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb493fff640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb49c0a6ba0 legacy=0x7fb49c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb47c004eb0 con 0x7fb49c0a67c0 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 client.?.objecter ms_handle_connect 0x7fb49c0a6ba0 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 client.?.objecter ms_handle_connect 0x7fb49c13f5c0 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb49ad9d640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6791/0 conn(0x7fb49c13f5c0 legacy=0x7fb49c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004471237 0 0) 0x7fb49c13bdd0 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): my global_id is 184440 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb49c139a70 con 0x7fb49c0a6ba0 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 30637274 0 0) 0x7fb47c004eb0 con 0x7fb49c0a67c0 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb49c13bdd0 con 0x7fb49c0a67c0 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3055697626 0 0) 0x7fb49c13af60 con 0x7fb49c13f5c0 2024-08-19T10:03:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): my global_id is 27707 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb47c004eb0 con 0x7fb49c13f5c0 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb48c003610 con 0x7fb49c0a67c0 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:03:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): dump: 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1149786618 0 0) 0x7fb49c13bdd0 con 0x7fb49c0a67c0 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6791/0 conn(0x7fb49c13f5c0 legacy=0x7fb49c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb49c0a6ba0 legacy=0x7fb49c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 monclient: found mon.d 2024-08-19T10:03:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c13b530 con 0x7fb49c0a67c0 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 5 monclient: authenticate success, global_id 194217 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 10 monclient: _renew_subs 2024-08-19T10:03:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb49c13e640 con 0x7fb49c0a67c0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 10 client.194217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 10 monclient: _renew_subs 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb49c13d010 con 0x7fb49c0a67c0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb4a0e27640 1 librados: init done 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb48c003c90 con 0x7fb49c0a67c0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb48c004200 con 0x7fb49c0a67c0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb492ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.660+0000 7fb49959a640 4 set_mon_vals no callback set 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.661+0000 7fb492ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.661+0000 7fb492ffd640 10 monclient: dump: 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.661+0000 7fb4a0e27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.661+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb4800065c0 con 0x7fb480002860 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.661+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb48c011a80 con 0x7fb49c0a67c0 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.662+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(489..489 src has 299..489) ==== 5888+0+0 (unknown 3091791536 0 0) 0x7fb48c04e460 con 0x7fb49c0a67c0 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.662+0000 7fb492ffd640 10 client.194217.objecter ms_dispatch 0x7fb49c000b90 osd_map(489..489 src has 299..489) 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.662+0000 7fb492ffd640 3 client.194217.objecter handle_osd_map got epochs [489,489] > 0 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.662+0000 7fb492ffd640 3 client.194217.objecter handle_osd_map decoding full epoch 489 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.662+0000 7fb492ffd640 20 client.194217.objecter dump_active .. 0 homeless 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.662+0000 7fb49ad9d640 10 monclient: get_auth_request con 0x7fb47c041230 auth_method 0 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb493fff640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb480002860 legacy=0x7fb480004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 client.194217.objecter ms_handle_connect 0x7fb480002860 2024-08-19T10:03:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 client.194217.objecter resend_mon_ops 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827067683 0 0) 0x7fb4800065c0 con 0x7fb480002860 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient(hunting): my global_id is 184449 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb49c13bdd0 con 0x7fb480002860 2024-08-19T10:03:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4247621334 0 0) 0x7fb49c13bdd0 con 0x7fb480002860 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4800065c0 con 0x7fb480002860 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb484001d30 con 0x7fb480002860 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 monclient: dump: 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 10 client.194217.objecter ms_handle_connect 0x7fb47c041230 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1617360438 0 0) 0x7fb4800065c0 con 0x7fb480002860 2024-08-19T10:03:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb48000a160 con 0x7fb480002860 2024-08-19T10:03:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.664+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb484004b90 con 0x7fb480002860 2024-08-19T10:03:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.665+0000 7fb492ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.665+0000 7fb492ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.665+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb480002860 legacy=0x7fb480004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.665+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb480002860 legacy=0x7fb480004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.671+0000 7fb4a0e27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.671+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb480009e70 con 0x7fb480009130 2024-08-19T10:03:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb493fff640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb480009130 legacy=0x7fb4800095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 client.194217.objecter ms_handle_connect 0x7fb480009130 2024-08-19T10:03:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 client.194217.objecter resend_mon_ops 2024-08-19T10:03:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667559443 0 0) 0x7fb480009e70 con 0x7fb480009130 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient(hunting): my global_id is 184458 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4800065c0 con 0x7fb480009130 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2986676295 0 0) 0x7fb4800065c0 con 0x7fb480009130 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb480009e70 con 0x7fb480009130 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb484005510 con 0x7fb480009130 2024-08-19T10:03:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.673+0000 7fb492ffd640 10 monclient: dump: 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 734082070 0 0) 0x7fb480009e70 con 0x7fb480009130 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb48000c410 con 0x7fb480009130 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1190073411) 0x7fb484008600 con 0x7fb480009130 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb480009130 legacy=0x7fb4800095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb492ffd640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6790/0 conn(0x7fb480009130 legacy=0x7fb4800095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.701 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 143928, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:46.702 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:46.703 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.704 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.705 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.706 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.711 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:46.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.714 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.715 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.716 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.717 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.720 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:46.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:46.724 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:46.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 10 librados: watch_flush enter 2024-08-19T10:03:46.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 10 librados: watch_flush exit 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 20 client.194217.objecter shutdown clearing up homeless session... 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 10 client.194217.objecter successfully canceled tick 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb47c041230 legacy=0x7fb47c0436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 10 monclient: shutdown 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 >> v1:172.21.15.1:6792/0 conn(0x7fb49c0a67c0 legacy=0x7fb49c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb49ad9d640 1 -- 172.21.15.1:0/1759174668 reap_dead start 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb49ad9d640 1 -- 172.21.15.1:0/1759174668 reap_dead start 2024-08-19T10:03:46.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 shutdown_connections 2024-08-19T10:03:46.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 >> 172.21.15.1:0/1759174668 conn(0x7fb49c0ac4b0 msgr2=0x7fb49c141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 shutdown_connections 2024-08-19T10:03:46.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 -- 172.21.15.1:0/1759174668 wait complete. 2024-08-19T10:03:46.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.674+0000 7fb4a0e27640 1 librados: shutdown 2024-08-19T10:03:46.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:46.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:03:46.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866384695}) 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2925836742}) 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.700+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=115939818}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3011080001}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1372795980}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=77531088}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=494931728}) 2024-08-19T10:03:46.734 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.734 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391269172}) 2024-08-19T10:03:46.734 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.701+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=161907591}) 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113542885}) 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480503311}) 2024-08-19T10:03:46.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4223920801}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2810586114}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2177201513}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3856155343}) 2024-08-19T10:03:46.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=339393039}) 2024-08-19T10:03:46.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758397787}) 2024-08-19T10:03:46.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.735+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1731675115}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744711339}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297761611}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4219031352}) 2024-08-19T10:03:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=772188224}) 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2061294067}) 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3538953567}) 2024-08-19T10:03:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3827868065}) 2024-08-19T10:03:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049576747}) 2024-08-19T10:03:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.762+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=554694672}) 2024-08-19T10:03:46.784 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.784 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2678026936}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666719155}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4098954202}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1351594253}) 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.785 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2511195539}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1795801580}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=363952838}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316324884}) 2024-08-19T10:03:46.786 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:46.782+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3092927441}) 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.851+0000 7fcbe12af640 10 monclient: get_monmap_and_config 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.851+0000 7fcbe12af640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.851+0000 7fcbe12af640 10 monclient: monmap: 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:46.852008+0000 2024-08-19T10:03:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:46.852008+0000 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.853 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.851+0000 7fcbe12af640 1 Processor -- start 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.851+0000 7fcbe12af640 1 -- start start 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.851+0000 7fcbe12af640 10 monclient: init 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient: _add_conns ranks=[8,7,6,2,1,0,3,4,5] 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient(hunting): picked mon.noname-e con 0x7fcbdc0a6830 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient(hunting): picked mon.noname-i con 0x7fcbdc0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient(hunting): picked mon.noname-d con 0x7fcbdc0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc0a3c70 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc0a33d0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc0a36b0 con 0x7fcbdc0a6830 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbe12af640 10 monclient(hunting): authenticate will time out at 1482.928344s 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbdaffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fcbdc0a6830 0x7fcbdc0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48974/0 (socket says 172.21.15.1:48974) 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.852+0000 7fcbdaffd640 1 -- 172.21.15.1:0/1512122827 learned_addr learned my addr 172.21.15.1:0/1512122827 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbdaffd640 1 -- 172.21.15.1:0/1512122827 >> v1:172.21.15.1:6793/0 conn(0x7fcbdc0a6830 legacy=0x7fcbdc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/1512122827 >> v1:172.21.15.1:6792/0 conn(0x7fcbdc0a46a0 legacy=0x7fcbdc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbda7fc640 1 -- 172.21.15.1:0/1512122827 >> v1:172.21.15.82:6792/0 conn(0x7fcbdc0694b0 legacy=0x7fcbdc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 969562429 0 0) 0x7fcbdc0a36b0 con 0x7fcbdc0a6830 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): my global_id is 184508 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbc8003520 con 0x7fcbdc0a6830 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869679168 0 0) 0x7fcbdc0a3c70 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): my global_id is 194226 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbdc0a36b0 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800920752 0 0) 0x7fcbdc0a33d0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 10 monclient(hunting): my global_id is 145186 2024-08-19T10:03:46.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.854+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbdc0a3c70 con 0x7fcbdc0694b0 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3172818152 0 0) 0x7fcbc8003520 con 0x7fcbdc0a6830 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 >> v1:172.21.15.82:6792/0 conn(0x7fcbdc0694b0 legacy=0x7fcbdc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 >> v1:172.21.15.1:6792/0 conn(0x7fcbdc0a46a0 legacy=0x7fcbdc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 monclient: found mon.noname-e 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc06dbb0 con 0x7fcbdc0a6830 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 5 monclient: authenticate success, global_id 184508 2024-08-19T10:03:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbd4001e60 con 0x7fcbdc0a6830 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: dump: 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbd4004c20 con 0x7fcbdc0a6830 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 1 -- 172.21.15.1:0/1512122827 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbd4005190 con 0x7fcbdc0a6830 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 4 set_mon_vals no callback set 2024-08-19T10:03:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbd9ffb640 10 monclient: dump: 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 10 monclient: shutdown 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 1 -- 172.21.15.1:0/1512122827 >> v1:172.21.15.1:6793/0 conn(0x7fcbdc0a6830 legacy=0x7fcbdc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 1 -- 172.21.15.1:0/1512122827 shutdown_connections 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 1 -- 172.21.15.1:0/1512122827 >> 172.21.15.1:0/1512122827 conn(0x7fcbdc0a9020 msgr2=0x7fcbdc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 1 -- 172.21.15.1:0/1512122827 shutdown_connections 2024-08-19T10:03:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.855+0000 7fcbe12af640 1 -- 172.21.15.1:0/1512122827 wait complete. 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 10 monclient: build_initial_monmap 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 10 monclient: monmap: 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:46.868 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 1 librados: starting msgr at 2024-08-19T10:03:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 1 librados: starting objecter 2024-08-19T10:03:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 1 Processor -- start 2024-08-19T10:03:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 1 -- start start 2024-08-19T10:03:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 1 librados: setting wanted keys 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 1 librados: calling monclient init 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.856+0000 7fcbe12af640 10 monclient: init 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient: _add_conns ranks=[3,5,2,8,4,1,0,6,7] 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient(hunting): picked mon.noname-g con 0x7fcbdc0694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient(hunting): picked mon.noname-h con 0x7fcbdc0a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc139520 con 0x7fcbdc0a6830 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc13b5a0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc13b880 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbe12af640 10 monclient(hunting): authenticate will time out at 1482.933327s 2024-08-19T10:03:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbdb7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 0x7fcbdc1d5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42330/0 (socket says 172.21.15.1:42330) 2024-08-19T10:03:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.857+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/2920937591 learned_addr learned my addr 172.21.15.1:0/2920937591 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:46.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 legacy=0x7fcbdc1d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 client.?.objecter ms_handle_connect 0x7fcbdc0a6830 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 client.?.objecter ms_handle_connect 0x7fcbdc0694b0 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbdaffd640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6790/0 conn(0x7fcbdc0694b0 legacy=0x7fcbdc1cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 client.?.objecter ms_handle_connect 0x7fcbdc0a46a0 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbdc0a46a0 legacy=0x7fcbdc141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130077021 0 0) 0x7fcbdc139520 con 0x7fcbdc0a6830 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): my global_id is 184412 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbbc004eb0 con 0x7fcbdc0a6830 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2160978141 0 0) 0x7fcbdc13b5a0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): my global_id is 184467 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbdc139520 con 0x7fcbdc0694b0 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888363742 0 0) 0x7fcbdc13b880 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 10 monclient(hunting): my global_id is 27716 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.859+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbdc13b5a0 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2668134000 0 0) 0x7fcbbc004eb0 con 0x7fcbdc0a6830 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbdc13b880 con 0x7fcbdc0a6830 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3870346523 0 0) 0x7fcbdc139520 con 0x7fcbdc0694b0 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbbc004eb0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbcc003610 con 0x7fcbdc0a6830 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 legacy=0x7fcbdc1d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 legacy=0x7fcbdc1d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbdb7fe640 1 --1- 172.21.15.1:0/2920937591 >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 0x7fcbdc1d5c50 :-1 s=OPENED pgs=3430 cs=1 l=1).handle_message read tag failed 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbdb7fe640 1 --1- 172.21.15.1:0/2920937591 >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 0x7fcbdc1d5c50 :-1 s=OPENED pgs=3430 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:03:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): dump: 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6790/0 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbd40020b0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): dump: 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 990685674 0 0) 0x7fcbdc13b5a0 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbdc139520 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbc4001350 con 0x7fcbdc0a46a0 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient(hunting): dump: 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4238430029 0 0) 0x7fcbbc004eb0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbdc0a46a0 legacy=0x7fcbdc141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.1:6790/0 conn(0x7fcbdc0a6830 legacy=0x7fcbdc1d5c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 monclient: found mon.g 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:03:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc142a20 con 0x7fcbdc0694b0 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: _finish_auth 0 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 5 monclient: authenticate success, global_id 184467 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 10 monclient: _renew_subs 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc0a9f30 con 0x7fcbdc0694b0 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 10 client.184467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 10 monclient: _renew_subs 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.82:6790/0 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcbdc1426f0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbe12af640 1 librados: init done 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbd4004c20 con 0x7fcbdc0694b0 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbd4001900 con 0x7fcbdc0694b0 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd97fa640 4 set_mon_vals no callback set 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.860+0000 7fcbd37fe640 10 monclient: dump: 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbe12af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcbc00065c0 con 0x7fcbc0002860 2024-08-19T10:03:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcbd40119c0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.3 v1:172.21.15.82:6790/0 8 ==== osd_map(489..489 src has 299..489) ==== 5888+0+0 (unknown 3091791536 0 0) 0x7fcbd404e6e0 con 0x7fcbdc0694b0 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbd37fe640 10 client.184467.objecter ms_dispatch 0x7fcbdc000b90 osd_map(489..489 src has 299..489) 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbd37fe640 3 client.184467.objecter handle_osd_map got epochs [489,489] > 0 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbd37fe640 3 client.184467.objecter handle_osd_map decoding full epoch 489 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbdb7fe640 10 monclient: get_auth_request con 0x7fcbbc0419c0 auth_method 0 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.861+0000 7fcbd37fe640 20 client.184467.objecter dump_active .. 0 homeless 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbc0002860 legacy=0x7fcbc0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 client.184467.objecter ms_handle_connect 0x7fcbc0002860 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 client.184467.objecter resend_mon_ops 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827342000 0 0) 0x7fcbc00065c0 con 0x7fcbc0002860 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient(hunting): my global_id is 27725 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbbc004eb0 con 0x7fcbc0002860 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 396567477 0 0) 0x7fcbbc004eb0 con 0x7fcbc0002860 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbc00065c0 con 0x7fcbc0002860 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbc4003c00 con 0x7fcbc0002860 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 monclient: dump: 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.863+0000 7fcbd37fe640 10 client.184467.objecter ms_handle_connect 0x7fcbbc0419c0 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3699878782 0 0) 0x7fcbc00065c0 con 0x7fcbc0002860 2024-08-19T10:03:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbc000a160 con 0x7fcbc0002860 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcbc4003ee0 con 0x7fcbc0002860 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbc0002860 legacy=0x7fcbc0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.864+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbc0002860 legacy=0x7fcbc0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.870+0000 7fcbe12af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.870+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcbc0009e70 con 0x7fcbc0009130 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.872+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbc0009130 legacy=0x7fcbc00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.872+0000 7fcbd37fe640 10 client.184467.objecter ms_handle_connect 0x7fcbc0009130 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.872+0000 7fcbd37fe640 10 client.184467.objecter resend_mon_ops 2024-08-19T10:03:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416066143 0 0) 0x7fcbc0009e70 con 0x7fcbc0009130 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient(hunting): my global_id is 27734 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbc00065c0 con 0x7fcbc0009130 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2483518512 0 0) 0x7fcbc00065c0 con 0x7fcbc0009130 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbc0009e70 con 0x7fcbc0009130 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbc4004300 con 0x7fcbc0009130 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient: dump: 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:46.894 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2503784102 0 0) 0x7fcbc0009e70 con 0x7fcbc0009130 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbc000c410 con 0x7fcbc0009130 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2348648624) 0x7fcbc4007b80 con 0x7fcbc0009130 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbc0009130 legacy=0x7fcbc00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.873+0000 7fcbd37fe640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6791/0 conn(0x7fcbc0009130 legacy=0x7fcbc00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:46.895 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 833139, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:46.896 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.897 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.898 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:46.899 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:46.900 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:46.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.902 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.904 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.906 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.908 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:46.911 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:46.912 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:46.913 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:46.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:46.916 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 10 librados: watch_flush enter 2024-08-19T10:03:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 10 librados: watch_flush exit 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 20 client.184467.objecter shutdown clearing up homeless session... 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 10 client.184467.objecter successfully canceled tick 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcbbc0419c0 legacy=0x7fcbbc043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 10 monclient: shutdown 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 >> v1:172.21.15.82:6790/0 conn(0x7fcbdc0694b0 legacy=0x7fcbdc1cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/2920937591 reap_dead start 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbdb7fe640 1 -- 172.21.15.1:0/2920937591 reap_dead start 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 shutdown_connections 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 >> 172.21.15.1:0/2920937591 conn(0x7fcbdc0a9020 msgr2=0x7fcbdc0a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 shutdown_connections 2024-08-19T10:03:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 -- 172.21.15.1:0/2920937591 wait complete. 2024-08-19T10:03:46.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:46.874+0000 7fcbe12af640 1 librados: shutdown 2024-08-19T10:03:46.919 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:46.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:03:46.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:03:47.053 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854160900}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691689452}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2713882596}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=8696533}) 2024-08-19T10:03:47.054 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=426552287}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=692016856}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1151233452}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149183891}) 2024-08-19T10:03:47.055 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:03:47.050+0000 7f16c7dce640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3391110454}) 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.057+0000 7f72a4ba0640 10 monclient: get_monmap_and_config 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.057+0000 7f72a4ba0640 10 monclient: build_initial_monmap 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.058+0000 7f72a4ba0640 10 monclient: monmap: 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:47.058922+0000 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:47.058922+0000 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:47.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.058+0000 7f72a4ba0640 1 Processor -- start 2024-08-19T10:03:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.058+0000 7f72a4ba0640 1 -- start start 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.058+0000 7f72a4ba0640 10 monclient: init 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient: _add_conns ranks=[2,7,8,5,0,6,4,3,1] 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient(hunting): picked mon.noname-b con 0x7f72a00a87e0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient(hunting): picked mon.noname-i con 0x7f72a0070f80 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient(hunting): picked mon.noname-e con 0x7f72a00a92d0 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a0070d40 con 0x7f72a00a87e0 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a0072d40 con 0x7f72a0070f80 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a0072a60 con 0x7f72a00a92d0 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f72a4ba0640 10 monclient(hunting): authenticate will time out at 1483.135402s 2024-08-19T10:03:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f729f7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f72a00a87e0 0x7f72a00a8bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42340/0 (socket says 172.21.15.1:42340) 2024-08-19T10:03:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f729ffff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f72a00a92d0 0x7f72a013fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48980/0 (socket says 172.21.15.1:48980) 2024-08-19T10:03:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.059+0000 7f729f7fe640 1 -- 172.21.15.1:0/4062868401 learned_addr learned my addr 172.21.15.1:0/4062868401 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:47.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729f7fe640 1 -- 172.21.15.1:0/4062868401 >> v1:172.21.15.1:6790/0 conn(0x7f72a00a87e0 legacy=0x7f72a00a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729ffff640 1 -- 172.21.15.1:0/4062868401 >> v1:172.21.15.1:6793/0 conn(0x7f72a00a92d0 legacy=0x7f72a013fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f7296ffd640 1 -- 172.21.15.1:0/4062868401 >> v1:172.21.15.82:6792/0 conn(0x7f72a0070f80 legacy=0x7f72a013c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226318295 0 0) 0x7f72a0070d40 con 0x7f72a00a87e0 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): my global_id is 184421 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f728c003520 con 0x7f72a00a87e0 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647062176 0 0) 0x7f72a0072a60 con 0x7f72a00a92d0 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): my global_id is 184517 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a0070d40 con 0x7f72a00a92d0 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813388902 0 0) 0x7f72a0072d40 con 0x7f72a0070f80 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 10 monclient(hunting): my global_id is 145195 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.061+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a0072a60 con 0x7f72a0070f80 2024-08-19T10:03:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2016054573 0 0) 0x7f728c003520 con 0x7f72a00a87e0 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 >> v1:172.21.15.1:6793/0 conn(0x7f72a00a92d0 legacy=0x7f72a013fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 >> v1:172.21.15.82:6792/0 conn(0x7f72a0070f80 legacy=0x7f72a013c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: _finish_hunting 0 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 monclient: found mon.noname-b 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a0073640 con 0x7f72a00a87e0 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: _finish_auth 0 2024-08-19T10:03:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: _check_auth_tickets 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7290001ae0 con 0x7f72a00a87e0 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: dump: 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 5 monclient: authenticate success, global_id 184421 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72900048a0 con 0x7f72a00a87e0 2024-08-19T10:03:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 1 -- 172.21.15.1:0/4062868401 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7290004e10 con 0x7f72a00a87e0 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 4 set_mon_vals no callback set 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f729effd640 10 monclient: dump: 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.073 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 10 monclient: shutdown 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4062868401 >> v1:172.21.15.1:6790/0 conn(0x7f72a00a87e0 legacy=0x7f72a00a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4062868401 shutdown_connections 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4062868401 >> 172.21.15.1:0/4062868401 conn(0x7f72a00a9fc0 msgr2=0x7f72a00a63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4062868401 shutdown_connections 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.062+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4062868401 wait complete. 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 10 monclient: build_initial_monmap 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 10 monclient: monmap: 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:47.075 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 1 librados: starting msgr at 2024-08-19T10:03:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 1 librados: starting objecter 2024-08-19T10:03:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 1 Processor -- start 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 1 -- start start 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 1 librados: setting wanted keys 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 1 librados: calling monclient init 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.063+0000 7f72a4ba0640 10 monclient: init 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient: _add_conns ranks=[4,0,7,2,1,6,5,8,3] 2024-08-19T10:03:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient(hunting): picked mon.noname-c con 0x7f72a0070f80 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient(hunting): picked mon.noname-a con 0x7f72a00a87e0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient(hunting): picked mon.noname-i con 0x7f72a00a92d0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a00a6350 con 0x7f72a00a87e0 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a01293b0 con 0x7f72a0070f80 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a0129690 con 0x7f72a00a92d0 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f72a4ba0640 10 monclient(hunting): authenticate will time out at 1483.140322s 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f729f7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f72a0070f80 0x7f72a006cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40766/0 (socket says 172.21.15.1:40766) 2024-08-19T10:03:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.064+0000 7f729f7fe640 1 -- 172.21.15.1:0/4094840057 learned_addr learned my addr 172.21.15.1:0/4094840057 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729f7fe640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.1:6791/0 conn(0x7f72a0070f80 legacy=0x7f72a006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 client.?.objecter ms_handle_connect 0x7f72a0070f80 2024-08-19T10:03:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 client.?.objecter ms_handle_connect 0x7f72a00a87e0 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f7296ffd640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.1:6789/0 conn(0x7f72a00a87e0 legacy=0x7f72a006d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729ffff640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f72a00a92d0 legacy=0x7f72a01d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359439872 0 0) 0x7f72a01293b0 con 0x7f72a0070f80 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729ffff640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f72a00a92d0 legacy=0x7f72a01d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729ffff640 1 --1- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f72a00a92d0 0x7f72a01d6200 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): my global_id is 194260 2024-08-19T10:03:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729ffff640 1 --1- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f72a00a92d0 0x7f72a01d6200 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7280004eb0 con 0x7f72a0070f80 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6792/0 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338468684 0 0) 0x7f72a0072a60 con 0x7f72a00a87e0 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): my global_id is 174798 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a01293b0 con 0x7f72a00a87e0 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2683954664 0 0) 0x7f7280004eb0 con 0x7f72a0070f80 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72a0072a60 con 0x7f72a0070f80 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3608800657 0 0) 0x7f72a01293b0 con 0x7f72a00a87e0 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7280004eb0 con 0x7f72a00a87e0 2024-08-19T10:03:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7290004d90 con 0x7f72a0070f80 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.066+0000 7f729cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient(hunting): dump: 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7288003610 con 0x7f72a00a87e0 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient(hunting): dump: 2024-08-19T10:03:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.089 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 690494200 0 0) 0x7f72a0072a60 con 0x7f72a0070f80 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f72a00a92d0 legacy=0x7f72a01d6200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.1:6789/0 conn(0x7f72a00a87e0 legacy=0x7f72a006d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: _finish_hunting 0 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 monclient: found mon.c 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a00a3370 con 0x7f72a0070f80 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: _finish_auth 0 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: _check_auth_tickets 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 5 monclient: authenticate success, global_id 194260 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 10 monclient: _renew_subs 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72a01290d0 con 0x7f72a0070f80 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 10 client.194260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 10 monclient: _renew_subs 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f72a006db10 con 0x7f72a0070f80 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f72a4ba0640 1 librados: init done 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7290001b00 con 0x7f72a0070f80 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7290005740 con 0x7f72a0070f80 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729e7fc640 4 set_mon_vals no callback set 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.067+0000 7f729cff9640 10 monclient: dump: 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f72a4ba0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7290011ba0 con 0x7f72a0070f80 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f72840065c0 con 0x7f7284002860 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(489..489 src has 299..489) ==== 5888+0+0 (unknown 3091791536 0 0) 0x7f729004e400 con 0x7f72a0070f80 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729cff9640 10 client.194260.objecter ms_dispatch 0x7f72a0000b90 osd_map(489..489 src has 299..489) 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729cff9640 3 client.194260.objecter handle_osd_map got epochs [489,489] > 0 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729cff9640 3 client.194260.objecter handle_osd_map decoding full epoch 489 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729cff9640 20 client.194260.objecter dump_active .. 0 homeless 2024-08-19T10:03:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.068+0000 7f729ffff640 10 monclient: get_auth_request con 0x7f72800414f0 auth_method 0 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f7296ffd640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f7284002860 legacy=0x7f7284004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 client.194260.objecter ms_handle_connect 0x7f7284002860 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 client.194260.objecter resend_mon_ops 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 888139823 0 0) 0x7f72840065c0 con 0x7f7284002860 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient(hunting): my global_id is 145204 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a0072a60 con 0x7f7284002860 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 client.194260.objecter ms_handle_connect 0x7f72800414f0 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2759306015 0 0) 0x7f72a0072a60 con 0x7f7284002860 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72840065c0 con 0x7f7284002860 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7288003da0 con 0x7f7284002860 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 10 monclient: dump: 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2222797818 0 0) 0x7f72840065c0 con 0x7f7284002860 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.070+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f728400a160 con 0x7f7284002860 2024-08-19T10:03:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.071+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7288004080 con 0x7f7284002860 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.071+0000 7f729cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.071+0000 7f729cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.071+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f7284002860 legacy=0x7f7284004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.071+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f7284002860 legacy=0x7f7284004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.077+0000 7f72a4ba0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.077+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7284009e50 con 0x7f7284009110 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.079+0000 7f7296ffd640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f7284009110 legacy=0x7f72840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.079+0000 7f729cff9640 10 client.194260.objecter ms_handle_connect 0x7f7284009110 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.079+0000 7f729cff9640 10 client.194260.objecter resend_mon_ops 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 687003891 0 0) 0x7f7284009e50 con 0x7f7284009110 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient(hunting): my global_id is 145213 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72840065c0 con 0x7f7284009110 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2288767396 0 0) 0x7f72840065c0 con 0x7f7284009110 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7284009e50 con 0x7f7284009110 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7288005b00 con 0x7f7284009110 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient: dump: 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1546825102 0 0) 0x7f7284009e50 con 0x7f7284009110 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f728400c3a0 con 0x7f7284009110 2024-08-19T10:03:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 497072680) 0x7f7288006f30 con 0x7f7284009110 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f7284009110 legacy=0x7f72840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f729cff9640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6792/0 conn(0x7f7284009110 legacy=0x7f72840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-08-19T10:03:47.100 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 551460, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:03:47.101 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:47.102 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:47.103 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:03:47.104 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:03:47.105 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.107 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.110 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.111 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:03:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.113 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:03:47.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:03:47.116 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.117 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:47.118 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:03:47.119 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:03:47.120 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:03:47.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:03:47.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:03:47.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:03:47.120 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:03:47.120 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:03:47.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.080+0000 7f72a4ba0640 10 librados: watch_flush enter 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 10 librados: watch_flush exit 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 20 client.194260.objecter shutdown clearing up homeless session... 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 10 client.194260.objecter successfully canceled tick 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.82:6812/2549104415 conn(0x7f72800414f0 legacy=0x7f72800439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 10 monclient: shutdown 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 >> v1:172.21.15.1:6791/0 conn(0x7f72a0070f80 legacy=0x7f72a006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f729ffff640 1 -- 172.21.15.1:0/4094840057 reap_dead start 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f729ffff640 1 -- 172.21.15.1:0/4094840057 reap_dead start 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 shutdown_connections 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 >> 172.21.15.1:0/4094840057 conn(0x7f72a00a9fc0 msgr2=0x7f72a00a59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 shutdown_connections 2024-08-19T10:03:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 -- 172.21.15.1:0/4094840057 wait complete. 2024-08-19T10:03:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.081+0000 7f72a4ba0640 1 librados: shutdown 2024-08-19T10:03:47.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:03:47.123 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d', 'e', 'a', 'i'] 2024-08-19T10:03:47.123 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-19T10:03:47.123 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T10:03:47.124 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T10:03:47.124 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T10:03:47.124 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:03:47.134 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T10:03:47.134 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-08-19T10:03:47.134 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-08-19T10:03:47.135 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T10:03:47.135 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:03:47.187 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T10:03:47.187 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T10:03:47.187 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T10:03:47.187 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T10:03:47.187 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:03:47.240 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T10:03:47.240 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-08-19T10:03:47.240 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-08-19T10:03:47.240 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-08-19T10:03:47.241 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:03:47.255 INFO:tasks.ceph.mon.i:Stopped 2024-08-19T10:03:47.255 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-08-19T10:03:47.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.377+0000 7f47b2415640 10 monclient: get_monmap_and_config 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.377+0000 7f47b2415640 10 monclient: build_initial_monmap 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.378+0000 7f47b2415640 10 monclient: monmap: 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:47.378911+0000 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:47.378911+0000 2024-08-19T10:03:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:47.380 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.378+0000 7f47b2415640 1 Processor -- start 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.378+0000 7f47b2415640 1 -- start start 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.378+0000 7f47b2415640 10 monclient: init 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient: _add_conns ranks=[3,6,2,0,1,8,4,7,5] 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient(hunting): picked mon.noname-g con 0x7f47ac0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient(hunting): picked mon.noname-d con 0x7f47ac0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient(hunting): picked mon.noname-b con 0x7f47ac13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac0a3b40 con 0x7f47ac13f5c0 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac0a8130 con 0x7f47ac0a67c0 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac06ede0 con 0x7f47ac0a6ba0 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b2415640 10 monclient(hunting): authenticate will time out at 1483.455383s 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47a3fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b09b2640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f47ac13f5c0 0x7f47ac1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42352/0 (socket says 172.21.15.1:42352) 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.379+0000 7f47b09b2640 1 -- 172.21.15.1:0/1611350422 learned_addr learned my addr 172.21.15.1:0/1611350422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47b09b2640 1 -- 172.21.15.1:0/1611350422 >> v1:172.21.15.1:6790/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47abfff640 1 -- 172.21.15.1:0/1611350422 >> v1:172.21.15.82:6790/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773831801 0 0) 0x7f47ac0a3b40 con 0x7f47ac13f5c0 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47ab7fe640 10 monclient(hunting): my global_id is 184430 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.381+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4798003520 con 0x7f47ac13f5c0 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303428467 0 0) 0x7f47ac0a8130 con 0x7f47ac0a67c0 2024-08-19T10:03:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient(hunting): my global_id is 184476 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47ac0a3b40 con 0x7f47ac0a67c0 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1008412004 0 0) 0x7f4798003520 con 0x7f47ac13f5c0 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac0a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 >> v1:172.21.15.82:6790/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 monclient: found mon.noname-b 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac0a92a0 con 0x7f47ac13f5c0 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: _finish_auth 0 2024-08-19T10:03:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a4001f20 con 0x7f47ac13f5c0 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 5 monclient: authenticate success, global_id 184430 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: dump: 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47a4004c20 con 0x7f47ac13f5c0 2024-08-19T10:03:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 1 -- 172.21.15.1:0/1611350422 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a4005210 con 0x7f47ac13f5c0 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 10 monclient: get_monmap_and_config success 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 4 set_mon_vals no callback set 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47ab7fe640 10 monclient: dump: 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 10 monclient: shutdown 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 1 -- 172.21.15.1:0/1611350422 >> v1:172.21.15.1:6790/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 1 -- 172.21.15.1:0/1611350422 shutdown_connections 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 1 -- 172.21.15.1:0/1611350422 >> 172.21.15.1:0/1611350422 conn(0x7f47ac0ac4b0 msgr2=0x7f47ac0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 1 -- 172.21.15.1:0/1611350422 shutdown_connections 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.382+0000 7f47b2415640 1 -- 172.21.15.1:0/1611350422 wait complete. 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 10 monclient: build_initial_monmap 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 10 monclient: monmap: 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 1 librados: starting msgr at 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 1 librados: starting objecter 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 1 Processor -- start 2024-08-19T10:03:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 1 -- start start 2024-08-19T10:03:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 1 librados: setting wanted keys 2024-08-19T10:03:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 1 librados: calling monclient init 2024-08-19T10:03:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.383+0000 7f47b2415640 10 monclient: init 2024-08-19T10:03:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient: _add_conns ranks=[0,6,7,3,1,2,5,8,4] 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient(hunting): picked mon.noname-a con 0x7f47ac0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient(hunting): picked mon.noname-d con 0x7f47ac0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient(hunting): picked mon.noname-i con 0x7f47ac13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47abfff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac139c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac13c2f0 con 0x7f47ac0a67c0 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47a3fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac13a310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac13d820 con 0x7f47ac0a6ba0 2024-08-19T10:03:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac13aa20 con 0x7f47ac13f5c0 2024-08-19T10:03:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b2415640 10 monclient(hunting): authenticate will time out at 1483.460346s 2024-08-19T10:03:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.384+0000 7f47b09b2640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.585+0000 7f47a3fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac13a310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.585+0000 7f47abfff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac139c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:03:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.585+0000 7f47b09b2640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.986+0000 7f47a3fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac13a310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.986+0000 7f47abfff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac139c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:03:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:47.986+0000 7f47b09b2640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:48.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:48.787+0000 7f47abfff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac139c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:03:48.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:48.787+0000 7f47a3fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac13a310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:48.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:48.787+0000 7f47b09b2640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): tick 2024-08-19T10:03:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:03:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 monclient(hunting): continuing hunt 2024-08-19T10:03:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:03:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f47ac13f5c0 legacy=0x7f47ac1d6250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:03:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f47ac0a6ba0 legacy=0x7f47ac13a310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f47ac0a67c0 legacy=0x7f47ac139c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient: _add_conns ranks=[4,3,1,2,5,8] 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4790003780 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): picked mon.noname-g con 0x7f4790007410 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): picked mon.noname-f con 0x7f479000b1a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac13b4c0 con 0x7f479000b1a0 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac13c2f0 con 0x7f4790007410 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac13d820 con 0x7f4790003780 2024-08-19T10:03:50.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47a37fe640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:50.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47abfff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4790003780 0x7f4790005bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40780/0 (socket says 172.21.15.1:40780) 2024-08-19T10:03:50.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.384+0000 7f47abfff640 1 -- 172.21.15.1:0/3114566253 learned_addr learned my addr 172.21.15.1:0/3114566253 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.386+0000 7f47abfff640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.1:6791/0 conn(0x7f4790003780 legacy=0x7f4790005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.386+0000 7f47a97fa640 10 client.?.objecter ms_handle_connect 0x7f4790003780 2024-08-19T10:03:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.386+0000 7f47a97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070011670 0 0) 0x7f47ac13d820 con 0x7f4790003780 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): my global_id is 194269 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f478c004eb0 con 0x7f4790003780 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 client.?.objecter ms_handle_connect 0x7f479000b1a0 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47b09b2640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.82:6789/0 conn(0x7f479000b1a0 legacy=0x7f479000d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 client.?.objecter ms_handle_connect 0x7f4790007410 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:03:50.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a3fff640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.82:6790/0 conn(0x7f4790007410 legacy=0x7f47900098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2222168842 0 0) 0x7f47ac13b4c0 con 0x7f479000b1a0 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): my global_id is 174682 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47ac13d820 con 0x7f479000b1a0 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952728864 0 0) 0x7f47ac13c2f0 con 0x7f4790007410 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _negotiate 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): my global_id is 184485 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47ac13b4c0 con 0x7f4790007410 2024-08-19T10:03:50.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2150793666 0 0) 0x7f478c004eb0 con 0x7f4790003780 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47ac13c2f0 con 0x7f4790003780 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f479c003610 con 0x7f4790003780 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): dump: 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:50.397 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:50.398 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 704499136 0 0) 0x7f47ac13d820 con 0x7f479000b1a0 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f478c004eb0 con 0x7f479000b1a0 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f47a4005210 con 0x7f479000b1a0 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:03:50.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient(hunting): dump: 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:50.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1242426199 0 0) 0x7f47ac13c2f0 con 0x7f4790003780 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.82:6790/0 conn(0x7f4790007410 legacy=0x7f47900098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.82:6789/0 conn(0x7f479000b1a0 legacy=0x7f479000d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient: _finish_hunting 0 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 monclient: found mon.c 2024-08-19T10:03:50.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47b09b2640 1 -- 172.21.15.1:0/3114566253 reap_dead start 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac13aa20 con 0x7f4790003780 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient: _finish_auth 0 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 10 monclient: _check_auth_tickets 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47b2415640 5 monclient: authenticate success, global_id 194269 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47b2415640 10 monclient: _renew_subs 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47b2415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.387+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47ac13d540 con 0x7f4790003780 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47b2415640 10 client.194269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:03:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47b2415640 10 monclient: _renew_subs 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47b2415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f47ac13eed0 con 0x7f4790003780 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47b2415640 1 librados: init done 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f479c003b10 con 0x7f4790003780 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47a97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f479c004080 con 0x7f4790003780 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47aaffd640 4 set_mon_vals no callback set 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.388+0000 7f47a97fa640 10 monclient: dump: 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:50.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f479c004220 con 0x7f4790003780 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47b2415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47b2415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47b2415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4784005080 con 0x7f4790003780 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(489..489 src has 299..489) ==== 5888+0+0 (unknown 3091791536 0 0) 0x7f479c04e340 con 0x7f4790003780 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a97fa640 10 client.194269.objecter ms_dispatch 0x7f47ac000b90 osd_map(489..489 src has 299..489) 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a97fa640 3 client.194269.objecter handle_osd_map got epochs [489,489] > 0 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a97fa640 3 client.194269.objecter handle_osd_map decoding full epoch 489 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a3fff640 10 monclient: get_auth_request con 0x7f478c0412d0 auth_method 0 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.389+0000 7f47a97fa640 20 client.194269.objecter dump_active .. 0 homeless 2024-08-19T10:03:50.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.392+0000 7f47a97fa640 10 client.194269.objecter ms_handle_connect 0x7f478c0412d0 2024-08-19T10:03:50.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.394+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f479c0175b0 con 0x7f4790003780 2024-08-19T10:03:50.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.394+0000 7f47a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:03:50.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.394+0000 7f47a97fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:03:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47b2415640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:03:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47b2415640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47b2415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:03:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4784005370 con 0x7f4790003780 2024-08-19T10:03:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47a97fa640 1 -- 172.21.15.1:0/3114566253 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2704 (unknown 254746657 0 1377298637) 0x7f479c016eb0 con 0x7f4790003780 2024-08-19T10:03:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:03:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.652+0000 7f47a97fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:03:50.654 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":118,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 10 librados: watch_flush enter 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 10 librados: watch_flush exit 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 20 client.194269.objecter shutdown clearing up homeless session... 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 10 client.194269.objecter successfully canceled tick 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.82:6812/2549104415 conn(0x7f478c0412d0 legacy=0x7f478c043790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 10 monclient: shutdown 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 >> v1:172.21.15.1:6791/0 conn(0x7f4790003780 legacy=0x7f4790005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 shutdown_connections 2024-08-19T10:03:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 >> 172.21.15.1:0/3114566253 conn(0x7f47ac0ac4b0 msgr2=0x7f47ac141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:03:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 shutdown_connections 2024-08-19T10:03:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 -- 172.21.15.1:0/3114566253 wait complete. 2024-08-19T10:03:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:50.654+0000 7f47b2415640 1 librados: shutdown 2024-08-19T10:03:53.667 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:03:53.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.777+0000 7f5d20f7f640 10 monclient: get_monmap_and_config 2024-08-19T10:03:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.777+0000 7f5d20f7f640 10 monclient: build_initial_monmap 2024-08-19T10:03:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.777+0000 7f5d20f7f640 10 monclient: monmap: 2024-08-19T10:03:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:03:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:03:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:03:53.778106+0000 2024-08-19T10:03:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:03:53.778106+0000 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:03:53.780 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:03:53.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.777+0000 7f5d20f7f640 1 Processor -- start 2024-08-19T10:03:53.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.777+0000 7f5d20f7f640 1 -- start start 2024-08-19T10:03:53.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.777+0000 7f5d20f7f640 10 monclient: init 2024-08-19T10:03:53.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient: _reopen_session rank -1 2024-08-19T10:03:53.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient: _add_conns ranks=[6,5,2,1,3,0,7,4,8] 2024-08-19T10:03:53.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient(hunting): picked mon.noname-d con 0x7f5d1c0a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:03:53.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient(hunting): picked mon.noname-h con 0x7f5d1c0694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:03:53.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1c0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:03:53.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c0a3c70 con 0x7f5d1c0a46a0 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c0a33d0 con 0x7f5d1c0694b0 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c0a36b0 con 0x7f5d1c0a6830 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d20f7f640 10 monclient(hunting): authenticate will time out at 1489.854569s 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d1a59c640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f5d1c0a6830 legacy=0x7f5d1c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d1ad9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f5d1c0a46a0 0x7f5d1c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42366/0 (socket says 172.21.15.1:42366) 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.778+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 learned_addr learned my addr 172.21.15.1:0/3571387725 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.780+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6790/0 conn(0x7f5d1c0a46a0 legacy=0x7f5d1c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:53.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.781+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6791/0 conn(0x7f5d1c0694b0 legacy=0x7f5d1c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:53.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:53.980+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6792/0 conn(0x7f5d1c0a6830 legacy=0x7f5d1c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:54.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:54.381+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6792/0 conn(0x7f5d1c0a6830 legacy=0x7f5d1c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:55.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:55.182+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6792/0 conn(0x7f5d1c0a6830 legacy=0x7f5d1c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): tick 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 monclient(hunting): continuing hunt 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6792/0 conn(0x7f5d1c0a6830 legacy=0x7f5d1c0a6c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6791/0 conn(0x7f5d1c0694b0 legacy=0x7f5d1c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:56.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6790/0 conn(0x7f5d1c0a46a0 legacy=0x7f5d1c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient: _add_conns ranks=[1,7,3,8,4,0] 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-f con 0x7f5cfc0038c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-i con 0x7f5cfc0074b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-g con 0x7f5cfc00b220 addr v1:172.21.15.82:6790/0 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c06dbb0 con 0x7f5cfc0038c0 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c0a36b0 con 0x7f5cfc00b220 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc011100 con 0x7f5cfc0074b0 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.778+0000 7f5d0bfff640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:56.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.779+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6792/0 conn(0x7f5cfc0074b0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:56.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.781+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6789/0 conn(0x7f5cfc0038c0 legacy=0x7f5cfc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:56.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.781+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6790/0 conn(0x7f5cfc00b220 legacy=0x7f5cfc00d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:56.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:56.980+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6792/0 conn(0x7f5cfc0074b0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:57.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:57.381+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6792/0 conn(0x7f5cfc0074b0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:58.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:58.182+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6792/0 conn(0x7f5cfc0074b0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:03:59.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): tick 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 monclient(hunting): continuing hunt 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6792/0 conn(0x7f5cfc0074b0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6790/0 conn(0x7f5cfc00b220 legacy=0x7f5cfc00d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6789/0 conn(0x7f5cfc0038c0 legacy=0x7f5cfc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient: _add_conns ranks=[8,4,0] 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 reap_dead start 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-e con 0x7f5cfc010210 addr v1:172.21.15.1:6793/0 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f5cfc00a120 addr v1:172.21.15.1:6791/0 2024-08-19T10:03:59.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f5cfc00a500 addr v1:172.21.15.1:6789/0 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 reap_dead start 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc012400 con 0x7f5cfc00a500 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc011100 con 0x7f5cfc00a120 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6793/0 conn(0x7f5cfc010210 legacy=0x7f5cfc0105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc00a9d0 con 0x7f5cfc010210 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d0bfff640 10 monclient(hunting): _renew_subs 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.779+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00a500 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:03:59.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.781+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6791/0 conn(0x7f5cfc00a120 legacy=0x7f5cfc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:03:59.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.980+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00a500 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:03:59.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.980+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6793/0 conn(0x7f5cfc010210 legacy=0x7f5cfc0105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:03:59.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:03:59.980+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:04:00.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:00.381+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6793/0 conn(0x7f5cfc010210 legacy=0x7f5cfc0105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:04:00.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:00.381+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00a500 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:04:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.182+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6793/0 conn(0x7f5cfc010210 legacy=0x7f5cfc0105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:04:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.182+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00a500 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:04:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281209637 0 0) 0x7f5cfc011100 con 0x7f5cfc00a120 2024-08-19T10:04:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 10 monclient(hunting): _negotiate 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 10 monclient(hunting): my global_id is 194278 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d04003520 con 0x7f5cfc00a120 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6791/0 conn(0x7f5cfc00a120 legacy=0x7f5cfc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6791/0 conn(0x7f5cfc00a120 legacy=0x7f5cfc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d19d9b640 1 --1- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6791/0 conn(0x7f5cfc00a120 0x7f5cfc005d10 :-1 s=OPENED pgs=3697 cs=1 l=1).handle_message read tag failed 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d19d9b640 1 --1- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6791/0 conn(0x7f5cfc00a120 0x7f5cfc005d10 :-1 s=OPENED pgs=3697 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:04:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:01.788+0000 7f5d1959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6791/0 2024-08-19T10:04:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): tick 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 monclient(hunting): continuing hunt 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6793/0 conn(0x7f5cfc010210 legacy=0x7f5cfc0105f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6791/0 conn(0x7f5cfc00a120 legacy=0x7f5cfc005d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00a500 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient: _add_conns ranks=[0,1,3,4,7,5,8,6,2] 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f5cfc00f000 addr v1:172.21.15.1:6789/0 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-f con 0x7f5cfc00f3e0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): picked mon.noname-g con 0x7f5cfc00f7c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:04:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc00ac40 con 0x7f5cfc00f000 2024-08-19T10:04:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc012400 con 0x7f5cfc00f3e0 2024-08-19T10:04:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc00a9d0 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d0bfff640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.779+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00f000 legacy=0x7f5cfc00d6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:04:02.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d19d9b640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6789/0 conn(0x7f5cfc00f3e0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6790/0 conn(0x7f5cfc00f7c0 legacy=0x7f5cfc017760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 395567616 0 0) 0x7f5cfc012400 con 0x7f5cfc00f3e0 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): _negotiate 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): my global_id is 174691 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cfc011100 con 0x7f5cfc00f3e0 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205620914 0 0) 0x7f5cfc00a9d0 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): _negotiate 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient(hunting): my global_id is 184494 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cfc012400 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1824100155 0 0) 0x7f5cfc012400 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6789/0 conn(0x7f5cfc00f3e0 legacy=0x7f5cfc009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.1:6789/0 conn(0x7f5cfc00f000 legacy=0x7f5cfc00d6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: _finish_hunting 0 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 monclient: found mon.noname-g 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc00fc70 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: _finish_auth 0 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: _check_auth_tickets 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5d14001ce0 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: dump: 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d20f7f640 5 monclient: authenticate success, global_id 184494 2024-08-19T10:04:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d20f7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d14001e80 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 1 -- 172.21.15.1:0/3571387725 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5d14004f60 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d1959a640 10 monclient: dump: 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d20f7f640 10 monclient: get_monmap_and_config success 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.782+0000 7f5d20f7f640 4 set_mon_vals no callback set 2024-08-19T10:04:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 10 monclient: shutdown 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3571387725 >> v1:172.21.15.82:6790/0 conn(0x7f5cfc00f7c0 legacy=0x7f5cfc017760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3571387725 reap_dead start 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3571387725 shutdown_connections 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3571387725 >> 172.21.15.1:0/3571387725 conn(0x7f5d1c0a9020 msgr2=0x7f5d1c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3571387725 shutdown_connections 2024-08-19T10:04:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.783+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3571387725 wait complete. 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.784+0000 7f5d20f7f640 10 monclient: build_initial_monmap 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.784+0000 7f5d20f7f640 10 monclient: monmap: 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 librados: starting msgr at 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 librados: starting objecter 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 Processor -- start 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 -- start start 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 librados: setting wanted keys 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 librados: calling monclient init 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient: init 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient: _add_conns ranks=[1,0,5,2,7,3,8,4,6] 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient(hunting): picked mon.noname-f con 0x7f5cfc00f7c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d1c13a030 addr v1:172.21.15.1:6789/0 2024-08-19T10:04:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient(hunting): picked mon.noname-h con 0x7f5d1c125ae0 addr v1:172.21.15.82:6791/0 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c124f80 con 0x7f5d1c13a030 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c0aa340 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d19d9b640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f5d1c13a030 legacy=0x7f5d1c1d3100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c0a2690 con 0x7f5d1c125ae0 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.785+0000 7f5d20f7f640 10 monclient(hunting): authenticate will time out at 1498.861872s 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.786+0000 7f5d1ad9d640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f5d1c125ae0 0x7f5d1c1d6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:47056/0 (socket says 172.21.15.1:47056) 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.786+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3821186225 learned_addr learned my addr 172.21.15.1:0/3821186225 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d1a59c640 1 -- 172.21.15.1:0/3821186225 >> v1:172.21.15.82:6789/0 conn(0x7f5cfc00f7c0 legacy=0x7f5d1c139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 client.?.objecter ms_handle_connect 0x7f5cfc00f7c0 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 client.?.objecter ms_handle_connect 0x7f5d1c125ae0 2024-08-19T10:04:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d1ad9d640 1 -- 172.21.15.1:0/3821186225 >> v1:172.21.15.82:6791/0 conn(0x7f5d1c125ae0 legacy=0x7f5d1c1d6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979711779 0 0) 0x7f5cfc00fc70 con 0x7f5d1c125ae0 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _negotiate 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): my global_id is 27743 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cfc00ac40 con 0x7f5d1c125ae0 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311778940 0 0) 0x7f5d1c06dbb0 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _negotiate 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): my global_id is 174700 2024-08-19T10:04:02.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cfc00fc70 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3895485153 0 0) 0x7f5cfc00ac40 con 0x7f5d1c125ae0 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d1c06dbb0 con 0x7f5d1c125ae0 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5d14001cc0 con 0x7f5d1c125ae0 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): dump: 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:02.802 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2113646090 0 0) 0x7f5cfc00fc70 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cfc00ac40 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5d0c004750 con 0x7f5cfc00f7c0 2024-08-19T10:04:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient(hunting): dump: 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:02.804 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2514285932 0 0) 0x7f5d1c06dbb0 con 0x7f5d1c125ae0 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 >> v1:172.21.15.82:6789/0 conn(0x7f5cfc00f7c0 legacy=0x7f5d1c139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 >> v1:172.21.15.1:6789/0 conn(0x7f5d1c13a030 legacy=0x7f5d1c1d3100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient: _finish_hunting 0 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 monclient: found mon.h 2024-08-19T10:04:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c127000 con 0x7f5d1c125ae0 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient: _finish_auth 0 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 10 monclient: _check_auth_tickets 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 5 monclient: authenticate success, global_id 27743 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 10 monclient: _renew_subs 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c123e30 con 0x7f5d1c125ae0 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 10 client.27743.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 10 monclient: _renew_subs 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:04:02.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d1c1d7f50 con 0x7f5d1c125ae0 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.788+0000 7f5d20f7f640 1 librados: init done 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d14002070 con 0x7f5d1c125ae0 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5d14005490 con 0x7f5d1c125ae0 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d18d99640 4 set_mon_vals no callback set 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 10 monclient: dump: 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:02.807 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5d140118f0 con 0x7f5d1c125ae0 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d20f7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:02.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d20f7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d20f7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.789+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d00005080 con 0x7f5d1c125ae0 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.790+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(491..491 src has 299..491) ==== 6280+0+0 (unknown 14408793 0 0) 0x7f5d1404e3f0 con 0x7f5d1c125ae0 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.790+0000 7f5d0affd640 10 client.27743.objecter ms_dispatch 0x7f5d1c000b90 osd_map(491..491 src has 299..491) 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.790+0000 7f5d0affd640 3 client.27743.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.790+0000 7f5d0affd640 3 client.27743.objecter handle_osd_map decoding full epoch 491 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.790+0000 7f5d0affd640 20 client.27743.objecter dump_active .. 0 homeless 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.790+0000 7f5d1a59c640 10 monclient: get_auth_request con 0x7f5cfc043370 auth_method 0 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.792+0000 7f5d0affd640 10 client.27743.objecter ms_handle_connect 0x7f5cfc043370 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.795+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f5d1c0a2690 con 0x7f5d1c125ae0 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.795+0000 7f5d0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:02.795+0000 7f5d0affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:04:03.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d20f7f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:04:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d20f7f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:04:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d20f7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:04:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5d00005370 con 0x7f5d1c125ae0 2024-08-19T10:04:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d0affd640 1 -- 172.21.15.1:0/3821186225 <== mon.5 v1:172.21.15.82:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2679 (unknown 254746657 0 307875475) 0x7f5d14017580 con 0x7f5d1c125ae0 2024-08-19T10:04:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:04:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.051+0000 7f5d0affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:04:03.174 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":120,"quorum":[1,2,3,4,5],"quorum_names":["f","b","g","c","h"],"quorum_leader_name":"f","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:04:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 10 librados: watch_flush enter 2024-08-19T10:04:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 10 librados: watch_flush exit 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 20 client.27743.objecter shutdown clearing up homeless session... 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 10 client.27743.objecter successfully canceled tick 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5cfc043370 legacy=0x7f5cfc045830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 10 monclient: shutdown 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 >> v1:172.21.15.82:6791/0 conn(0x7f5d1c125ae0 legacy=0x7f5d1c1d6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 shutdown_connections 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 >> 172.21.15.1:0/3821186225 conn(0x7f5d1c0a9020 msgr2=0x7f5d1c1c18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 shutdown_connections 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 -- 172.21.15.1:0/3821186225 wait complete. 2024-08-19T10:04:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.053+0000 7f5d20f7f640 1 librados: shutdown 2024-08-19T10:04:03.178 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-08-19T10:04:03.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-08-19T10:04:03.178 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:04:03.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.277+0000 7fef46a9f640 10 monclient: get_monmap_and_config 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.277+0000 7fef46a9f640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.277+0000 7fef46a9f640 10 monclient: monmap: 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:04:03.278723+0000 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:04:03.278723+0000 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.279 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.280 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.278+0000 7fef46a9f640 1 Processor -- start 2024-08-19T10:04:03.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.278+0000 7fef46a9f640 1 -- start start 2024-08-19T10:04:03.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.278+0000 7fef46a9f640 10 monclient: init 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.278+0000 7fef46a9f640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.278+0000 7fef46a9f640 10 monclient: _add_conns ranks=[6,1,8,4,7,0,3,5,2] 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 10 monclient(hunting): picked mon.noname-d con 0x7fef400a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 10 monclient(hunting): picked mon.noname-f con 0x7fef400a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 10 monclient(hunting): picked mon.noname-e con 0x7fef4013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef400a3b40 con 0x7fef400a6ba0 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef400a8130 con 0x7fef400a67c0 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4006ede0 con 0x7fef4013f5c0 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef4483b640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7fef400a67c0 legacy=0x7fef400a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:04:03.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef46a9f640 10 monclient(hunting): authenticate will time out at 1499.355097s 2024-08-19T10:04:03.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef4503c640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7fef4013f5c0 legacy=0x7fef401419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:04:03.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef3ffff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fef400a6ba0 0x7fef400a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:35856/0 (socket says 172.21.15.1:35856) 2024-08-19T10:04:03.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.279+0000 7fef3ffff640 1 -- 172.21.15.1:0/2117682326 learned_addr learned my addr 172.21.15.1:0/2117682326 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.281+0000 7fef3ffff640 1 -- 172.21.15.1:0/2117682326 >> v1:172.21.15.82:6789/0 conn(0x7fef400a6ba0 legacy=0x7fef400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880313599 0 0) 0x7fef400a3b40 con 0x7fef400a6ba0 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient(hunting): my global_id is 174709 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef28003520 con 0x7fef400a6ba0 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2853912177 0 0) 0x7fef28003520 con 0x7fef400a6ba0 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 >> v1:172.21.15.1:6793/0 conn(0x7fef4013f5c0 legacy=0x7fef401419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 >> v1:172.21.15.1:6792/0 conn(0x7fef400a67c0 legacy=0x7fef400a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 monclient: found mon.noname-f 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef400a92a0 con 0x7fef400a6ba0 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 5 monclient: authenticate success, global_id 174709 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef34001ba0 con 0x7fef400a6ba0 2024-08-19T10:04:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: dump: 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef340048a0 con 0x7fef400a6ba0 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 10 monclient: get_monmap_and_config success 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 1 -- 172.21.15.1:0/2117682326 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef34004ed0 con 0x7fef400a6ba0 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 4 set_mon_vals no callback set 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef3f7fe640 10 monclient: dump: 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 10 monclient: shutdown 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 1 -- 172.21.15.1:0/2117682326 >> v1:172.21.15.82:6789/0 conn(0x7fef400a6ba0 legacy=0x7fef400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.282+0000 7fef46a9f640 1 -- 172.21.15.1:0/2117682326 shutdown_connections 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 -- 172.21.15.1:0/2117682326 >> 172.21.15.1:0/2117682326 conn(0x7fef400ac4b0 msgr2=0x7fef400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 -- 172.21.15.1:0/2117682326 shutdown_connections 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 -- 172.21.15.1:0/2117682326 wait complete. 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 10 monclient: monmap: 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 librados: starting msgr at 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 librados: starting objecter 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 Processor -- start 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 -- start start 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 librados: setting wanted keys 2024-08-19T10:04:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 1 librados: calling monclient init 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.283+0000 7fef46a9f640 10 monclient: init 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient: _add_conns ranks=[1,4,3,6,0,8,5,7,2] 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient(hunting): picked mon.noname-f con 0x7fef400a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient(hunting): picked mon.noname-c con 0x7fef400a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient(hunting): picked mon.noname-g con 0x7fef4013f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4013a820 con 0x7fef400a67c0 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef40142e20 con 0x7fef4013f5c0 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef400a9f70 con 0x7fef400a6ba0 2024-08-19T10:04:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef46a9f640 10 monclient(hunting): authenticate will time out at 1499.360535s 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef3ffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fef400a6ba0 0x7fef400a0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:35454/0 (socket says 172.21.15.1:35454) 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.284+0000 7fef3ffff640 1 -- 172.21.15.1:0/4149580180 learned_addr learned my addr 172.21.15.1:0/4149580180 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3ffff640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6791/0 conn(0x7fef400a6ba0 legacy=0x7fef400a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 client.?.objecter ms_handle_connect 0x7fef400a6ba0 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef4483b640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.82:6789/0 conn(0x7fef400a67c0 legacy=0x7fef4009fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 client.?.objecter ms_handle_connect 0x7fef400a67c0 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 client.?.objecter ms_handle_connect 0x7fef4013f5c0 2024-08-19T10:04:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef4503c640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.82:6790/0 conn(0x7fef4013f5c0 legacy=0x7fef401d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 582818930 0 0) 0x7fef400a9f70 con 0x7fef400a6ba0 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 194287 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.286+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef2c004eb0 con 0x7fef400a6ba0 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177761860 0 0) 0x7fef4013a820 con 0x7fef400a67c0 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 174718 2024-08-19T10:04:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef400a9f70 con 0x7fef400a67c0 2024-08-19T10:04:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975348933 0 0) 0x7fef40142e20 con 0x7fef4013f5c0 2024-08-19T10:04:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 184503 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef4013a820 con 0x7fef4013f5c0 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2869034096 0 0) 0x7fef2c004eb0 con 0x7fef400a6ba0 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef40142e20 con 0x7fef400a6ba0 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef34001b60 con 0x7fef400a6ba0 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): dump: 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1079377527 0 0) 0x7fef4013a820 con 0x7fef4013f5c0 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef2c004eb0 con 0x7fef4013f5c0 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef38004800 con 0x7fef4013f5c0 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient(hunting): dump: 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.312 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 719589123 0 0) 0x7fef400a9f70 con 0x7fef400a67c0 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef4013a820 con 0x7fef400a67c0 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2261442590 0 0) 0x7fef40142e20 con 0x7fef400a6ba0 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.82:6790/0 conn(0x7fef4013f5c0 legacy=0x7fef401d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.82:6789/0 conn(0x7fef400a67c0 legacy=0x7fef4009fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 monclient: found mon.c 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4013a260 con 0x7fef400a6ba0 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 5 monclient: authenticate success, global_id 194287 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 10 monclient: _renew_subs 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef4013a540 con 0x7fef400a6ba0 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 10 client.194287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 10 monclient: _renew_subs 2024-08-19T10:04:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fef400a0e00 con 0x7fef400a6ba0 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef46a9f640 1 librados: init done 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef340048a0 con 0x7fef400a6ba0 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef340054d0 con 0x7fef400a6ba0 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3e7fc640 4 set_mon_vals no callback set 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.287+0000 7fef3d7fa640 10 monclient: dump: 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.315 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.316 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef46a9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fef200065c0 con 0x7fef20002860 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fef34011930 con 0x7fef400a6ba0 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(491..491 src has 299..491) ==== 6280+0+0 (unknown 14408793 0 0) 0x7fef3404e570 con 0x7fef400a6ba0 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef3d7fa640 10 client.194287.objecter ms_dispatch 0x7fef40000b90 osd_map(491..491 src has 299..491) 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef3d7fa640 3 client.194287.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef3d7fa640 3 client.194287.objecter handle_osd_map decoding full epoch 491 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef3d7fa640 20 client.194287.objecter dump_active .. 0 homeless 2024-08-19T10:04:03.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.288+0000 7fef4503c640 10 monclient: get_auth_request con 0x7fef2c0417d0 auth_method 0 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 10 client.194287.objecter ms_handle_connect 0x7fef20002860 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef4483b640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6790/0 conn(0x7fef20002860 legacy=0x7fef20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 10 client.194287.objecter resend_mon_ops 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2628832202 0 0) 0x7fef200065c0 con 0x7fef20002860 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 184439 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.290+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef40142e20 con 0x7fef20002860 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 464675779 0 0) 0x7fef40142e20 con 0x7fef20002860 2024-08-19T10:04:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef200065c0 con 0x7fef20002860 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef30004db0 con 0x7fef20002860 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 10 monclient: dump: 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.319 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3098061885 0 0) 0x7fef200065c0 con 0x7fef20002860 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef2000a160 con 0x7fef20002860 2024-08-19T10:04:03.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 10 client.194287.objecter ms_handle_connect 0x7fef2c0417d0 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fef300070d0 con 0x7fef20002860 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6790/0 conn(0x7fef20002860 legacy=0x7fef20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.291+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6790/0 conn(0x7fef20002860 legacy=0x7fef20004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.297+0000 7fef46a9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.297+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fef20009e50 con 0x7fef20009110 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.299+0000 7fef4483b640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6790/0 conn(0x7fef20009110 legacy=0x7fef200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 client.194287.objecter ms_handle_connect 0x7fef20009110 2024-08-19T10:04:03.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 client.194287.objecter resend_mon_ops 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598314417 0 0) 0x7fef20009e50 con 0x7fef20009110 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 184448 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef200065c0 con 0x7fef20009110 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4192042684 0 0) 0x7fef200065c0 con 0x7fef20009110 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef20009e50 con 0x7fef20009110 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef30005700 con 0x7fef20009110 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:04:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient: dump: 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3667202311 0 0) 0x7fef20009e50 con 0x7fef20009110 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fef2000c3a0 con 0x7fef20009110 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7641 (unknown 0 0 1640051721) 0x7fef3000a0a0 con 0x7fef20009110 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6790/0 conn(0x7fef20009110 legacy=0x7fef200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef3d7fa640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6790/0 conn(0x7fef20009110 legacy=0x7fef200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.325 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:04:03.325 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:04:03.325 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:04:03.325 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:04:03.325 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 120, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 160548, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:04:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.327 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.328 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:04:03.329 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:04:03.330 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.331 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.333 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:04:03.334 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.335 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.336 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:04:03.339 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.340 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.342 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.342 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:04:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:04:03.347 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.347 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.348 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.348 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.348 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.348 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:04:03.349 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:04:03.349 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.349 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.353 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:04:03.353 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:04:03.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.354 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:04:03.356 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:04:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef46a9f640 10 librados: watch_flush enter 2024-08-19T10:04:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef46a9f640 10 librados: watch_flush exit 2024-08-19T10:04:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef46a9f640 20 client.194287.objecter shutdown clearing up homeless session... 2024-08-19T10:04:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.300+0000 7fef46a9f640 10 client.194287.objecter successfully canceled tick 2024-08-19T10:04:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.82:6812/2549104415 conn(0x7fef2c0417d0 legacy=0x7fef2c043c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 10 monclient: shutdown 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 >> v1:172.21.15.1:6791/0 conn(0x7fef400a6ba0 legacy=0x7fef400a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef4503c640 1 -- 172.21.15.1:0/4149580180 reap_dead start 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef4503c640 1 -- 172.21.15.1:0/4149580180 reap_dead start 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 shutdown_connections 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 >> 172.21.15.1:0/4149580180 conn(0x7fef400ac4b0 msgr2=0x7fef400a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 shutdown_connections 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 -- 172.21.15.1:0/4149580180 wait complete. 2024-08-19T10:04:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.301+0000 7fef46a9f640 1 librados: shutdown 2024-08-19T10:04:03.359 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T10:04:03.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-08-19T10:04:03.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.459+0000 7f1706c30640 10 monclient: get_monmap_and_config 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.459+0000 7f1706c30640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.459+0000 7f1706c30640 10 monclient: monmap: 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:04:03.460039+0000 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:04:03.460039+0000 2024-08-19T10:04:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.461 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.459+0000 7f1706c30640 1 Processor -- start 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.459+0000 7f1706c30640 1 -- start start 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.459+0000 7f1706c30640 10 monclient: init 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient: _add_conns ranks=[2,8,1,5,0,4,6,3,7] 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient(hunting): picked mon.noname-b con 0x7f17000a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient(hunting): picked mon.noname-e con 0x7f17000694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient(hunting): picked mon.noname-f con 0x7f17000a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17000a3c70 con 0x7f17000a46a0 2024-08-19T10:04:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17000a33d0 con 0x7f17000a6830 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17000a36b0 con 0x7f17000694b0 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f1706c30640 10 monclient(hunting): authenticate will time out at 1499.536380s 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f16fffff640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f17000694b0 legacy=0x7f17000a7320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f17049cc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f17000a6830 0x7f17000a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:56632/0 (socket says 172.21.15.1:56632) 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.460+0000 7f17049cc640 1 -- 172.21.15.1:0/2535642253 learned_addr learned my addr 172.21.15.1:0/2535642253 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f17049cc640 1 -- 172.21.15.1:0/2535642253 >> v1:172.21.15.1:6790/0 conn(0x7f17000a6830 legacy=0x7f17000a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4181885924 0 0) 0x7f17000a33d0 con 0x7f17000a6830 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f16ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f16ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f16ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f16ff7fe640 10 monclient(hunting): my global_id is 184457 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16e8003520 con 0x7f17000a6830 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.462+0000 7f17051cd640 1 -- 172.21.15.1:0/2535642253 >> v1:172.21.15.82:6789/0 conn(0x7f17000a46a0 legacy=0x7f1700141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603064829 0 0) 0x7f17000a3c70 con 0x7f17000a46a0 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient(hunting): my global_id is 174727 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17000a33d0 con 0x7f17000a46a0 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1084552839 0 0) 0x7f16e8003520 con 0x7f17000a6830 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 >> v1:172.21.15.1:6793/0 conn(0x7f17000694b0 legacy=0x7f17000a7320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 >> v1:172.21.15.82:6789/0 conn(0x7f17000a46a0 legacy=0x7f1700141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 monclient: found mon.noname-b 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170006dbb0 con 0x7f17000a6830 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f16f8001e60 con 0x7f17000a6830 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: dump: 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 5 monclient: authenticate success, global_id 184457 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16f8004c20 con 0x7f17000a6830 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 1 -- 172.21.15.1:0/2535642253 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f16f8005190 con 0x7f17000a6830 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 10 monclient: get_monmap_and_config success 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 4 set_mon_vals no callback set 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f16ff7fe640 10 monclient: dump: 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 10 monclient: shutdown 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 1 -- 172.21.15.1:0/2535642253 >> v1:172.21.15.1:6790/0 conn(0x7f17000a6830 legacy=0x7f17000a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 1 -- 172.21.15.1:0/2535642253 shutdown_connections 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 1 -- 172.21.15.1:0/2535642253 >> 172.21.15.1:0/2535642253 conn(0x7f17000a9020 msgr2=0x7f1700068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 1 -- 172.21.15.1:0/2535642253 shutdown_connections 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.463+0000 7f1706c30640 1 -- 172.21.15.1:0/2535642253 wait complete. 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 10 monclient: monmap: 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 1 librados: starting msgr at 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 1 librados: starting objecter 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 1 Processor -- start 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 1 -- start start 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 1 librados: setting wanted keys 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 1 librados: calling monclient init 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.464+0000 7f1706c30640 10 monclient: init 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient: _add_conns ranks=[1,8,2,3,5,4,6,7,0] 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient(hunting): picked mon.noname-f con 0x7f17000694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient(hunting): picked mon.noname-e con 0x7f17000a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient(hunting): picked mon.noname-b con 0x7f17000a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1700137d40 con 0x7f17000694b0 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1700136850 con 0x7f17000a6830 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1700135e30 con 0x7f17000a46a0 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f16fffff640 1 -- >> v1:172.21.15.1:6793/0 conn(0x7f17000a46a0 legacy=0x7f1700135720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6793/0 2024-08-19T10:04:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f1706c30640 10 monclient(hunting): authenticate will time out at 1499.541434s 2024-08-19T10:04:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f17051cd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f17000a6830 0x7f17001d6400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:56644/0 (socket says 172.21.15.1:56644) 2024-08-19T10:04:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.465+0000 7f17051cd640 1 -- 172.21.15.1:0/1865532533 learned_addr learned my addr 172.21.15.1:0/1865532533 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f17051cd640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6790/0 conn(0x7f17000a6830 legacy=0x7f17001d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 client.?.objecter ms_handle_connect 0x7f17000a6830 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688618250 0 0) 0x7f1700136850 con 0x7f17000a6830 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): my global_id is 184466 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16ec004eb0 con 0x7f17000a6830 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 client.?.objecter ms_handle_connect 0x7f17000694b0 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f17049cc640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.82:6789/0 conn(0x7f17000694b0 legacy=0x7f1700135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415663789 0 0) 0x7f1700137d40 con 0x7f17000694b0 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 10 monclient(hunting): my global_id is 174736 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.467+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1700136850 con 0x7f17000694b0 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2030929292 0 0) 0x7f16ec004eb0 con 0x7f17000a6830 2024-08-19T10:04:03.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1700137d40 con 0x7f17000a6830 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f16f0003610 con 0x7f17000a6830 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient(hunting): dump: 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.487 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4064214108 0 0) 0x7f1700137d40 con 0x7f17000a6830 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6793/0 conn(0x7f17000a46a0 legacy=0x7f1700135720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.82:6789/0 conn(0x7f17000694b0 legacy=0x7f1700135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 monclient: found mon.b 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1700138bb0 con 0x7f17000a6830 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 5 monclient: authenticate success, global_id 184466 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 10 monclient: _renew_subs 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1700138310 con 0x7f17000a6830 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 10 client.184466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 10 monclient: _renew_subs 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1700142490 con 0x7f17000a6830 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f1706c30640 1 librados: init done 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16f0003b70 con 0x7f17000a6830 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f16f0004060 con 0x7f17000a6830 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16feffd640 4 set_mon_vals no callback set 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.468+0000 7f16fd7fa640 10 monclient: dump: 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.490 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f16f0004200 con 0x7f17000a6830 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f1706c30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f16e00065c0 con 0x7f16e0002860 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(491..491 src has 299..491) ==== 6280+0+0 (unknown 14408793 0 0) 0x7f16f004e5d0 con 0x7f17000a6830 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fd7fa640 10 client.184466.objecter ms_dispatch 0x7f1700000b90 osd_map(491..491 src has 299..491) 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fd7fa640 3 client.184466.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fd7fa640 3 client.184466.objecter handle_osd_map decoding full epoch 491 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fd7fa640 20 client.184466.objecter dump_active .. 0 homeless 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.469+0000 7f16fffff640 10 monclient: get_auth_request con 0x7f16ec040e30 auth_method 0 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f17049cc640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6791/0 conn(0x7f16e0002860 legacy=0x7f16e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 client.184466.objecter ms_handle_connect 0x7f16e0002860 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 client.184466.objecter resend_mon_ops 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609294791 0 0) 0x7f16e00065c0 con 0x7f16e0002860 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient(hunting): my global_id is 194296 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1700137d40 con 0x7f16e0002860 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 477182566 0 0) 0x7f1700137d40 con 0x7f16e0002860 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16e00065c0 con 0x7f16e0002860 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f16f8004c20 con 0x7f16e0002860 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 monclient: dump: 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.493 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.471+0000 7f16fd7fa640 10 client.184466.objecter ms_handle_connect 0x7f16ec040e30 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3795438644 0 0) 0x7f16e00065c0 con 0x7f16e0002860 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16e000a160 con 0x7f16e0002860 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f16f8007570 con 0x7f16e0002860 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6791/0 conn(0x7f16e0002860 legacy=0x7f16e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.472+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6791/0 conn(0x7f16e0002860 legacy=0x7f16e0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.478+0000 7f1706c30640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.478+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f16e0009e50 con 0x7f16e0009110 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.480+0000 7f17049cc640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6791/0 conn(0x7f16e0009110 legacy=0x7f16e00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.480+0000 7f16fd7fa640 10 client.184466.objecter ms_handle_connect 0x7f16e0009110 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.480+0000 7f16fd7fa640 10 client.184466.objecter resend_mon_ops 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747413901 0 0) 0x7f16e0009e50 con 0x7f16e0009110 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient(hunting): my global_id is 194305 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16e00065c0 con 0x7f16e0009110 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3385082610 0 0) 0x7f16e00065c0 con 0x7f16e0009110 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16e0009e50 con 0x7f16e0009110 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f16f8005da0 con 0x7f16e0009110 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient: dump: 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1383633656 0 0) 0x7f16e0009e50 con 0x7f16e0009110 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16e000c3a0 con 0x7f16e0009110 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7640 (unknown 0 0 2995748995) 0x7f16f8007270 con 0x7f16e0009110 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6791/0 conn(0x7f16e0009110 legacy=0x7f16e00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f16fd7fa640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6791/0 conn(0x7f16e0009110 legacy=0x7f16e00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 120, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 31168, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:04:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:04:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:04:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:04:03.518 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f1706c30640 10 librados: watch_flush enter 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f1706c30640 10 librados: watch_flush exit 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f1706c30640 20 client.184466.objecter shutdown clearing up homeless session... 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.481+0000 7f1706c30640 10 client.184466.objecter successfully canceled tick 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.82:6812/2549104415 conn(0x7f16ec040e30 legacy=0x7f16ec0432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 10 monclient: shutdown 2024-08-19T10:04:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 >> v1:172.21.15.1:6790/0 conn(0x7f17000a6830 legacy=0x7f17001d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f17051cd640 1 -- 172.21.15.1:0/1865532533 reap_dead start 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f17051cd640 1 -- 172.21.15.1:0/1865532533 reap_dead start 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 shutdown_connections 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 >> 172.21.15.1:0/1865532533 conn(0x7f17000a9020 msgr2=0x7f17000a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 shutdown_connections 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 -- 172.21.15.1:0/1865532533 wait complete. 2024-08-19T10:04:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.482+0000 7f1706c30640 1 librados: shutdown 2024-08-19T10:04:03.521 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T10:04:03.521 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-08-19T10:04:03.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:04:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.620+0000 7f8cbde15640 10 monclient: get_monmap_and_config 2024-08-19T10:04:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.620+0000 7f8cbde15640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.620+0000 7f8cbde15640 10 monclient: monmap: 2024-08-19T10:04:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:04:03.621670+0000 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:04:03.621670+0000 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.623 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.624 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.621+0000 7f8cbde15640 1 Processor -- start 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.621+0000 7f8cbde15640 1 -- start start 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.621+0000 7f8cbde15640 10 monclient: init 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.621+0000 7f8cbde15640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.621+0000 7f8cbde15640 10 monclient: _add_conns ranks=[6,7,3,0,5,1,4,2,8] 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.621+0000 7f8cbde15640 10 monclient(hunting): picked mon.noname-d con 0x7f8cb80a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 10 monclient(hunting): picked mon.noname-i con 0x7f8cb80a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 10 monclient(hunting): picked mon.noname-g con 0x7f8cb813f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb80a3b40 con 0x7f8cb813f5c0 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb80a8130 con 0x7f8cb80a67c0 2024-08-19T10:04:03.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb806ede0 con 0x7f8cb80a6ba0 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cbde15640 10 monclient(hunting): authenticate will time out at 1499.698058s 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cb77fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8cb80a67c0 legacy=0x7f8cb80a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cb6ffd640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7f8cb80a6ba0 legacy=0x7f8cb80a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cb7fff640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7f8cb813f5c0 0x7f8cb81419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:36916/0 (socket says 172.21.15.1:36916) 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.622+0000 7f8cb7fff640 1 -- 172.21.15.1:0/3324515373 learned_addr learned my addr 172.21.15.1:0/3324515373 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb7fff640 1 -- 172.21.15.1:0/3324515373 >> v1:172.21.15.82:6790/0 conn(0x7f8cb813f5c0 legacy=0x7f8cb81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579307844 0 0) 0x7f8cb80a3b40 con 0x7f8cb813f5c0 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 184512 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.624+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ca0003520 con 0x7f8cb813f5c0 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2400275136 0 0) 0x7f8ca0003520 con 0x7f8cb813f5c0 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 >> v1:172.21.15.82:6792/0 conn(0x7f8cb80a6ba0 legacy=0x7f8cb80a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 >> v1:172.21.15.1:6792/0 conn(0x7f8cb80a67c0 legacy=0x7f8cb80a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 monclient: found mon.noname-g 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb80a92a0 con 0x7f8cb813f5c0 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8cb0001e60 con 0x7f8cb813f5c0 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 5 monclient: authenticate success, global_id 184512 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: dump: 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.633 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8cb0004c20 con 0x7f8cb813f5c0 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 10 monclient: get_monmap_and_config success 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 4 set_mon_vals no callback set 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 1 -- 172.21.15.1:0/3324515373 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8cb00051d0 con 0x7f8cb813f5c0 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:04:03.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cb67fc640 10 monclient: dump: 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.635 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 10 monclient: shutdown 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 1 -- 172.21.15.1:0/3324515373 >> v1:172.21.15.82:6790/0 conn(0x7f8cb813f5c0 legacy=0x7f8cb81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 1 -- 172.21.15.1:0/3324515373 shutdown_connections 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 1 -- 172.21.15.1:0/3324515373 >> 172.21.15.1:0/3324515373 conn(0x7f8cb80ac4b0 msgr2=0x7f8cb80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 1 -- 172.21.15.1:0/3324515373 shutdown_connections 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.625+0000 7f8cbde15640 1 -- 172.21.15.1:0/3324515373 wait complete. 2024-08-19T10:04:03.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 10 monclient: monmap: 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.637 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 1 librados: starting msgr at 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 1 librados: starting objecter 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 1 Processor -- start 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 1 -- start start 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 1 librados: setting wanted keys 2024-08-19T10:04:03.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 1 librados: calling monclient init 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.626+0000 7f8cbde15640 10 monclient: init 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient: _add_conns ranks=[2,5,6,8,3,4,7,1,0] 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb80a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient(hunting): picked mon.noname-h con 0x7f8cb80a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient(hunting): picked mon.noname-d con 0x7f8cb813f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb80a9e50 con 0x7f8cb80a67c0 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb80a53e0 con 0x7f8cb80a6ba0 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb813c420 con 0x7f8cb813f5c0 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cbde15640 10 monclient(hunting): authenticate will time out at 1499.703400s 2024-08-19T10:04:03.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cb7fff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7f8cb813f5c0 legacy=0x7f8cb81d5a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:04:03.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cb77fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f8cb80a67c0 0x7f8cb81cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:56658/0 (socket says 172.21.15.1:56658) 2024-08-19T10:04:03.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.627+0000 7f8cb77fe640 1 -- 172.21.15.1:0/2628361582 learned_addr learned my addr 172.21.15.1:0/2628361582 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8cb77fe640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.1:6790/0 conn(0x7f8cb80a67c0 legacy=0x7f8cb81cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 client.?.objecter ms_handle_connect 0x7f8cb80a67c0 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8cb6ffd640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6791/0 conn(0x7f8cb80a6ba0 legacy=0x7f8cb813ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 client.?.objecter ms_handle_connect 0x7f8cb80a6ba0 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978539009 0 0) 0x7f8cb80a9e50 con 0x7f8cb80a67c0 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 184475 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ca4004eb0 con 0x7f8cb80a67c0 2024-08-19T10:04:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864057240 0 0) 0x7f8cb80a53e0 con 0x7f8cb80a6ba0 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.629+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 27752 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cb80a9e50 con 0x7f8cb80a6ba0 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3412705346 0 0) 0x7f8ca4004eb0 con 0x7f8cb80a67c0 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cb80a53e0 con 0x7f8cb80a67c0 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8ca8004620 con 0x7f8cb80a67c0 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient(hunting): dump: 2024-08-19T10:04:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.649 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3009316270 0 0) 0x7f8cb80a9e50 con 0x7f8cb80a6ba0 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ca4004eb0 con 0x7f8cb80a6ba0 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2060014074 0 0) 0x7f8cb80a53e0 con 0x7f8cb80a67c0 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.1:6792/0 conn(0x7f8cb813f5c0 legacy=0x7f8cb81d5a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:04:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6791/0 conn(0x7f8cb80a6ba0 legacy=0x7f8cb813ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 monclient: found mon.b 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb813bd20 con 0x7f8cb80a67c0 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 5 monclient: authenticate success, global_id 184475 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 10 monclient: _renew_subs 2024-08-19T10:04:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb813f160 con 0x7f8cb80a67c0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 10 client.184475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 10 monclient: _renew_subs 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8cb81d7310 con 0x7f8cb80a67c0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cbde15640 1 librados: init done 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ca80048f0 con 0x7f8cb80a67c0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8ca8004e60 con 0x7f8cb80a67c0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:04:03.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8cb5ffb640 4 set_mon_vals no callback set 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.630+0000 7f8c9ffff640 10 monclient: dump: 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8cbde15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8c980065c0 con 0x7f8c98002860 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8ca8005000 con 0x7f8cb80a67c0 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(491..491 src has 299..491) ==== 6280+0+0 (unknown 14408793 0 0) 0x7f8ca804e5b0 con 0x7f8cb80a67c0 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8c9ffff640 10 client.184475.objecter ms_dispatch 0x7f8cb8000b90 osd_map(491..491 src has 299..491) 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8c9ffff640 3 client.184475.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8c9ffff640 3 client.184475.objecter handle_osd_map decoding full epoch 491 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8c9ffff640 20 client.184475.objecter dump_active .. 0 homeless 2024-08-19T10:04:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.631+0000 7f8cb7fff640 10 monclient: get_auth_request con 0x7f8ca4041170 auth_method 0 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8cb6ffd640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6789/0 conn(0x7f8c98002860 legacy=0x7f8c98004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 10 client.184475.objecter ms_handle_connect 0x7f8c98002860 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 10 client.184475.objecter resend_mon_ops 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347838858 0 0) 0x7f8c980065c0 con 0x7f8c98002860 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 174745 2024-08-19T10:04:03.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.633+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cb80a53e0 con 0x7f8c98002860 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3802508384 0 0) 0x7f8cb80a53e0 con 0x7f8c98002860 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c980065c0 con 0x7f8c98002860 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8cac004ec0 con 0x7f8c98002860 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 10 monclient: dump: 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 10 client.184475.objecter ms_handle_connect 0x7f8ca4041170 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 73665132 0 0) 0x7f8c980065c0 con 0x7f8c98002860 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c9800a160 con 0x7f8c98002860 2024-08-19T10:04:03.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8cac007200 con 0x7f8c98002860 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6789/0 conn(0x7f8c98002860 legacy=0x7f8c98004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.634+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6789/0 conn(0x7f8c98002860 legacy=0x7f8c98004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.640+0000 7f8cbde15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.640+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8c98009e50 con 0x7f8c98009110 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8cb6ffd640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6789/0 conn(0x7f8c98009110 legacy=0x7f8c980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 client.184475.objecter ms_handle_connect 0x7f8c98009110 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 client.184475.objecter resend_mon_ops 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515541303 0 0) 0x7f8c98009e50 con 0x7f8c98009110 2024-08-19T10:04:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 174754 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c980065c0 con 0x7f8c98009110 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 505583243 0 0) 0x7f8c980065c0 con 0x7f8c98009110 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c98009e50 con 0x7f8c98009110 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8cac006910 con 0x7f8c98009110 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 10 monclient: dump: 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2902635063 0 0) 0x7f8c98009e50 con 0x7f8c98009110 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.643+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8c9800c3f0 con 0x7f8c98009110 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7643 (unknown 0 0 3074249559) 0x7f8cac00a1d0 con 0x7f8c98009110 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8c9ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8c9ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6789/0 conn(0x7f8c98009110 legacy=0x7f8c980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8c9ffff640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6789/0 conn(0x7f8c98009110 legacy=0x7f8c980095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 120, 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 293701, 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:04:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:04:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:04:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:04:03.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.675 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.678 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:04:03.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.681 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:04:03.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:04:03.687 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 10 librados: watch_flush enter 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 10 librados: watch_flush exit 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 20 client.184475.objecter shutdown clearing up homeless session... 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 10 client.184475.objecter successfully canceled tick 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8ca4041170 legacy=0x7f8ca4043630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 10 monclient: shutdown 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 >> v1:172.21.15.1:6790/0 conn(0x7f8cb80a67c0 legacy=0x7f8cb81cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cb7fff640 1 -- 172.21.15.1:0/2628361582 reap_dead start 2024-08-19T10:04:03.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cb7fff640 1 -- 172.21.15.1:0/2628361582 reap_dead start 2024-08-19T10:04:03.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 shutdown_connections 2024-08-19T10:04:03.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 >> 172.21.15.1:0/2628361582 conn(0x7f8cb80ac4b0 msgr2=0x7f8cb80aafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 shutdown_connections 2024-08-19T10:04:03.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 -- 172.21.15.1:0/2628361582 wait complete. 2024-08-19T10:04:03.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.644+0000 7f8cbde15640 1 librados: shutdown 2024-08-19T10:04:03.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T10:04:03.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-08-19T10:04:03.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.790+0000 7fae3a4e5640 10 monclient: get_monmap_and_config 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.790+0000 7fae3a4e5640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.790+0000 7fae3a4e5640 10 monclient: monmap: 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:04:03.791868+0000 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:04:03.791868+0000 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.792 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.793 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.791+0000 7fae3a4e5640 1 Processor -- start 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.791+0000 7fae3a4e5640 1 -- start start 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.791+0000 7fae3a4e5640 10 monclient: init 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient: _add_conns ranks=[5,6,1,4,3,0,2,7,8] 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient(hunting): picked mon.noname-h con 0x7fae340a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:04:03.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient(hunting): picked mon.noname-d con 0x7fae340a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient(hunting): picked mon.noname-f con 0x7fae3413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae340a3b40 con 0x7fae3413f5c0 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae340a8130 con 0x7fae340a67c0 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae3406ede0 con 0x7fae340a6ba0 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae3a4e5640 10 monclient(hunting): authenticate will time out at 1499.868230s 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae337fe640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7fae340a6ba0 legacy=0x7fae340a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae38a82640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fae3413f5c0 0x7fae341419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:35922/0 (socket says 172.21.15.1:35922) 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae38a82640 1 -- 172.21.15.1:0/2161764813 learned_addr learned my addr 172.21.15.1:0/2161764813 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.792+0000 7fae33fff640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fae340a67c0 0x7fae340a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:47082/0 (socket says 172.21.15.1:47082) 2024-08-19T10:04:03.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.794+0000 7fae33fff640 1 -- 172.21.15.1:0/2161764813 >> v1:172.21.15.82:6791/0 conn(0x7fae340a67c0 legacy=0x7fae340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.794+0000 7fae38a82640 1 -- 172.21.15.1:0/2161764813 >> v1:172.21.15.82:6789/0 conn(0x7fae3413f5c0 legacy=0x7fae341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1529980524 0 0) 0x7fae340a3b40 con 0x7fae3413f5c0 2024-08-19T10:04:03.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): my global_id is 174763 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae1c003520 con 0x7fae3413f5c0 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1612034062 0 0) 0x7fae340a8130 con 0x7fae340a67c0 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient(hunting): my global_id is 27761 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae340a3b40 con 0x7fae340a67c0 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1606016157 0 0) 0x7fae1c003520 con 0x7fae3413f5c0 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 >> v1:172.21.15.1:6792/0 conn(0x7fae340a6ba0 legacy=0x7fae340a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 >> v1:172.21.15.82:6791/0 conn(0x7fae340a67c0 legacy=0x7fae340a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 monclient: found mon.noname-f 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae340a92a0 con 0x7fae3413f5c0 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae3a4e5640 5 monclient: authenticate success, global_id 174763 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae3a4e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae2c001e50 con 0x7fae3413f5c0 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:04:03.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: dump: 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae3a4e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae2c004c20 con 0x7fae3413f5c0 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae3a4e5640 10 monclient: get_monmap_and_config success 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae3a4e5640 4 set_mon_vals no callback set 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 1 -- 172.21.15.1:0/2161764813 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae2c005110 con 0x7fae3413f5c0 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.795+0000 7fae32ffd640 10 monclient: dump: 2024-08-19T10:04:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 10 monclient: shutdown 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 1 -- 172.21.15.1:0/2161764813 >> v1:172.21.15.82:6789/0 conn(0x7fae3413f5c0 legacy=0x7fae341419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 1 -- 172.21.15.1:0/2161764813 shutdown_connections 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 1 -- 172.21.15.1:0/2161764813 >> 172.21.15.1:0/2161764813 conn(0x7fae340ac4b0 msgr2=0x7fae340a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 1 -- 172.21.15.1:0/2161764813 shutdown_connections 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 1 -- 172.21.15.1:0/2161764813 wait complete. 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.796+0000 7fae3a4e5640 10 monclient: monmap: 2024-08-19T10:04:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 librados: starting msgr at 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 librados: starting objecter 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 Processor -- start 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 -- start start 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 librados: setting wanted keys 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 librados: calling monclient init 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient: init 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient: _add_conns ranks=[1,0,5,3,7,4,2,8,6] 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient(hunting): picked mon.noname-f con 0x7fae340a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient(hunting): picked mon.noname-a con 0x7fae340a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient(hunting): picked mon.noname-h con 0x7fae3413f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae34139a40 con 0x7fae340a6ba0 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae340a1620 con 0x7fae340a67c0 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae3413a6f0 con 0x7fae3413f5c0 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae3a4e5640 10 monclient(hunting): authenticate will time out at 1499.873851s 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.797+0000 7fae337fe640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fae340a6ba0 legacy=0x7fae341d2bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.798+0000 7fae33fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fae340a67c0 0x7fae3413ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:35936/0 (socket says 172.21.15.1:35936) 2024-08-19T10:04:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.798+0000 7fae33fff640 1 -- 172.21.15.1:0/984140745 learned_addr learned my addr 172.21.15.1:0/984140745 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae38a82640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6791/0 conn(0x7fae3413f5c0 legacy=0x7fae341d6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae33fff640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6789/0 conn(0x7fae340a67c0 legacy=0x7fae3413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 client.?.objecter ms_handle_connect 0x7fae3413f5c0 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 client.?.objecter ms_handle_connect 0x7fae340a67c0 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636445686 0 0) 0x7fae340a1620 con 0x7fae340a67c0 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): my global_id is 174772 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae20004eb0 con 0x7fae340a67c0 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018161834 0 0) 0x7fae3413a6f0 con 0x7fae3413f5c0 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): my global_id is 27770 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae340a1620 con 0x7fae3413f5c0 2024-08-19T10:04:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3301867557 0 0) 0x7fae20004eb0 con 0x7fae340a67c0 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae3413a6f0 con 0x7fae340a67c0 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae240047e0 con 0x7fae340a67c0 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): dump: 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2829699498 0 0) 0x7fae340a1620 con 0x7fae3413f5c0 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae20004eb0 con 0x7fae3413f5c0 2024-08-19T10:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae2c005110 con 0x7fae3413f5c0 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient(hunting): dump: 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1470075686 0 0) 0x7fae3413a6f0 con 0x7fae340a67c0 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6791/0 conn(0x7fae3413f5c0 legacy=0x7fae341d6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.1:6789/0 conn(0x7fae340a6ba0 legacy=0x7fae341d2bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 monclient: found mon.f 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.800+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae34139e50 con 0x7fae340a67c0 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 5 monclient: authenticate success, global_id 174772 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 10 monclient: _renew_subs 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fae3413f360 con 0x7fae340a67c0 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 10 client.174772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 10 monclient: _renew_subs 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fae341cad90 con 0x7fae340a67c0 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 1 librados: init done 2024-08-19T10:04:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae24004cb0 con 0x7fae340a67c0 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae240051e0 con 0x7fae340a67c0 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae327fc640 4 set_mon_vals no callback set 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae30ff9640 10 monclient: dump: 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.801+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fae140065c0 con 0x7fae14002860 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fae24005380 con 0x7fae340a67c0 2024-08-19T10:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(492..492 src has 299..492) ==== 6288+0+0 (unknown 3038875124 0 0) 0x7fae2404e5f0 con 0x7fae340a67c0 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae30ff9640 10 client.174772.objecter ms_dispatch 0x7fae34000b90 osd_map(492..492 src has 299..492) 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae30ff9640 3 client.174772.objecter handle_osd_map got epochs [492,492] > 0 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae30ff9640 3 client.174772.objecter handle_osd_map decoding full epoch 492 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae38a82640 10 monclient: get_auth_request con 0x7fae200413c0 auth_method 0 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.802+0000 7fae30ff9640 20 client.174772.objecter dump_active .. 0 homeless 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 client.174772.objecter ms_handle_connect 0x7fae14002860 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 client.174772.objecter resend_mon_ops 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae337fe640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6790/0 conn(0x7fae14002860 legacy=0x7fae14004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783235727 0 0) 0x7fae140065c0 con 0x7fae14002860 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient(hunting): my global_id is 184521 2024-08-19T10:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae3413a6f0 con 0x7fae14002860 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2048126009 0 0) 0x7fae3413a6f0 con 0x7fae14002860 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae140065c0 con 0x7fae14002860 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae28004480 con 0x7fae14002860 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 monclient: dump: 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 10 client.174772.objecter ms_handle_connect 0x7fae200413c0 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 337476884 0 0) 0x7fae140065c0 con 0x7fae14002860 2024-08-19T10:04:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.804+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae1400a160 con 0x7fae14002860 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.805+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fae280059c0 con 0x7fae14002860 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.805+0000 7fae30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.805+0000 7fae30ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.805+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6790/0 conn(0x7fae14002860 legacy=0x7fae14004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.805+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6790/0 conn(0x7fae14002860 legacy=0x7fae14004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.811+0000 7fae3a4e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:04:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.811+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fae14009e70 con 0x7fae14009130 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.813+0000 7fae337fe640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6790/0 conn(0x7fae14009130 legacy=0x7fae140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.813+0000 7fae30ff9640 10 client.174772.objecter ms_handle_connect 0x7fae14009130 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.813+0000 7fae30ff9640 10 client.174772.objecter resend_mon_ops 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3629657506 0 0) 0x7fae14009e70 con 0x7fae14009130 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient(hunting): my global_id is 184530 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae140065c0 con 0x7fae14009130 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3030924444 0 0) 0x7fae140065c0 con 0x7fae14009130 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae14009e70 con 0x7fae14009130 2024-08-19T10:04:03.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fae28004d00 con 0x7fae14009130 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient: dump: 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2460554449 0 0) 0x7fae14009e70 con 0x7fae14009130 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae1400c410 con 0x7fae14009130 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7641 (unknown 0 0 3530811513) 0x7fae28004fe0 con 0x7fae14009130 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:04:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6790/0 conn(0x7fae14009130 legacy=0x7fae140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.814+0000 7fae30ff9640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6790/0 conn(0x7fae14009130 legacy=0x7fae140095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.831 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:04:03.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:04:03.831 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:04:03.831 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:04:03.831 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 120, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 161068, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:04:03.832 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.833 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.834 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:04:03.835 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.836 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:04:03.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:04:03.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.841 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.843 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:04:03.844 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.845 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.846 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.850 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:04:03.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:04:03.852 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:04:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 10 librados: watch_flush enter 2024-08-19T10:04:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 10 librados: watch_flush exit 2024-08-19T10:04:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 20 client.174772.objecter shutdown clearing up homeless session... 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 10 client.174772.objecter successfully canceled tick 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6812/2549104415 conn(0x7fae200413c0 legacy=0x7fae20043880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 10 monclient: shutdown 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 >> v1:172.21.15.82:6789/0 conn(0x7fae340a67c0 legacy=0x7fae3413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae38a82640 1 -- 172.21.15.1:0/984140745 reap_dead start 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae38a82640 1 -- 172.21.15.1:0/984140745 reap_dead start 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 shutdown_connections 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 >> 172.21.15.1:0/984140745 conn(0x7fae340ac4b0 msgr2=0x7fae340aa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 shutdown_connections 2024-08-19T10:04:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 -- 172.21.15.1:0/984140745 wait complete. 2024-08-19T10:04:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.815+0000 7fae3a4e5640 1 librados: shutdown 2024-08-19T10:04:03.856 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T10:04:03.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-08-19T10:04:03.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:04:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.955+0000 7fa0b1c4f640 10 monclient: get_monmap_and_config 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.955+0000 7fa0b1c4f640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.955+0000 7fa0b1c4f640 10 monclient: monmap: 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:04:03.956692+0000 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:04:03.956692+0000 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.958 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.956+0000 7fa0b1c4f640 1 Processor -- start 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.956+0000 7fa0b1c4f640 1 -- start start 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.956+0000 7fa0b1c4f640 10 monclient: init 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient: _add_conns ranks=[5,3,6,8,4,0,2,1,7] 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient(hunting): picked mon.noname-h con 0x7fa0ac0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:04:03.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient(hunting): picked mon.noname-g con 0x7fa0ac0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient(hunting): picked mon.noname-d con 0x7fa0ac13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac0a3b40 con 0x7fa0ac0a6ba0 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac0a8130 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac06ede0 con 0x7fa0ac13f5c0 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0b1c4f640 10 monclient(hunting): authenticate will time out at 1500.033137s 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0abfff640 1 -- >> v1:172.21.15.1:6792/0 conn(0x7fa0ac13f5c0 legacy=0x7fa0ac1419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6792/0 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0ab7fe640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fa0ac0a67c0 0x7fa0ac0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:47086/0 (socket says 172.21.15.1:47086) 2024-08-19T10:04:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.957+0000 7fa0ab7fe640 1 -- 172.21.15.1:0/1560383435 learned_addr learned my addr 172.21.15.1:0/1560383435 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.959+0000 7fa0ab7fe640 1 -- 172.21.15.1:0/1560383435 >> v1:172.21.15.82:6791/0 conn(0x7fa0ac0a67c0 legacy=0x7fa0ac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.959+0000 7fa0aaffd640 1 -- 172.21.15.1:0/1560383435 >> v1:172.21.15.82:6790/0 conn(0x7fa0ac0a6ba0 legacy=0x7fa0ac0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250712772 0 0) 0x7fa0ac0a8130 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): my global_id is 27779 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa094003520 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322702287 0 0) 0x7fa0ac0a3b40 con 0x7fa0ac0a6ba0 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient(hunting): my global_id is 184539 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0ac0a8130 con 0x7fa0ac0a6ba0 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2815314775 0 0) 0x7fa094003520 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 >> v1:172.21.15.1:6792/0 conn(0x7fa0ac13f5c0 legacy=0x7fa0ac1419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 >> v1:172.21.15.82:6790/0 conn(0x7fa0ac0a6ba0 legacy=0x7fa0ac0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 monclient: found mon.noname-h 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ac0a92a0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa09c001ae0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 5 monclient: authenticate success, global_id 27779 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: dump: 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa09c0048a0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 10 monclient: get_monmap_and_config success 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 1 -- 172.21.15.1:0/1560383435 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa09c004e50 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 4 set_mon_vals no callback set 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0aa7fc640 10 monclient: dump: 2024-08-19T10:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 10 monclient: shutdown 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.960+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/1560383435 >> v1:172.21.15.82:6791/0 conn(0x7fa0ac0a67c0 legacy=0x7fa0ac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/1560383435 shutdown_connections 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/1560383435 >> 172.21.15.1:0/1560383435 conn(0x7fa0ac0ac4b0 msgr2=0x7fa0ac0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/1560383435 shutdown_connections 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/1560383435 wait complete. 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 10 monclient: build_initial_monmap 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 10 monclient: monmap: 2024-08-19T10:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 librados: starting msgr at 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 librados: starting objecter 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.961+0000 7fa0b1c4f640 1 Processor -- start 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 1 -- start start 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 1 librados: setting wanted keys 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 1 librados: calling monclient init 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient: init 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient: _reopen_session rank -1 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient: _add_conns ranks=[1,7,0,4,5,8,6,3,2] 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient(hunting): picked mon.noname-f con 0x7fa0ac0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient(hunting): picked mon.noname-i con 0x7fa0ac0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient(hunting): picked mon.noname-a con 0x7fa0ac13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac1cac90 con 0x7fa0ac13f5c0 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac13d6a0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0abfff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fa0ac13f5c0 legacy=0x7fa0ac1d61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.1:6789/0 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac13bea0 con 0x7fa0ac0a6ba0 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient(hunting): _renew_subs 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0b1c4f640 10 monclient(hunting): authenticate will time out at 1500.038632s 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0aaffd640 1 -- >> v1:172.21.15.82:6792/0 conn(0x7fa0ac0a6ba0 legacy=0x7fa0ac1d2ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6792/0 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0ab7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa0ac0a67c0 0x7fa0ac13ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:35938/0 (socket says 172.21.15.1:35938) 2024-08-19T10:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.962+0000 7fa0ab7fe640 1 -- 172.21.15.1:0/4046185014 learned_addr learned my addr 172.21.15.1:0/4046185014 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:04:03.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0ab7fe640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6789/0 conn(0x7fa0ac0a67c0 legacy=0x7fa0ac13ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 client.?.objecter ms_handle_connect 0x7fa0ac0a67c0 2024-08-19T10:04:03.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:04:03.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951354887 0 0) 0x7fa0ac13d6a0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): my global_id is 174781 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa098004ee0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2360704153 0 0) 0x7fa098004ee0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0ac13d6a0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa09c004d90 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): dump: 2024-08-19T10:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 97958052 0 0) 0x7fa0ac13d6a0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6792/0 conn(0x7fa0ac0a6ba0 legacy=0x7fa0ac1d2ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.1:6789/0 conn(0x7fa0ac13f5c0 legacy=0x7fa0ac1d61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: _finish_hunting 0 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 monclient: found mon.f 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ac13d3c0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: _finish_auth 0 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: _check_auth_tickets 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 5 monclient: authenticate success, global_id 174781 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 10 monclient: _renew_subs 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa0ac13b340 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 10 client.174781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 10 monclient: _renew_subs 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa0ac13f2e0 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0b1c4f640 1 librados: init done 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa09c001d30 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa09c005700 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa0a9ffb640 4 set_mon_vals no callback set 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.965+0000 7fa093fff640 10 monclient: dump: 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa09c011b60 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa0b1c4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa08c0065c0 con 0x7fa08c002860 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(492..492 src has 299..492) ==== 6288+0+0 (unknown 3038875124 0 0) 0x7fa09c04e890 con 0x7fa0ac0a67c0 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa093fff640 10 client.174781.objecter ms_dispatch 0x7fa0ac000b90 osd_map(492..492 src has 299..492) 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa093fff640 3 client.174781.objecter handle_osd_map got epochs [492,492] > 0 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.966+0000 7fa093fff640 3 client.174781.objecter handle_osd_map decoding full epoch 492 2024-08-19T10:04:03.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.967+0000 7fa093fff640 20 client.174781.objecter dump_active .. 0 homeless 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.967+0000 7fa0abfff640 10 monclient: get_auth_request con 0x7fa098040ca0 auth_method 0 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 client.174781.objecter ms_handle_connect 0x7fa08c002860 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 client.174781.objecter resend_mon_ops 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa0aaffd640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6791/0 conn(0x7fa08c002860 legacy=0x7fa08c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533956769 0 0) 0x7fa08c0065c0 con 0x7fa08c002860 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient(hunting): my global_id is 27788 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0ac13d6a0 con 0x7fa08c002860 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 client.174781.objecter ms_handle_connect 0x7fa098040ca0 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2615499515 0 0) 0x7fa0ac13d6a0 con 0x7fa08c002860 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa08c0065c0 con 0x7fa08c002860 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0a0003950 con 0x7fa08c002860 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 10 monclient: dump: 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.988 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 938456918 0 0) 0x7fa08c0065c0 con 0x7fa08c002860 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.969+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa08c00a160 con 0x7fa08c002860 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.970+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa0a00039b0 con 0x7fa08c002860 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.970+0000 7fa093fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.970+0000 7fa093fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.970+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6791/0 conn(0x7fa08c002860 legacy=0x7fa08c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.970+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6791/0 conn(0x7fa08c002860 legacy=0x7fa08c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.976+0000 7fa0b1c4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:04:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.976+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa08c009e50 con 0x7fa08c009110 2024-08-19T10:04:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.978+0000 7fa0aaffd640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6791/0 conn(0x7fa08c009110 legacy=0x7fa08c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:04:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.978+0000 7fa093fff640 10 client.174781.objecter ms_handle_connect 0x7fa08c009110 2024-08-19T10:04:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.978+0000 7fa093fff640 10 client.174781.objecter resend_mon_ops 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199542104 0 0) 0x7fa08c009e50 con 0x7fa08c009110 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient(hunting): _negotiate 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient(hunting): my global_id is 27797 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c0065c0 con 0x7fa08c009110 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2848947633 0 0) 0x7fa08c0065c0 con 0x7fa08c009110 2024-08-19T10:04:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa08c009e50 con 0x7fa08c009110 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0a0004150 con 0x7fa08c009110 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient: dump: 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:04:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:04:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1885780176 0 0) 0x7fa08c009e50 con 0x7fa08c009110 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa08c00c3f0 con 0x7fa08c009110 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7641 (unknown 0 0 3172966287) 0x7fa0a00043f0 con 0x7fa08c009110 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6791/0 conn(0x7fa08c009110 legacy=0x7fa08c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa093fff640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6791/0 conn(0x7fa08c009110 legacy=0x7fa08c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:04:03.996 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:04:03.996 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:04:03.996 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:04:03.996 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:04:03.996 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 120, 2024-08-19T10:04:03.996 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 850245, 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.997 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:04:03.998 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:03.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:04:04.000 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:04:04.001 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:04:04.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:04:04.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:04:04.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.007 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.010 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:04.015 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:04:04.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:04:04.017 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:04:04.017 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:04:04.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.979+0000 7fa0b1c4f640 10 librados: watch_flush enter 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 10 librados: watch_flush exit 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 20 client.174781.objecter shutdown clearing up homeless session... 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 10 client.174781.objecter successfully canceled tick 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa098040ca0 legacy=0x7fa098043160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 10 monclient: shutdown 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 >> v1:172.21.15.82:6789/0 conn(0x7fa0ac0a67c0 legacy=0x7fa0ac13ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0abfff640 1 -- 172.21.15.1:0/4046185014 reap_dead start 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0abfff640 1 -- 172.21.15.1:0/4046185014 reap_dead start 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 shutdown_connections 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 >> 172.21.15.1:0/4046185014 conn(0x7fa0ac0ac4b0 msgr2=0x7fa0ac1419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 shutdown_connections 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 -- 172.21.15.1:0/4046185014 wait complete. 2024-08-19T10:04:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:04:03.980+0000 7fa0b1c4f640 1 librados: shutdown 2024-08-19T10:04:04.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T10:04:04.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T10:05:34.021 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T10:05:34.021 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T10:05:34.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T10:05:34.027 INFO:tasks.ceph.mon.d:Started 2024-08-19T10:05:34.027 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-08-19T10:05:34.027 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T10:05:34.027 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T10:05:34.030 INFO:tasks.ceph.mon.e:Started 2024-08-19T10:05:34.030 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T10:05:34.030 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T10:05:34.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T10:05:34.032 INFO:tasks.ceph.mon.a:Started 2024-08-19T10:05:34.033 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-08-19T10:05:34.033 INFO:tasks.ceph.mon.i:Restarting daemon 2024-08-19T10:05:34.033 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-08-19T10:05:34.036 INFO:tasks.ceph.mon.i:Started 2024-08-19T10:05:34.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:05:34.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.168+0000 7fc8f68fd640 10 monclient: get_monmap_and_config 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.168+0000 7fc8f68fd640 10 monclient: build_initial_monmap 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.168+0000 7fc8f68fd640 10 monclient: monmap: 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:34.169735+0000 2024-08-19T10:05:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:34.169735+0000 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:34.172 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.169+0000 7fc8f68fd640 1 Processor -- start 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.169+0000 7fc8f68fd640 1 -- start start 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.169+0000 7fc8f68fd640 10 monclient: init 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient: _add_conns ranks=[8,0,1,7,5,6,3,2,4] 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient(hunting): picked mon.noname-e con 0x7fc8f00a6830 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f00694b0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient(hunting): picked mon.noname-f con 0x7fc8f00a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f00a3c70 con 0x7fc8f00694b0 2024-08-19T10:05:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f00a33d0 con 0x7fc8f00a46a0 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f00a36b0 con 0x7fc8f00a6830 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8effff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc8f00a6830 0x7fc8f00a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:56924/0 (socket says 172.21.15.1:56924) 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8effff640 1 -- 172.21.15.1:0/390284722 learned_addr learned my addr 172.21.15.1:0/390284722 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.170+0000 7fc8f68fd640 10 monclient(hunting): authenticate will time out at 1590.246975s 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.173+0000 7fc8effff640 1 -- 172.21.15.1:0/390284722 >> v1:172.21.15.1:6793/0 conn(0x7fc8f00a6830 legacy=0x7fc8f00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.173+0000 7fc8ef7fe640 1 -- 172.21.15.1:0/390284722 >> v1:172.21.15.1:6789/0 conn(0x7fc8f00694b0 legacy=0x7fc8f00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8f4e9a640 1 -- 172.21.15.1:0/390284722 >> v1:172.21.15.82:6789/0 conn(0x7fc8f00a46a0 legacy=0x7fc8f0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2359105240 0 0) 0x7fc8f00a33d0 con 0x7fc8f00a46a0 2024-08-19T10:05:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient(hunting): my global_id is 174817 2024-08-19T10:05:34.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8d8003520 con 0x7fc8f00a46a0 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 987719957 0 0) 0x7fc8d8003520 con 0x7fc8f00a46a0 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 >> v1:172.21.15.1:6793/0 conn(0x7fc8f00a6830 legacy=0x7fc8f00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 >> v1:172.21.15.1:6789/0 conn(0x7fc8f00694b0 legacy=0x7fc8f00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 monclient: found mon.noname-f 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.82:6789/0 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f006dbb0 con 0x7fc8f00a46a0 2024-08-19T10:05:34.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8e8001eb0 con 0x7fc8f00a46a0 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8eeffd640 10 monclient: dump: 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8f68fd640 5 monclient: authenticate success, global_id 174817 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.177+0000 7fc8f68fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8e8003990 con 0x7fc8f00a46a0 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8eeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8eeffd640 1 -- 172.21.15.1:0/390284722 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8e8003ec0 con 0x7fc8f00a46a0 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8eeffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8eeffd640 10 monclient: dump: 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8f68fd640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.178+0000 7fc8f68fd640 4 set_mon_vals no callback set 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 10 monclient: shutdown 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 1 -- 172.21.15.1:0/390284722 >> v1:172.21.15.82:6789/0 conn(0x7fc8f00a46a0 legacy=0x7fc8f0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 1 -- 172.21.15.1:0/390284722 shutdown_connections 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 1 -- 172.21.15.1:0/390284722 >> 172.21.15.1:0/390284722 conn(0x7fc8f00a9020 msgr2=0x7fc8f0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 1 -- 172.21.15.1:0/390284722 shutdown_connections 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 1 -- 172.21.15.1:0/390284722 wait complete. 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.179+0000 7fc8f68fd640 10 monclient: build_initial_monmap 2024-08-19T10:05:34.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 10 monclient: monmap: 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:34.187 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 1 librados: starting msgr at 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 1 librados: starting objecter 2024-08-19T10:05:34.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 1 Processor -- start 2024-08-19T10:05:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 1 -- start start 2024-08-19T10:05:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 1 librados: setting wanted keys 2024-08-19T10:05:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 1 librados: calling monclient init 2024-08-19T10:05:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.180+0000 7fc8f68fd640 10 monclient: init 2024-08-19T10:05:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient: _add_conns ranks=[2,7,3,4,8,6,0,1,5] 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f00694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient(hunting): picked mon.noname-i con 0x7fc8f00a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient(hunting): picked mon.noname-g con 0x7fc8f00a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f0142c40 con 0x7fc8f00694b0 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8effff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc8f00694b0 0x7fc8f0132d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:56406/0 (socket says 172.21.15.1:56406) 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8effff640 1 -- 172.21.15.1:0/626506418 learned_addr learned my addr 172.21.15.1:0/626506418 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8ef7fe640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7fc8f00a46a0 0x7fc8f01d77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:43180/0 (socket says 172.21.15.1:43180) 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f0138f00 con 0x7fc8f00a6830 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f01393c0 con 0x7fc8f00a46a0 2024-08-19T10:05:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.181+0000 7fc8f68fd640 10 monclient(hunting): authenticate will time out at 1590.257895s 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.183+0000 7fc8effff640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.1:6790/0 conn(0x7fc8f00694b0 legacy=0x7fc8f0132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.183+0000 7fc8ecff9640 10 client.?.objecter ms_handle_connect 0x7fc8f00694b0 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.183+0000 7fc8ecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.183+0000 7fc8ecff9640 10 client.?.objecter ms_handle_connect 0x7fc8f00a46a0 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.183+0000 7fc8ecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.183+0000 7fc8ef7fe640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.82:6792/0 conn(0x7fc8f00a46a0 legacy=0x7fc8f01d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 client.?.objecter ms_handle_connect 0x7fc8f00a6830 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8f4e9a640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.82:6790/0 conn(0x7fc8f00a6830 legacy=0x7fc8f01daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918114741 0 0) 0x7fc8f0138f00 con 0x7fc8f00a6830 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): my global_id is 184557 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8dc004eb0 con 0x7fc8f00a6830 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354734036 0 0) 0x7fc8f0142c40 con 0x7fc8f00694b0 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 10 monclient(hunting): my global_id is 184484 2024-08-19T10:05:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.185+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8f0138f00 con 0x7fc8f00694b0 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 235053574 0 0) 0x7fc8f0138f00 con 0x7fc8f00694b0 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8f0142c40 con 0x7fc8f00694b0 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8e0001d00 con 0x7fc8f00694b0 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): dump: 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:34.197 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3506198543 0 0) 0x7fc8dc004eb0 con 0x7fc8f00a6830 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8f0138f00 con 0x7fc8f00a6830 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8e8003e80 con 0x7fc8f00a6830 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient(hunting): dump: 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:34.199 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1184365446 0 0) 0x7fc8f0142c40 con 0x7fc8f00694b0 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.82:6792/0 conn(0x7fc8f00a46a0 legacy=0x7fc8f01d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.82:6790/0 conn(0x7fc8f00a6830 legacy=0x7fc8f01daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: _finish_hunting 0 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 monclient: found mon.b 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f006f080 con 0x7fc8f00694b0 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: _finish_auth 0 2024-08-19T10:05:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: _check_auth_tickets 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 5 monclient: authenticate success, global_id 184484 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 10 monclient: _renew_subs 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f006f360 con 0x7fc8f00694b0 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8e0004b70 con 0x7fc8f00694b0 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8e0005060 con 0x7fc8f00694b0 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 10 client.184484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ecff9640 10 monclient: dump: 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 10 monclient: _renew_subs 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8ee7fc640 4 set_mon_vals no callback set 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc8f006eab0 con 0x7fc8f00694b0 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.186+0000 7fc8f68fd640 1 librados: init done 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.187+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc8e00052b0 con 0x7fc8f00694b0 2024-08-19T10:05:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.187+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(568..568 src has 299..568) ==== 5931+0+0 (unknown 154201812 0 0) 0x7fc8e004e390 con 0x7fc8f00694b0 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.187+0000 7fc8ecff9640 10 client.184484.objecter ms_dispatch 0x7fc8f0000b90 osd_map(568..568 src has 299..568) 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.187+0000 7fc8ecff9640 3 client.184484.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.187+0000 7fc8ecff9640 3 client.184484.objecter handle_osd_map decoding full epoch 568 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.188+0000 7fc8ecff9640 20 client.184484.objecter dump_active .. 0 homeless 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.188+0000 7fc8f68fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.188+0000 7fc8f68fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.188+0000 7fc8f68fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.188+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f00a5d40 con 0x7fc8f00694b0 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.188+0000 7fc8ef7fe640 10 monclient: get_auth_request con 0x7fc8dc041590 auth_method 0 2024-08-19T10:05:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.190+0000 7fc8ecff9640 10 client.184484.objecter ms_handle_connect 0x7fc8dc041590 2024-08-19T10:05:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.197+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fc8f00a5d40 con 0x7fc8f00694b0 2024-08-19T10:05:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.197+0000 7fc8ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.197+0000 7fc8ecff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.476+0000 7fc8f68fd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:05:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.476+0000 7fc8f68fd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.476+0000 7fc8f68fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:34.476+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc8f00a33d0 con 0x7fc8f00694b0 2024-08-19T10:05:37.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:37.181+0000 7fc8d77fe640 10 monclient: tick 2024-08-19T10:05:37.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:37.181+0000 7fc8d77fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:37.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:37.181+0000 7fc8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:37.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:37.181+0000 7fc8d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:38.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:38.181+0000 7fc8d77fe640 10 monclient: tick 2024-08-19T10:05:38.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:38.181+0000 7fc8d77fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:38.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:38.181+0000 7fc8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:39.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.181+0000 7fc8d77fe640 10 monclient: tick 2024-08-19T10:05:39.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.181+0000 7fc8d77fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:39.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.181+0000 7fc8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:39.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.187+0000 7fc8ed7fa640 10 client.184484.objecter tick 2024-08-19T10:05:39.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.244+0000 7fc8ecff9640 1 -- 172.21.15.1:0/626506418 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2685 (unknown 254746657 0 2180162828) 0x7fc8f00a33d0 con 0x7fc8f00694b0 2024-08-19T10:05:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.244+0000 7fc8ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:39.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.244+0000 7fc8ecff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:39.246 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":126,"quorum":[1,2,3,4,5,7],"quorum_names":["f","b","g","c","h","i"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 10 librados: watch_flush enter 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 10 librados: watch_flush exit 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 20 client.184484.objecter shutdown clearing up homeless session... 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 10 client.184484.objecter successfully canceled tick 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc8dc041590 legacy=0x7fc8dc043a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 10 monclient: shutdown 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 >> v1:172.21.15.1:6790/0 conn(0x7fc8f00694b0 legacy=0x7fc8f0132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 shutdown_connections 2024-08-19T10:05:39.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 >> 172.21.15.1:0/626506418 conn(0x7fc8f00a9020 msgr2=0x7fc8f00a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:39.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 shutdown_connections 2024-08-19T10:05:39.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 -- 172.21.15.1:0/626506418 wait complete. 2024-08-19T10:05:39.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:39.247+0000 7fc8f68fd640 1 librados: shutdown 2024-08-19T10:05:42.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:05:42.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.375+0000 7f95085f4640 10 monclient: get_monmap_and_config 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.375+0000 7f95085f4640 10 monclient: build_initial_monmap 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.375+0000 7f95085f4640 10 monclient: monmap: 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:42.376659+0000 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:42.376659+0000 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:42.377 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:42.378 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.376+0000 7f95085f4640 1 Processor -- start 2024-08-19T10:05:42.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.376+0000 7f95085f4640 1 -- start start 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.376+0000 7f95085f4640 10 monclient: init 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.376+0000 7f95085f4640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.376+0000 7f95085f4640 10 monclient: _add_conns ranks=[4,2,0,1,5,3,6,8,7] 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.376+0000 7f95085f4640 10 monclient(hunting): picked mon.noname-c con 0x7f95000a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 10 monclient(hunting): picked mon.noname-b con 0x7f95000a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 10 monclient(hunting): picked mon.noname-a con 0x7f950013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95000a3b40 con 0x7f950013f5c0 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95000a8130 con 0x7f95000a6ba0 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f950006ede0 con 0x7f95000a67c0 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f95085f4640 10 monclient(hunting): authenticate will time out at 1598.453077s 2024-08-19T10:05:42.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f9506390640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f95000a67c0 0x7f95000a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56252/0 (socket says 172.21.15.1:56252) 2024-08-19T10:05:42.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f9506390640 1 -- 172.21.15.1:0/4250373536 learned_addr learned my addr 172.21.15.1:0/4250373536 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:42.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.377+0000 7f9505b8f640 1 --1- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6790/0 conn(0x7f95000a6ba0 0x7f95000a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51190/0 (socket says 172.21.15.1:51190) 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f9506390640 1 -- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6791/0 conn(0x7f95000a67c0 legacy=0x7f95000a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f9506b91640 1 -- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6789/0 conn(0x7f950013f5c0 legacy=0x7f95001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f9505b8f640 1 -- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6790/0 conn(0x7f95000a6ba0 legacy=0x7f95000a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956335814 0 0) 0x7f95000a3b40 con 0x7f950013f5c0 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient(hunting): _negotiate 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient(hunting): my global_id is 204102 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94ec003520 con 0x7f950013f5c0 2024-08-19T10:05:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4261907119 0 0) 0x7f94ec003520 con 0x7f950013f5c0 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6791/0 conn(0x7f95000a67c0 legacy=0x7f95000a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6790/0 conn(0x7f95000a6ba0 legacy=0x7f95000a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: _finish_hunting 0 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 monclient: found mon.noname-a 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95000a92a0 con 0x7f950013f5c0 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: _finish_auth 0 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: _check_auth_tickets 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94f8001de0 con 0x7f950013f5c0 2024-08-19T10:05:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f95085f4640 5 monclient: authenticate success, global_id 204102 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f95085f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.379+0000 7f950538e640 10 monclient: dump: 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94f8003990 con 0x7f950013f5c0 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f950538e640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f950538e640 1 -- 172.21.15.1:0/4250373536 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94f8003ec0 con 0x7f950013f5c0 2024-08-19T10:05:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f950538e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f950538e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f950538e640 10 monclient: dump: 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 4 set_mon_vals no callback set 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 10 monclient: shutdown 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 1 -- 172.21.15.1:0/4250373536 >> v1:172.21.15.1:6789/0 conn(0x7f950013f5c0 legacy=0x7f95001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 1 -- 172.21.15.1:0/4250373536 shutdown_connections 2024-08-19T10:05:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 1 -- 172.21.15.1:0/4250373536 >> 172.21.15.1:0/4250373536 conn(0x7f95000ac4b0 msgr2=0x7f95000a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 1 -- 172.21.15.1:0/4250373536 shutdown_connections 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.380+0000 7f95085f4640 1 -- 172.21.15.1:0/4250373536 wait complete. 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 10 monclient: build_initial_monmap 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 10 monclient: monmap: 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 1 librados: starting msgr at 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 1 librados: starting objecter 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 1 Processor -- start 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 1 -- start start 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 1 librados: setting wanted keys 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 1 librados: calling monclient init 2024-08-19T10:05:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.381+0000 7f95085f4640 10 monclient: init 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient: _add_conns ranks=[4,1,7,2,0,5,6,3,8] 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient(hunting): picked mon.noname-c con 0x7f95000a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient(hunting): picked mon.noname-f con 0x7f95000a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient(hunting): picked mon.noname-i con 0x7f950013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f950013e620 con 0x7f95000a6ba0 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95001cf020 con 0x7f95000a67c0 2024-08-19T10:05:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95000a9cc0 con 0x7f950013f5c0 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f95085f4640 10 monclient(hunting): authenticate will time out at 1598.458146s 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f9506390640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f95000a67c0 0x7f95001ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56256/0 (socket says 172.21.15.1:56256) 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.382+0000 7f9506390640 1 -- 172.21.15.1:0/313986745 learned_addr learned my addr 172.21.15.1:0/313986745 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f9506390640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.1:6791/0 conn(0x7f95000a67c0 legacy=0x7f95001ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f94f6ffd640 10 client.?.objecter ms_handle_connect 0x7f95000a67c0 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f94f6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f94f6ffd640 10 client.?.objecter ms_handle_connect 0x7f950013f5c0 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f9506b91640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.82:6792/0 conn(0x7f950013f5c0 legacy=0x7f95001d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f94f6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f94f6ffd640 10 client.?.objecter ms_handle_connect 0x7f95000a6ba0 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f94f6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.384+0000 7f9505b8f640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.82:6789/0 conn(0x7f95000a6ba0 legacy=0x7f95001ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.652+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 780159024 0 0) 0x7f95001cf020 con 0x7f95000a67c0 2024-08-19T10:05:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.652+0000 7f94f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.652+0000 7f94f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.652+0000 7f94f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.652+0000 7f94f6ffd640 10 monclient(hunting): my global_id is 194323 2024-08-19T10:05:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.652+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94e4004eb0 con 0x7f95000a67c0 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.653+0000 7f9505b8f640 1 --1- 172.21.15.1:0/313986745 >> v1:172.21.15.82:6789/0 conn(0x7f95000a6ba0 0x7f95001ce910 :-1 s=OPENED pgs=3716 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f95000a8130 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.654+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308075844 0 0) 0x7f95000a9cc0 con 0x7f950013f5c0 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.654+0000 7f94f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.654+0000 7f94f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.654+0000 7f94f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.654+0000 7f94f6ffd640 10 monclient(hunting): my global_id is 204109 2024-08-19T10:05:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.654+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95001cf020 con 0x7f950013f5c0 2024-08-19T10:05:42.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1354550139 0 0) 0x7f94e4004eb0 con 0x7f95000a67c0 2024-08-19T10:05:42.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95000a9cc0 con 0x7f95000a67c0 2024-08-19T10:05:42.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3150694166 0 0) 0x7f95001cf020 con 0x7f950013f5c0 2024-08-19T10:05:42.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94e4004eb0 con 0x7f950013f5c0 2024-08-19T10:05:42.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94fc001d00 con 0x7f95000a67c0 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94f8003e80 con 0x7f950013f5c0 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.835+0000 7f94f6ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:42.841 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:42.842 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 294815636 0 0) 0x7f94e4004eb0 con 0x7f950013f5c0 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.1:6791/0 conn(0x7f95000a67c0 legacy=0x7f95001ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.82:6789/0 conn(0x7f95000a6ba0 legacy=0x7f95001ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 monclient: found mon.i 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950013af40 con 0x7f950013f5c0 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 5 monclient: authenticate success, global_id 204109 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 10 monclient: _renew_subs 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575982375 0 0) 0x7f95000a8130 con 0x7f95000a6ba0 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f950013bdb0 con 0x7f950013f5c0 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94f8002030 con 0x7f950013f5c0 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 10 client.204109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f94f8001de0 con 0x7f950013f5c0 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 10 monclient: _renew_subs 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f95000aa020 con 0x7f950013f5c0 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f95085f4640 1 librados: init done 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f9504b8d640 4 set_mon_vals no callback set 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.836+0000 7f94f6ffd640 10 monclient: dump: 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f94f8011e00 con 0x7f950013f5c0 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(570..570 src has 299..570) ==== 5931+0+0 (unknown 292137540 0 0) 0x7f94f804e420 con 0x7f950013f5c0 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f95085f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f94f6ffd640 10 client.204109.objecter ms_dispatch 0x7f9500000b90 osd_map(570..570 src has 299..570) 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f94f6ffd640 3 client.204109.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f94f6ffd640 3 client.204109.objecter handle_osd_map decoding full epoch 570 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f95085f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f95085f4640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f950009f160 con 0x7f950013f5c0 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f94f6ffd640 20 client.204109.objecter dump_active .. 0 homeless 2024-08-19T10:05:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.837+0000 7f9506390640 10 monclient: get_auth_request con 0x7f94e4041510 auth_method 0 2024-08-19T10:05:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.839+0000 7f94f6ffd640 10 client.204109.objecter ms_handle_connect 0x7f94e4041510 2024-08-19T10:05:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.842+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f94f80176a0 con 0x7f950013f5c0 2024-08-19T10:05:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.842+0000 7f94f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:42.842+0000 7f94f6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:43.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.100+0000 7f95085f4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:05:43.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.100+0000 7f95085f4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:43.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.100+0000 7f95085f4640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:43.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.100+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 --> v1:172.21.15.82:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9500068f60 con 0x7f950013f5c0 2024-08-19T10:05:43.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.101+0000 7f94f6ffd640 1 -- 172.21.15.1:0/313986745 <== mon.7 v1:172.21.15.82:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 2223157765) 0x7f94f8016fa0 con 0x7f950013f5c0 2024-08-19T10:05:43.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.101+0000 7f94f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:43.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.101+0000 7f94f6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:43.102 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":130,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:05:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.102+0000 7f95085f4640 10 librados: watch_flush enter 2024-08-19T10:05:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.102+0000 7f95085f4640 10 librados: watch_flush exit 2024-08-19T10:05:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.102+0000 7f95085f4640 20 client.204109.objecter shutdown clearing up homeless session... 2024-08-19T10:05:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.102+0000 7f95085f4640 10 client.204109.objecter successfully canceled tick 2024-08-19T10:05:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.82:6812/2549104415 conn(0x7f94e4041510 legacy=0x7f94e40439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 10 monclient: shutdown 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 >> v1:172.21.15.82:6792/0 conn(0x7f950013f5c0 legacy=0x7f95001d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 shutdown_connections 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 >> 172.21.15.1:0/313986745 conn(0x7f95000ac4b0 msgr2=0x7f9500142070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 shutdown_connections 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 -- 172.21.15.1:0/313986745 wait complete. 2024-08-19T10:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.103+0000 7f95085f4640 1 librados: shutdown 2024-08-19T10:05:43.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:05:43.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:05:43.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:05:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.214+0000 7fc5f727e640 10 monclient: get_monmap_and_config 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.214+0000 7fc5f727e640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.214+0000 7fc5f727e640 10 monclient: monmap: 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:43.215553+0000 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:43.215553+0000 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.217 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 1 Processor -- start 2024-08-19T10:05:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 1 -- start start 2024-08-19T10:05:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient: init 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient: _add_conns ranks=[6,0,1,4,7,5,3,8,2] 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient(hunting): picked mon.noname-d con 0x7fc5f00a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5f00a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient(hunting): picked mon.noname-f con 0x7fc5f013f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a3b40 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a8130 con 0x7fc5f013f5c0 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f006ede0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f727e640 10 monclient(hunting): authenticate will time out at 1599.291898s 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f4819640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5f00a6ba0 0x7fc5f00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60272/0 (socket says 172.21.15.1:60272) 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.215+0000 7fc5f501a640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fc5f00a67c0 0x7fc5f00a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45076/0 (socket says 172.21.15.1:45076) 2024-08-19T10:05:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.216+0000 7fc5f4819640 1 -- 172.21.15.1:0/1071157991 learned_addr learned my addr 172.21.15.1:0/1071157991 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f501a640 1 -- 172.21.15.1:0/1071157991 >> v1:172.21.15.1:6792/0 conn(0x7fc5f00a67c0 legacy=0x7fc5f00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f4819640 1 -- 172.21.15.1:0/1071157991 >> v1:172.21.15.1:6789/0 conn(0x7fc5f00a6ba0 legacy=0x7fc5f00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f581b640 1 -- 172.21.15.1:0/1071157991 >> v1:172.21.15.82:6789/0 conn(0x7fc5f013f5c0 legacy=0x7fc5f01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050973776 0 0) 0x7fc5f006ede0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): my global_id is 204108 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5dc003520 con 0x7fc5f00a67c0 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847706696 0 0) 0x7fc5f00a3b40 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): my global_id is 204111 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f006ede0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551385621 0 0) 0x7fc5f00a8130 con 0x7fc5f013f5c0 2024-08-19T10:05:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient(hunting): my global_id is 174835 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f00a3b40 con 0x7fc5f013f5c0 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4017280588 0 0) 0x7fc5dc003520 con 0x7fc5f00a67c0 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 >> v1:172.21.15.82:6789/0 conn(0x7fc5f013f5c0 legacy=0x7fc5f01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 >> v1:172.21.15.1:6789/0 conn(0x7fc5f00a6ba0 legacy=0x7fc5f00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 monclient: found mon.noname-d 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f00a92a0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5ec001ad0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f727e640 5 monclient: authenticate success, global_id 204108 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f727e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: dump: 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f727e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5ec0047e0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 1 -- 172.21.15.1:0/1071157991 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5ec004d50 con 0x7fc5f00a67c0 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f727e640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5f727e640 4 set_mon_vals no callback set 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.218+0000 7fc5e7fff640 10 monclient: dump: 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.233 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 10 monclient: shutdown 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 1 -- 172.21.15.1:0/1071157991 >> v1:172.21.15.1:6792/0 conn(0x7fc5f00a67c0 legacy=0x7fc5f00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 1 -- 172.21.15.1:0/1071157991 shutdown_connections 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 1 -- 172.21.15.1:0/1071157991 >> 172.21.15.1:0/1071157991 conn(0x7fc5f00ac4b0 msgr2=0x7fc5f00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 1 -- 172.21.15.1:0/1071157991 shutdown_connections 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 1 -- 172.21.15.1:0/1071157991 wait complete. 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.219+0000 7fc5f727e640 10 monclient: monmap: 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 librados: starting msgr at 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 librados: starting objecter 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 Processor -- start 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 -- start start 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 librados: setting wanted keys 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 librados: calling monclient init 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient: init 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient: _add_conns ranks=[7,6,2,4,3,8,0,1,5] 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient(hunting): picked mon.noname-i con 0x7fc5f00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient(hunting): picked mon.noname-d con 0x7fc5f00a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5f013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f0139a70 con 0x7fc5f013f5c0 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a9cc0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f013bdb0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f727e640 10 monclient(hunting): authenticate will time out at 1599.296756s 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f4819640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fc5f00a6ba0 0x7fc5f01ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45092/0 (socket says 172.21.15.1:45092) 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.220+0000 7fc5f4819640 1 -- 172.21.15.1:0/2433770902 learned_addr learned my addr 172.21.15.1:0/2433770902 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.222+0000 7fc5f4819640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6792/0 conn(0x7fc5f00a6ba0 legacy=0x7fc5f01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.222+0000 7fc5f581b640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6790/0 conn(0x7fc5f013f5c0 legacy=0x7fc5f01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.222+0000 7fc5e5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5f00a6ba0 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.222+0000 7fc5e5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.222+0000 7fc5e5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5f013f5c0 2024-08-19T10:05:43.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.222+0000 7fc5e5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106262113 0 0) 0x7fc5f006ede0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): my global_id is 204117 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f501a640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.82:6792/0 conn(0x7fc5f00a67c0 legacy=0x7fc5f01ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d4004eb0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5f00a67c0 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3756689377 0 0) 0x7fc5f0139a70 con 0x7fc5f013f5c0 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): my global_id is 184493 2024-08-19T10:05:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f006ede0 con 0x7fc5f013f5c0 2024-08-19T10:05:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1649342271 0 0) 0x7fc5d4004eb0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5f0139a70 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5e00047e0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): dump: 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2494768952 0 0) 0x7fc5f006ede0 con 0x7fc5f013f5c0 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5d4004eb0 con 0x7fc5f013f5c0 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5e80020b0 con 0x7fc5f013f5c0 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): dump: 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2687124696 0 0) 0x7fc5f013bdb0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient(hunting): my global_id is 204118 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f006ede0 con 0x7fc5f00a67c0 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2388958692 0 0) 0x7fc5f0139a70 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.82:6792/0 conn(0x7fc5f00a67c0 legacy=0x7fc5f01ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6790/0 conn(0x7fc5f013f5c0 legacy=0x7fc5f01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 monclient: found mon.d 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f01cf020 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 5 monclient: authenticate success, global_id 204117 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 10 monclient: _renew_subs 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f013af40 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5e0004d60 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5e0005250 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e5ffb640 10 monclient: dump: 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5e77fe640 4 set_mon_vals no callback set 2024-08-19T10:05:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 10 client.204117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 10 monclient: _renew_subs 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5f013a880 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.223+0000 7fc5f727e640 1 librados: init done 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.224+0000 7fc5f727e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.224+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc5f00008d0 con 0x7fc5f013d1e0 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.224+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc5e00116b0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.225+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(570..570 src has 299..570) ==== 5931+0+0 (unknown 292137540 0 0) 0x7fc5e004e3b0 con 0x7fc5f00a6ba0 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.225+0000 7fc5e5ffb640 10 client.204117.objecter ms_dispatch 0x7fc5f0000b90 osd_map(570..570 src has 299..570) 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.225+0000 7fc5e5ffb640 3 client.204117.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.225+0000 7fc5e5ffb640 3 client.204117.objecter handle_osd_map decoding full epoch 570 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.225+0000 7fc5e5ffb640 20 client.204117.objecter dump_active .. 0 homeless 2024-08-19T10:05:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.225+0000 7fc5f581b640 10 monclient: get_auth_request con 0x7fc5d40417e0 auth_method 0 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 client.204117.objecter ms_handle_connect 0x7fc5f013d1e0 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5f501a640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6789/0 conn(0x7fc5f013d1e0 legacy=0x7fc5f0063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 client.204117.objecter resend_mon_ops 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265671845 0 0) 0x7fc5f00008d0 con 0x7fc5f013d1e0 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient(hunting): my global_id is 204120 2024-08-19T10:05:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f0139a70 con 0x7fc5f013d1e0 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 745202587 0 0) 0x7fc5f0139a70 con 0x7fc5f013d1e0 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5f00008d0 con 0x7fc5f013d1e0 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5ec001d20 con 0x7fc5f013d1e0 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient: dump: 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 client.204117.objecter ms_handle_connect 0x7fc5d40417e0 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2627729351 0 0) 0x7fc5f00008d0 con 0x7fc5f013d1e0 2024-08-19T10:05:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5f0064190 con 0x7fc5f013d1e0 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc5ec004890 con 0x7fc5f013d1e0 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6789/0 conn(0x7fc5f013d1e0 legacy=0x7fc5f0063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.227+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6789/0 conn(0x7fc5f013d1e0 legacy=0x7fc5f0063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.234+0000 7fc5f727e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.234+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc5f009f170 con 0x7fc5f002e340 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 client.204117.objecter ms_handle_connect 0x7fc5f002e340 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 client.204117.objecter resend_mon_ops 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5f501a640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6789/0 conn(0x7fc5f002e340 legacy=0x7fc5f009ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995839251 0 0) 0x7fc5f009f170 con 0x7fc5f002e340 2024-08-19T10:05:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient(hunting): my global_id is 204129 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f00008d0 con 0x7fc5f002e340 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2529252463 0 0) 0x7fc5f00008d0 con 0x7fc5f002e340 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5f009f170 con 0x7fc5f002e340 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc5ec005090 con 0x7fc5f002e340 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 10 monclient: dump: 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4090872358 0 0) 0x7fc5f009f170 con 0x7fc5f002e340 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.236+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5f002e720 con 0x7fc5f002e340 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1688600135) 0x7fc5ec008540 con 0x7fc5f002e340 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5e5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5e5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6789/0 conn(0x7fc5f002e340 legacy=0x7fc5f009ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5e5ffb640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6789/0 conn(0x7fc5f002e340 legacy=0x7fc5f009ea60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9107, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:43.260 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.261 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.262 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:43.263 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:43.264 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.265 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.269 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:43.271 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.272 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:43.274 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.275 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.278 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.279 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:43.280 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 10 librados: watch_flush enter 2024-08-19T10:05:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 10 librados: watch_flush exit 2024-08-19T10:05:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 20 client.204117.objecter shutdown clearing up homeless session... 2024-08-19T10:05:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 10 client.204117.objecter successfully canceled tick 2024-08-19T10:05:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc5d40417e0 legacy=0x7fc5d4043ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 10 monclient: shutdown 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 >> v1:172.21.15.1:6792/0 conn(0x7fc5f00a6ba0 legacy=0x7fc5f01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f581b640 1 -- 172.21.15.1:0/2433770902 reap_dead start 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f581b640 1 -- 172.21.15.1:0/2433770902 reap_dead start 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 shutdown_connections 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 >> 172.21.15.1:0/2433770902 conn(0x7fc5f00ac4b0 msgr2=0x7fc5f01421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 shutdown_connections 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 -- 172.21.15.1:0/2433770902 wait complete. 2024-08-19T10:05:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.237+0000 7fc5f727e640 1 librados: shutdown 2024-08-19T10:05:43.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:43.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:05:43.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:05:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.382+0000 7f86e8cf6640 10 monclient: get_monmap_and_config 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.382+0000 7f86e8cf6640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.382+0000 7f86e8cf6640 10 monclient: monmap: 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:43.383588+0000 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:43.383588+0000 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.385 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 1 Processor -- start 2024-08-19T10:05:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 1 -- start start 2024-08-19T10:05:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient: init 2024-08-19T10:05:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient: _add_conns ranks=[5,8,2,7,6,0,1,3,4] 2024-08-19T10:05:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient(hunting): picked mon.noname-h con 0x7f86e40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient(hunting): picked mon.noname-e con 0x7f86e40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f86e413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e40a3b40 con 0x7f86e413f5c0 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e40a8130 con 0x7f86e40a67c0 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e406ede0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.383+0000 7f86e8cf6640 10 monclient(hunting): authenticate will time out at 1599.459911s 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.384+0000 7f86e2d9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f86e413f5c0 0x7f86e41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51206/0 (socket says 172.21.15.1:51206) 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.384+0000 7f86e1d9b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f86e40a6ba0 0x7f86e40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54650/0 (socket says 172.21.15.1:54650) 2024-08-19T10:05:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.384+0000 7f86e2d9d640 1 -- 172.21.15.1:0/3156300083 learned_addr learned my addr 172.21.15.1:0/3156300083 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e2d9d640 1 -- 172.21.15.1:0/3156300083 >> v1:172.21.15.1:6790/0 conn(0x7f86e413f5c0 legacy=0x7f86e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e1d9b640 1 -- 172.21.15.1:0/3156300083 >> v1:172.21.15.1:6793/0 conn(0x7f86e40a6ba0 legacy=0x7f86e40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e259c640 1 -- 172.21.15.1:0/3156300083 >> v1:172.21.15.82:6791/0 conn(0x7f86e40a67c0 legacy=0x7f86e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009335260 0 0) 0x7f86e40a3b40 con 0x7f86e413f5c0 2024-08-19T10:05:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): my global_id is 184502 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86cc003520 con 0x7f86e413f5c0 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3628680230 0 0) 0x7f86e406ede0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): my global_id is 204110 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e40a3b40 con 0x7f86e40a6ba0 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944294377 0 0) 0x7f86e40a8130 con 0x7f86e40a67c0 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient(hunting): my global_id is 27806 2024-08-19T10:05:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e406ede0 con 0x7f86e40a67c0 2024-08-19T10:05:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 392202689 0 0) 0x7f86e40a3b40 con 0x7f86e40a6ba0 2024-08-19T10:05:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 >> v1:172.21.15.82:6791/0 conn(0x7f86e40a67c0 legacy=0x7f86e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 >> v1:172.21.15.1:6790/0 conn(0x7f86e413f5c0 legacy=0x7f86e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 monclient: found mon.noname-e 2024-08-19T10:05:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e40a92a0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d8001980 con 0x7f86e40a6ba0 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e8cf6640 5 monclient: authenticate success, global_id 204110 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e8cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: dump: 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.400 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e8cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86d8001c50 con 0x7f86e40a6ba0 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 1 -- 172.21.15.1:0/3156300083 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d8004b20 con 0x7f86e40a6ba0 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e8cf6640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e8cf6640 4 set_mon_vals no callback set 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.386+0000 7f86e159a640 10 monclient: dump: 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.401 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 10 monclient: shutdown 2024-08-19T10:05:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3156300083 >> v1:172.21.15.1:6793/0 conn(0x7f86e40a6ba0 legacy=0x7f86e40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3156300083 shutdown_connections 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3156300083 >> 172.21.15.1:0/3156300083 conn(0x7f86e40ac4b0 msgr2=0x7f86e40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3156300083 shutdown_connections 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.387+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3156300083 wait complete. 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient: monmap: 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 librados: starting msgr at 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 librados: starting objecter 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 Processor -- start 2024-08-19T10:05:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 -- start start 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 librados: setting wanted keys 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 librados: calling monclient init 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient: init 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient: _add_conns ranks=[4,8,1,7,6,5,0,2,3] 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f86e40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient(hunting): picked mon.noname-e con 0x7f86e40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient(hunting): picked mon.noname-f con 0x7f86e413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e4139970 con 0x7f86e413f5c0 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e413bcd0 con 0x7f86e40a67c0 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e41cf020 con 0x7f86e40a6ba0 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.388+0000 7f86e8cf6640 10 monclient(hunting): authenticate will time out at 1599.464931s 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.389+0000 7f86e1d9b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f86e40a6ba0 0x7f86e41ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54656/0 (socket says 172.21.15.1:54656) 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.389+0000 7f86e259c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f86e40a67c0 0x7f86e41ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56264/0 (socket says 172.21.15.1:56264) 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.389+0000 7f86e1d9b640 1 -- 172.21.15.1:0/3919899164 learned_addr learned my addr 172.21.15.1:0/3919899164 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e1d9b640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6793/0 conn(0x7f86e40a6ba0 legacy=0x7f86e41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e259c640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6791/0 conn(0x7f86e40a67c0 legacy=0x7f86e41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 client.?.objecter ms_handle_connect 0x7f86e40a6ba0 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 client.?.objecter ms_handle_connect 0x7f86e40a67c0 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671798310 0 0) 0x7f86e41cf020 con 0x7f86e40a6ba0 2024-08-19T10:05:43.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 204119 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86c4004eb0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993092163 0 0) 0x7f86e413bcd0 con 0x7f86e40a67c0 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 194332 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e41cf020 con 0x7f86e40a67c0 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 client.?.objecter ms_handle_connect 0x7f86e413f5c0 2024-08-19T10:05:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e2d9d640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.82:6789/0 conn(0x7f86e413f5c0 legacy=0x7f86e41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1214218517 0 0) 0x7f86c4004eb0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86e413bcd0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d8004b20 con 0x7f86e40a6ba0 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1511302214 0 0) 0x7f86e41cf020 con 0x7f86e40a67c0 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86c4004eb0 con 0x7f86e40a67c0 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d4003610 con 0x7f86e40a67c0 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244852524 0 0) 0x7f86e4139970 con 0x7f86e413f5c0 2024-08-19T10:05:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 174844 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e41cf020 con 0x7f86e413f5c0 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3883810936 0 0) 0x7f86e413bcd0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6791/0 conn(0x7f86e40a67c0 legacy=0x7f86e41ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.82:6789/0 conn(0x7f86e413f5c0 legacy=0x7f86e41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 monclient: found mon.e 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e413b9f0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 5 monclient: authenticate success, global_id 204119 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 10 monclient: _renew_subs 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86e413a8a0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 10 client.204119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 10 monclient: _renew_subs 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f86e40a26d0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e8cf6640 1 librados: init done 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86d8005150 con 0x7f86e40a6ba0 2024-08-19T10:05:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d8005640 con 0x7f86e40a6ba0 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86e0d99640 4 set_mon_vals no callback set 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.391+0000 7f86d2ffd640 10 monclient: dump: 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.392+0000 7f86e8cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.392+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86e4143070 con 0x7f86e409da50 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.392+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f86d8011aa0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.393+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(570..570 src has 299..570) ==== 5931+0+0 (unknown 292137540 0 0) 0x7f86d804e2f0 con 0x7f86e40a6ba0 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.393+0000 7f86d2ffd640 10 client.204119.objecter ms_dispatch 0x7f86e4000b90 osd_map(570..570 src has 299..570) 2024-08-19T10:05:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.393+0000 7f86d2ffd640 3 client.204119.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T10:05:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.393+0000 7f86d2ffd640 3 client.204119.objecter handle_osd_map decoding full epoch 570 2024-08-19T10:05:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.393+0000 7f86d2ffd640 20 client.204119.objecter dump_active .. 0 homeless 2024-08-19T10:05:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.393+0000 7f86e2d9d640 10 monclient: get_auth_request con 0x7f86c40416b0 auth_method 0 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86e259c640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6790/0 conn(0x7f86e409da50 legacy=0x7f86e409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 client.204119.objecter ms_handle_connect 0x7f86e409da50 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 client.204119.objecter resend_mon_ops 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136413581 0 0) 0x7f86c4004eb0 con 0x7f86e409da50 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 184511 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e413bcd0 con 0x7f86e409da50 2024-08-19T10:05:43.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 484544553 0 0) 0x7f86e413bcd0 con 0x7f86e409da50 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86c4004eb0 con 0x7f86e409da50 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d4003e10 con 0x7f86e409da50 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient: dump: 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1673559567 0 0) 0x7f86c4004eb0 con 0x7f86e409da50 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86e413b430 con 0x7f86e409da50 2024-08-19T10:05:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 client.204119.objecter ms_handle_connect 0x7f86c40416b0 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f86d4003fd0 con 0x7f86e409da50 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6790/0 conn(0x7f86e409da50 legacy=0x7f86e409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.395+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6790/0 conn(0x7f86e409da50 legacy=0x7f86e409de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.401+0000 7f86e8cf6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.401+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86e4142c60 con 0x7f86e409e540 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 client.204119.objecter ms_handle_connect 0x7f86e409e540 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 client.204119.objecter resend_mon_ops 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86e259c640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6790/0 conn(0x7f86e409e540 legacy=0x7f86e41dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2520503800 0 0) 0x7f86e4142c60 con 0x7f86e409e540 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 184520 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86c4004eb0 con 0x7f86e409e540 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2245857552 0 0) 0x7f86c4004eb0 con 0x7f86e409e540 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86e4142c60 con 0x7f86e409e540 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f86d4005b00 con 0x7f86e409e540 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient: dump: 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2373420671 0 0) 0x7f86e4142c60 con 0x7f86e409e540 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86e40ab690 con 0x7f86e409e540 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 4278476129) 0x7f86d4006fb0 con 0x7f86e409e540 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6790/0 conn(0x7f86e409e540 legacy=0x7f86e41dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.404+0000 7f86d2ffd640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6790/0 conn(0x7f86e409e540 legacy=0x7f86e41dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.426 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 260652, 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:43.427 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:43.429 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.430 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:43.431 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.432 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.433 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.434 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.435 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:43.436 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.439 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.441 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.444 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:43.446 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.448 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.450 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:43.451 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:43.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 10 librados: watch_flush enter 2024-08-19T10:05:43.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 10 librados: watch_flush exit 2024-08-19T10:05:43.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 20 client.204119.objecter shutdown clearing up homeless session... 2024-08-19T10:05:43.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 10 client.204119.objecter successfully canceled tick 2024-08-19T10:05:43.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.82:6812/2549104415 conn(0x7f86c40416b0 legacy=0x7f86c4043b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 10 monclient: shutdown 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 >> v1:172.21.15.1:6793/0 conn(0x7f86e40a6ba0 legacy=0x7f86e41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e2d9d640 1 -- 172.21.15.1:0/3919899164 reap_dead start 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e2d9d640 1 -- 172.21.15.1:0/3919899164 reap_dead start 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 shutdown_connections 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 >> 172.21.15.1:0/3919899164 conn(0x7f86e40ac4b0 msgr2=0x7f86e41419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 shutdown_connections 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 -- 172.21.15.1:0/3919899164 wait complete. 2024-08-19T10:05:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.405+0000 7f86e8cf6640 1 librados: shutdown 2024-08-19T10:05:43.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:43.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:05:43.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.553+0000 7f1d6db3a640 10 monclient: get_monmap_and_config 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.553+0000 7f1d6db3a640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.553+0000 7f1d6db3a640 10 monclient: monmap: 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:43.554813+0000 2024-08-19T10:05:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:43.554813+0000 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.557 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.554+0000 7f1d6db3a640 1 Processor -- start 2024-08-19T10:05:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.554+0000 7f1d6db3a640 1 -- start start 2024-08-19T10:05:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.554+0000 7f1d6db3a640 10 monclient: init 2024-08-19T10:05:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient: _add_conns ranks=[0,5,2,3,7,6,8,1,4] 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient(hunting): picked mon.noname-a con 0x7f1d680a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient(hunting): picked mon.noname-h con 0x7f1d680a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient(hunting): picked mon.noname-b con 0x7f1d6813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d680a3b40 con 0x7f1d680a67c0 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d680a8130 con 0x7f1d6813f5c0 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d6806ede0 con 0x7f1d680a6ba0 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d6db3a640 10 monclient(hunting): authenticate will time out at 1599.631185s 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d677fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1d680a67c0 0x7f1d680a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60290/0 (socket says 172.21.15.1:60290) 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.555+0000 7f1d677fe640 1 -- 172.21.15.1:0/2499238755 learned_addr learned my addr 172.21.15.1:0/2499238755 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d677fe640 1 -- 172.21.15.1:0/2499238755 >> v1:172.21.15.1:6789/0 conn(0x7f1d680a67c0 legacy=0x7f1d680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d67fff640 1 -- 172.21.15.1:0/2499238755 >> v1:172.21.15.1:6790/0 conn(0x7f1d6813f5c0 legacy=0x7f1d681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286769602 0 0) 0x7f1d680a3b40 con 0x7f1d680a67c0 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): my global_id is 204138 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d50003520 con 0x7f1d680a67c0 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3600835524 0 0) 0x7f1d680a8130 con 0x7f1d6813f5c0 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient(hunting): my global_id is 184529 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d680a3b40 con 0x7f1d6813f5c0 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d66ffd640 1 -- 172.21.15.1:0/2499238755 >> v1:172.21.15.82:6791/0 conn(0x7f1d680a6ba0 legacy=0x7f1d680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 559245057 0 0) 0x7f1d50003520 con 0x7f1d680a67c0 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 >> v1:172.21.15.82:6791/0 conn(0x7f1d680a6ba0 legacy=0x7f1d680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 >> v1:172.21.15.1:6790/0 conn(0x7f1d6813f5c0 legacy=0x7f1d681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 monclient: found mon.noname-a 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:43.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d680a92a0 con 0x7f1d680a67c0 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d58001ae0 con 0x7f1d680a67c0 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d6db3a640 5 monclient: authenticate success, global_id 204138 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d6db3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d667fc640 10 monclient: dump: 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.567 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.557+0000 7f1d6db3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1d580048a0 con 0x7f1d680a67c0 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d667fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d667fc640 1 -- 172.21.15.1:0/2499238755 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d58004e10 con 0x7f1d680a67c0 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d667fc640 10 monclient: dump: 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 4 set_mon_vals no callback set 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 10 monclient: shutdown 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 1 -- 172.21.15.1:0/2499238755 >> v1:172.21.15.1:6789/0 conn(0x7f1d680a67c0 legacy=0x7f1d680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 1 -- 172.21.15.1:0/2499238755 shutdown_connections 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 1 -- 172.21.15.1:0/2499238755 >> 172.21.15.1:0/2499238755 conn(0x7f1d680ac4b0 msgr2=0x7f1d680a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 1 -- 172.21.15.1:0/2499238755 shutdown_connections 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.558+0000 7f1d6db3a640 1 -- 172.21.15.1:0/2499238755 wait complete. 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 10 monclient: monmap: 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 1 librados: starting msgr at 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 1 librados: starting objecter 2024-08-19T10:05:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 1 Processor -- start 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 1 -- start start 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 1 librados: setting wanted keys 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 1 librados: calling monclient init 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.559+0000 7f1d6db3a640 10 monclient: init 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient: _add_conns ranks=[3,4,0,1,7,2,8,6,5] 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient(hunting): picked mon.noname-g con 0x7f1d680a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient(hunting): picked mon.noname-c con 0x7f1d680a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:43.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient(hunting): picked mon.noname-a con 0x7f1d6813f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d680a18a0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d68142bb0 con 0x7f1d680a67c0 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d6813a930 con 0x7f1d680a6ba0 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d6db3a640 10 monclient(hunting): authenticate will time out at 1599.636173s 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d67fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1d6813f5c0 0x7f1d681d6be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60302/0 (socket says 172.21.15.1:60302) 2024-08-19T10:05:43.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.560+0000 7f1d67fff640 1 -- 172.21.15.1:0/816062858 learned_addr learned my addr 172.21.15.1:0/816062858 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d66ffd640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d680a6ba0 legacy=0x7f1d681d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 client.?.objecter ms_handle_connect 0x7f1d680a6ba0 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d67fff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6789/0 conn(0x7f1d6813f5c0 legacy=0x7f1d681d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 client.?.objecter ms_handle_connect 0x7f1d6813f5c0 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481496603 0 0) 0x7f1d680a18a0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): my global_id is 204147 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d54004eb0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347278539 0 0) 0x7f1d6813a930 con 0x7f1d680a6ba0 2024-08-19T10:05:43.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): my global_id is 194341 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d680a18a0 con 0x7f1d680a6ba0 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d677fe640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.82:6790/0 conn(0x7f1d680a67c0 legacy=0x7f1d681ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 client.?.objecter ms_handle_connect 0x7f1d680a67c0 2024-08-19T10:05:43.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 263500617 0 0) 0x7f1d680a92a0 con 0x7f1d680a67c0 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): my global_id is 184566 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d6813a930 con 0x7f1d680a67c0 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2302033257 0 0) 0x7f1d54004eb0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d680a92a0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d60003990 con 0x7f1d6813f5c0 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:43.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): dump: 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.584 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2311396632 0 0) 0x7f1d680a18a0 con 0x7f1d680a6ba0 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d54004eb0 con 0x7f1d680a6ba0 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d5c001ce0 con 0x7f1d680a6ba0 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 10 monclient(hunting): dump: 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.562+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 208866788 0 0) 0x7f1d680a92a0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d680a6ba0 legacy=0x7f1d681d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.82:6790/0 conn(0x7f1d680a67c0 legacy=0x7f1d681ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 1 monclient: found mon.a 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6813ac10 con 0x7f1d6813f5c0 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 5 monclient: authenticate success, global_id 204147 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 10 monclient: _renew_subs 2024-08-19T10:05:43.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d680a2400 con 0x7f1d6813f5c0 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 10 client.204147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 10 monclient: _renew_subs 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d680a11a0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 1 librados: init done 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1d60003e50 con 0x7f1d6813f5c0 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d60004340 con 0x7f1d6813f5c0 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d65ffb640 4 set_mon_vals no callback set 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d4ffff640 10 monclient: dump: 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.589 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.563+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1d480065c0 con 0x7f1d48002860 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1d600044e0 con 0x7f1d6813f5c0 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(570..570 src has 299..570) ==== 5931+0+0 (unknown 292137540 0 0) 0x7f1d6004e740 con 0x7f1d6813f5c0 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d4ffff640 10 client.204147.objecter ms_dispatch 0x7f1d68000b90 osd_map(570..570 src has 299..570) 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d4ffff640 3 client.204147.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T10:05:43.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d4ffff640 3 client.204147.objecter handle_osd_map decoding full epoch 570 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d4ffff640 20 client.204147.objecter dump_active .. 0 homeless 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.564+0000 7f1d66ffd640 10 monclient: get_auth_request con 0x7f1d54041430 auth_method 0 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 client.204147.objecter ms_handle_connect 0x7f1d48002860 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 client.204147.objecter resend_mon_ops 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d677fe640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d48002860 legacy=0x7f1d48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997293463 0 0) 0x7f1d480065c0 con 0x7f1d48002860 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient(hunting): my global_id is 194350 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d680a92a0 con 0x7f1d48002860 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 101224895 0 0) 0x7f1d680a92a0 con 0x7f1d48002860 2024-08-19T10:05:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d480065c0 con 0x7f1d48002860 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d58001320 con 0x7f1d48002860 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient: dump: 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.592 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2349568758 0 0) 0x7f1d480065c0 con 0x7f1d48002860 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d4800a160 con 0x7f1d48002860 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 client.204147.objecter ms_handle_connect 0x7f1d54041430 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1d58005490 con 0x7f1d48002860 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d48002860 legacy=0x7f1d48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.566+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d48002860 legacy=0x7f1d48004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.573+0000 7f1d6db3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.573+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1d48009e70 con 0x7f1d48009130 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 client.204147.objecter ms_handle_connect 0x7f1d48009130 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d677fe640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d48009130 legacy=0x7f1d480095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 client.204147.objecter resend_mon_ops 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252693463 0 0) 0x7f1d48009e70 con 0x7f1d48009130 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient(hunting): my global_id is 194359 2024-08-19T10:05:43.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d480065c0 con 0x7f1d48009130 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3814687528 0 0) 0x7f1d480065c0 con 0x7f1d48009130 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d48009e70 con 0x7f1d48009130 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1d58005c00 con 0x7f1d48009130 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 10 monclient: dump: 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.595 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.575+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2496936416 0 0) 0x7f1d48009e70 con 0x7f1d48009130 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d4800c3f0 con 0x7f1d48009130 2024-08-19T10:05:43.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2051221256) 0x7f1d58007190 con 0x7f1d48009130 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d4ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d48009130 legacy=0x7f1d480095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d4ffff640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6791/0 conn(0x7f1d48009130 legacy=0x7f1d480095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:43.597 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 131263, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:43.598 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.600 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:43.601 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.602 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.603 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:43.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:43.605 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.606 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.608 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:43.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.615 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:43.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:43.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:43.619 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 10 librados: watch_flush enter 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 10 librados: watch_flush exit 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 20 client.204147.objecter shutdown clearing up homeless session... 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 10 client.204147.objecter successfully canceled tick 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1d54041430 legacy=0x7f1d540438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 10 monclient: shutdown 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 >> v1:172.21.15.1:6789/0 conn(0x7f1d6813f5c0 legacy=0x7f1d681d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d67fff640 1 -- 172.21.15.1:0/816062858 reap_dead start 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d67fff640 1 -- 172.21.15.1:0/816062858 reap_dead start 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 shutdown_connections 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 >> 172.21.15.1:0/816062858 conn(0x7f1d680ac4b0 msgr2=0x7f1d68138980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 shutdown_connections 2024-08-19T10:05:43.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 -- 172.21.15.1:0/816062858 wait complete. 2024-08-19T10:05:43.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.576+0000 7f1d6db3a640 1 librados: shutdown 2024-08-19T10:05:43.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:43.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:05:43.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:05:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.722+0000 7f7472543640 10 monclient: get_monmap_and_config 2024-08-19T10:05:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.722+0000 7f7472543640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.722+0000 7f7472543640 10 monclient: monmap: 2024-08-19T10:05:43.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:43.723336+0000 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:43.723336+0000 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.725 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 1 Processor -- start 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 1 -- start start 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient: init 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient: _add_conns ranks=[5,7,0,8,6,1,3,4,2] 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient(hunting): picked mon.noname-h con 0x7f746c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient(hunting): picked mon.noname-i con 0x7f746c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient(hunting): picked mon.noname-a con 0x7f746c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:43.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0a3b40 con 0x7f746c13f5c0 2024-08-19T10:05:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0a8130 con 0x7f746c0a67c0 2024-08-19T10:05:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c06ede0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7472543640 10 monclient(hunting): authenticate will time out at 1599.799688s 2024-08-19T10:05:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7470ae0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f746c13f5c0 0x7f746c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60314/0 (socket says 172.21.15.1:60314) 2024-08-19T10:05:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.723+0000 7f7470ae0640 1 -- 172.21.15.1:0/3023875320 learned_addr learned my addr 172.21.15.1:0/3023875320 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f7470ae0640 1 -- 172.21.15.1:0/3023875320 >> v1:172.21.15.1:6789/0 conn(0x7f746c13f5c0 legacy=0x7f746c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746b7fe640 1 -- 172.21.15.1:0/3023875320 >> v1:172.21.15.82:6792/0 conn(0x7f746c0a6ba0 legacy=0x7f746c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 395493763 0 0) 0x7f746c0a3b40 con 0x7f746c13f5c0 2024-08-19T10:05:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746bfff640 1 -- 172.21.15.1:0/3023875320 >> v1:172.21.15.82:6791/0 conn(0x7f746c0a67c0 legacy=0x7f746c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient(hunting): my global_id is 204156 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7454003520 con 0x7f746c13f5c0 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2092568964 0 0) 0x7f7454003520 con 0x7f746c13f5c0 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 >> v1:172.21.15.82:6792/0 conn(0x7f746c0a6ba0 legacy=0x7f746c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 >> v1:172.21.15.82:6791/0 conn(0x7f746c0a67c0 legacy=0x7f746c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 monclient: found mon.noname-a 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0a92a0 con 0x7f746c13f5c0 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f7472543640 5 monclient: authenticate success, global_id 204156 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7464001e60 con 0x7f746c13f5c0 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f7472543640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: dump: 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f7472543640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7464004c20 con 0x7f746c13f5c0 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 1 -- 172.21.15.1:0/3023875320 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7464005150 con 0x7f746c13f5c0 2024-08-19T10:05:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f7472543640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f7472543640 4 set_mon_vals no callback set 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.726+0000 7f746affd640 10 monclient: dump: 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 10 monclient: shutdown 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 -- 172.21.15.1:0/3023875320 >> v1:172.21.15.1:6789/0 conn(0x7f746c13f5c0 legacy=0x7f746c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 -- 172.21.15.1:0/3023875320 shutdown_connections 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 -- 172.21.15.1:0/3023875320 >> 172.21.15.1:0/3023875320 conn(0x7f746c0ac4b0 msgr2=0x7f746c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 -- 172.21.15.1:0/3023875320 shutdown_connections 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 -- 172.21.15.1:0/3023875320 wait complete. 2024-08-19T10:05:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 10 monclient: monmap: 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 librados: starting msgr at 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.727+0000 7f7472543640 1 librados: starting objecter 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 Processor -- start 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 -- start start 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 librados: setting wanted keys 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 librados: calling monclient init 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient: init 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient: _add_conns ranks=[3,2,8,0,6,7,5,1,4] 2024-08-19T10:05:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient(hunting): picked mon.noname-g con 0x7f746c0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient(hunting): picked mon.noname-b con 0x7f746c0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient(hunting): picked mon.noname-e con 0x7f746c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c13adf0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c139900 con 0x7f746c0a67c0 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c13bc60 con 0x7f746c13f5c0 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7472543640 10 monclient(hunting): authenticate will time out at 1599.804743s 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7470ae0640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f746c13f5c0 0x7f746c1d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54662/0 (socket says 172.21.15.1:54662) 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f746b7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f746c0a6ba0 0x7f746c13c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51230/0 (socket says 172.21.15.1:51230) 2024-08-19T10:05:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.728+0000 7f7470ae0640 1 -- 172.21.15.1:0/385244053 learned_addr learned my addr 172.21.15.1:0/385244053 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.730+0000 7f746b7fe640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6790/0 conn(0x7f746c0a6ba0 legacy=0x7f746c13c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.730+0000 7f7468ff9640 10 client.?.objecter ms_handle_connect 0x7f746c0a6ba0 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.730+0000 7f7468ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.730+0000 7f7470ae0640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6793/0 conn(0x7f746c13f5c0 legacy=0x7f746c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.730+0000 7f7468ff9640 10 client.?.objecter ms_handle_connect 0x7f746c13f5c0 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.730+0000 7f7468ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 client.?.objecter ms_handle_connect 0x7f746c0a67c0 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f746bfff640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.82:6790/0 conn(0x7f746c0a67c0 legacy=0x7f746c13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246213339 0 0) 0x7f746c13adf0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): my global_id is 184538 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7458004eb0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021703413 0 0) 0x7f746c13bc60 con 0x7f746c13f5c0 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): my global_id is 204128 2024-08-19T10:05:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c13adf0 con 0x7f746c13f5c0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4278151495 0 0) 0x7f746c139900 con 0x7f746c0a67c0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): my global_id is 184575 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c13bc60 con 0x7f746c0a67c0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2882466780 0 0) 0x7f7458004eb0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f746c139900 con 0x7f746c0a6ba0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7460001d20 con 0x7f746c0a6ba0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:05:43.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): dump: 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.749 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2607266893 0 0) 0x7f746c13adf0 con 0x7f746c13f5c0 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7458004eb0 con 0x7f746c13f5c0 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f74640020b0 con 0x7f746c13f5c0 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): dump: 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.750 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 297757666 0 0) 0x7f746c13bc60 con 0x7f746c0a67c0 2024-08-19T10:05:43.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f746c13adf0 con 0x7f746c0a67c0 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f745c001cd0 con 0x7f746c0a67c0 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient(hunting): dump: 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.752 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1700519247 0 0) 0x7f746c139900 con 0x7f746c0a6ba0 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6793/0 conn(0x7f746c13f5c0 legacy=0x7f746c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.82:6790/0 conn(0x7f746c0a67c0 legacy=0x7f746c13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 monclient: found mon.b 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c13b3c0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 5 monclient: authenticate success, global_id 184538 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 10 monclient: _renew_subs 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f746c13a6d0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 10 client.184538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 10 monclient: _renew_subs 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f746c13e6b0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7472543640 1 librados: init done 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7460003a90 con 0x7f746c0a6ba0 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7460003fc0 con 0x7f746c0a6ba0 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f746a7fc640 4 set_mon_vals no callback set 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.731+0000 7f7468ff9640 10 monclient: dump: 2024-08-19T10:05:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.732+0000 7f7472543640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.732+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f744c0065c0 con 0x7f744c002860 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.732+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7460004160 con 0x7f746c0a6ba0 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.733+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(570..570 src has 299..570) ==== 5931+0+0 (unknown 292137540 0 0) 0x7f746004e460 con 0x7f746c0a6ba0 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.733+0000 7f7468ff9640 10 client.184538.objecter ms_dispatch 0x7f746c000b90 osd_map(570..570 src has 299..570) 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.733+0000 7f7468ff9640 3 client.184538.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.733+0000 7f7468ff9640 3 client.184538.objecter handle_osd_map decoding full epoch 570 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.733+0000 7f7468ff9640 20 client.184538.objecter dump_active .. 0 homeless 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.733+0000 7f7470ae0640 10 monclient: get_auth_request con 0x7f7458041a80 auth_method 0 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f746bfff640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6792/0 conn(0x7f744c002860 legacy=0x7f744c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 10 client.184538.objecter ms_handle_connect 0x7f744c002860 2024-08-19T10:05:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 10 client.184538.objecter resend_mon_ops 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617451916 0 0) 0x7f744c0065c0 con 0x7f744c002860 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.734+0000 7f7468ff9640 10 monclient(hunting): my global_id is 204126 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c139900 con 0x7f744c002860 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3996124434 0 0) 0x7f746c139900 con 0x7f744c002860 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f744c0065c0 con 0x7f744c002860 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f745c003cd0 con 0x7f744c002860 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 10 monclient: dump: 2024-08-19T10:05:43.758 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3736523644 0 0) 0x7f744c0065c0 con 0x7f744c002860 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f744c00a160 con 0x7f744c002860 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 10 client.184538.objecter ms_handle_connect 0x7f7458041a80 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f745c003f80 con 0x7f744c002860 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6792/0 conn(0x7f744c002860 legacy=0x7f744c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.735+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6792/0 conn(0x7f744c002860 legacy=0x7f744c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.741+0000 7f7472543640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:43.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.741+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f744c009e50 con 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f746bfff640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6792/0 conn(0x7f744c009110 legacy=0x7f744c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 client.184538.objecter ms_handle_connect 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 client.184538.objecter resend_mon_ops 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980138173 0 0) 0x7f744c009e50 con 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient(hunting): my global_id is 204135 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f744c0065c0 con 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4001165482 0 0) 0x7f744c0065c0 con 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f744c009e50 con 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f745c004430 con 0x7f744c009110 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 10 monclient: dump: 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 207894552 0 0) 0x7f744c009e50 con 0x7f744c009110 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.744+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f744c00c3a0 con 0x7f744c009110 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 2979265173) 0x7f745c007b80 con 0x7f744c009110 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7468ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7468ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6792/0 conn(0x7f744c009110 legacy=0x7f744c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7468ff9640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6792/0 conn(0x7f744c009110 legacy=0x7f744c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9614, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:43.783 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 10 librados: watch_flush enter 2024-08-19T10:05:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 10 librados: watch_flush exit 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 20 client.184538.objecter shutdown clearing up homeless session... 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 10 client.184538.objecter successfully canceled tick 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7458041a80 legacy=0x7f7458043f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 10 monclient: shutdown 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 >> v1:172.21.15.1:6790/0 conn(0x7f746c0a6ba0 legacy=0x7f746c13c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7470ae0640 1 -- 172.21.15.1:0/385244053 reap_dead start 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7470ae0640 1 -- 172.21.15.1:0/385244053 reap_dead start 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 shutdown_connections 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 >> 172.21.15.1:0/385244053 conn(0x7f746c0ac4b0 msgr2=0x7f746c13ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 shutdown_connections 2024-08-19T10:05:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 -- 172.21.15.1:0/385244053 wait complete. 2024-08-19T10:05:43.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.745+0000 7f7472543640 1 librados: shutdown 2024-08-19T10:05:43.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:43.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:05:43.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.891+0000 7f1cd0432640 10 monclient: get_monmap_and_config 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.891+0000 7f1cd0432640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.891+0000 7f1cd0432640 10 monclient: monmap: 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:43.892857+0000 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:43.892857+0000 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.894 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.892+0000 7f1cd0432640 1 Processor -- start 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.892+0000 7f1cd0432640 1 -- start start 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.892+0000 7f1cd0432640 10 monclient: init 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient: _add_conns ranks=[5,0,7,3,6,4,8,2,1] 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient(hunting): picked mon.noname-h con 0x7f1cc8139580 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc80a4500 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient(hunting): picked mon.noname-i con 0x7f1cc806f250 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc80a77d0 con 0x7f1cc80a4500 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc80a3330 con 0x7f1cc8139580 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc80a3bb0 con 0x7f1cc806f250 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1cd0432640 10 monclient(hunting): authenticate will time out at 1599.969251s 2024-08-19T10:05:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1ccd9cd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1cc80a4500 0x7f1cc813e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60318/0 (socket says 172.21.15.1:60318) 2024-08-19T10:05:43.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.893+0000 7f1ccd9cd640 1 -- 172.21.15.1:0/1664624008 learned_addr learned my addr 172.21.15.1:0/1664624008 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd9cd640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.1:6789/0 conn(0x7f1cc80a4500 legacy=0x7f1cc813e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859373065 0 0) 0x7f1cc80a77d0 con 0x7f1cc80a4500 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient(hunting): my global_id is 204165 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cb0003520 con 0x7f1cc80a4500 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cce1ce640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.82:6791/0 conn(0x7f1cc8139580 legacy=0x7f1cc8139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cce1ce640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.82:6791/0 conn(0x7f1cc8139580 legacy=0x7f1cc8139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cce1ce640 1 --1- 172.21.15.1:0/1664624008 >> v1:172.21.15.82:6791/0 conn(0x7f1cc8139580 0x7f1cc8139960 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cce9cf640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.82:6792/0 conn(0x7f1cc806f250 legacy=0x7f1cc8141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 832413147 0 0) 0x7f1cb0003520 con 0x7f1cc80a4500 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.82:6792/0 conn(0x7f1cc806f250 legacy=0x7f1cc8141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.82:6791/0 conn(0x7f1cc8139580 legacy=0x7f1cc8139960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 monclient: found mon.noname-a 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc80a95b0 con 0x7f1cc80a4500 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cbc001ae0 con 0x7f1cc80a4500 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cd0432640 5 monclient: authenticate success, global_id 204165 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cd0432640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1ccd1cc640 10 monclient: dump: 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.895+0000 7f1cd0432640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1cbc0048a0 con 0x7f1cc80a4500 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1ccd1cc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1ccd1cc640 1 -- 172.21.15.1:0/1664624008 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cbc004e10 con 0x7f1cc80a4500 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1ccd1cc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1ccd1cc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1ccd1cc640 10 monclient: dump: 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.906 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 4 set_mon_vals no callback set 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 10 monclient: shutdown 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 1 -- 172.21.15.1:0/1664624008 >> v1:172.21.15.1:6789/0 conn(0x7f1cc80a4500 legacy=0x7f1cc813e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 1 -- 172.21.15.1:0/1664624008 shutdown_connections 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 1 -- 172.21.15.1:0/1664624008 >> 172.21.15.1:0/1664624008 conn(0x7f1cc80ac5d0 msgr2=0x7f1cc80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 1 -- 172.21.15.1:0/1664624008 shutdown_connections 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.896+0000 7f1cd0432640 1 -- 172.21.15.1:0/1664624008 wait complete. 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 10 monclient: build_initial_monmap 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 10 monclient: monmap: 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:43.908 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 1 librados: starting msgr at 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 1 librados: starting objecter 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 1 Processor -- start 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 1 -- start start 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 1 librados: setting wanted keys 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 1 librados: calling monclient init 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 10 monclient: init 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 10 monclient: _add_conns ranks=[2,1,8,0,6,3,5,4,7] 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.897+0000 7f1cd0432640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc806f250 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 10 monclient(hunting): picked mon.noname-f con 0x7f1cc8139580 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 10 monclient(hunting): picked mon.noname-e con 0x7f1cc81cb000 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc80ab650 con 0x7f1cc8139580 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc8140720 con 0x7f1cc806f250 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc8140a00 con 0x7f1cc81cb000 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cd0432640 10 monclient(hunting): authenticate will time out at 1599.974089s 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cce1ce640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f1cc806f250 0x7f1cc81ca8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51246/0 (socket says 172.21.15.1:51246) 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.898+0000 7f1cce1ce640 1 -- 172.21.15.1:0/3407560534 learned_addr learned my addr 172.21.15.1:0/3407560534 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.900+0000 7f1cce1ce640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6790/0 conn(0x7f1cc806f250 legacy=0x7f1cc81ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.900+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.900+0000 7f1cce9cf640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1cc81cb000 legacy=0x7f1cc81d6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.900+0000 7f1ccd9cd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.82:6789/0 conn(0x7f1cc8139580 legacy=0x7f1cc81d2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 client.?.objecter ms_handle_connect 0x7f1cc806f250 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 client.?.objecter ms_handle_connect 0x7f1cc81cb000 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 client.?.objecter ms_handle_connect 0x7f1cc8139580 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964323305 0 0) 0x7f1cc8140720 con 0x7f1cc806f250 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): my global_id is 184547 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cb4004eb0 con 0x7f1cc806f250 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287979351 0 0) 0x7f1cc80a95b0 con 0x7f1cc8139580 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): my global_id is 174853 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cc8140720 con 0x7f1cc8139580 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3469661067 0 0) 0x7f1cc8140a00 con 0x7f1cc81cb000 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): my global_id is 204137 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cc80a95b0 con 0x7f1cc81cb000 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3834029180 0 0) 0x7f1cc80a95b0 con 0x7f1cc81cb000 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cc8140a00 con 0x7f1cc81cb000 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cb8001fe0 con 0x7f1cc81cb000 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1956949177 0 0) 0x7f1cb4004eb0 con 0x7f1cc806f250 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cc80a95b0 con 0x7f1cc806f250 2024-08-19T10:05:43.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cc0000ed0 con 0x7f1cc806f250 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.921 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2139356785 0 0) 0x7f1cc8140720 con 0x7f1cc8139580 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cb4004eb0 con 0x7f1cc8139580 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cbc004d90 con 0x7f1cc8139580 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3765870216 0 0) 0x7f1cc8140a00 con 0x7f1cc81cb000 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6790/0 conn(0x7f1cc806f250 legacy=0x7f1cc81ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.82:6789/0 conn(0x7f1cc8139580 legacy=0x7f1cc81d2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 monclient: found mon.e 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc80ab0e0 con 0x7f1cc81cb000 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:43.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cd0432640 5 monclient: authenticate success, global_id 204137 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cd0432640 10 monclient: _renew_subs 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cd0432640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.902+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc80a7230 con 0x7f1cc81cb000 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 10 client.204137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 10 monclient: _renew_subs 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f1cc8140440 con 0x7f1cc81cb000 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 1 librados: init done 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1cb8003c40 con 0x7f1cc81cb000 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cb80041b0 con 0x7f1cc81cb000 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1ccc9cb640 4 set_mon_vals no callback set 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 10 monclient: dump: 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1ca80065c0 con 0x7f1ca8002860 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.903+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f1cb8004350 con 0x7f1cc81cb000 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.904+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(571..571 src has 299..571) ==== 5931+0+0 (unknown 491063308 0 0) 0x7f1cb804e5f0 con 0x7f1cc81cb000 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.904+0000 7f1cc6ffd640 10 client.204137.objecter ms_dispatch 0x7f1cc8000b90 osd_map(571..571 src has 299..571) 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.904+0000 7f1cc6ffd640 3 client.204137.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T10:05:43.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.904+0000 7f1cc6ffd640 3 client.204137.objecter handle_osd_map decoding full epoch 571 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.904+0000 7f1cc6ffd640 20 client.204137.objecter dump_active .. 0 homeless 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.904+0000 7f1ccd9cd640 10 monclient: get_auth_request con 0x7f1cb40418e0 auth_method 0 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.905+0000 7f1cc6ffd640 10 client.204137.objecter ms_handle_connect 0x7f1ca8002860 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.905+0000 7f1cce1ce640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1ca8002860 legacy=0x7f1ca8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 client.204137.objecter resend_mon_ops 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317963732 0 0) 0x7f1ca80065c0 con 0x7f1ca8002860 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient(hunting): my global_id is 204146 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cc8140a00 con 0x7f1ca8002860 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2628223785 0 0) 0x7f1cc8140a00 con 0x7f1ca8002860 2024-08-19T10:05:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ca80065c0 con 0x7f1ca8002860 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cc0001980 con 0x7f1ca8002860 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient: dump: 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.929 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4235636928 0 0) 0x7f1ca80065c0 con 0x7f1ca8002860 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ca800a160 con 0x7f1ca8002860 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 client.204137.objecter ms_handle_connect 0x7f1cb40418e0 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1cc0001cb0 con 0x7f1ca8002860 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1ca8002860 legacy=0x7f1ca8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.906+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1ca8002860 legacy=0x7f1ca8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.912+0000 7f1cd0432640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.913+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1ca8009e70 con 0x7f1ca8009130 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cce1ce640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1ca8009130 legacy=0x7f1ca80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 client.204137.objecter ms_handle_connect 0x7f1ca8009130 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 client.204137.objecter resend_mon_ops 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221756173 0 0) 0x7f1ca8009e70 con 0x7f1ca8009130 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient(hunting): my global_id is 204155 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ca80065c0 con 0x7f1ca8009130 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3549608958 0 0) 0x7f1ca80065c0 con 0x7f1ca8009130 2024-08-19T10:05:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ca8009e70 con 0x7f1ca8009130 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1cc0005cf0 con 0x7f1ca8009130 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient: dump: 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1132302784 0 0) 0x7f1ca8009e70 con 0x7f1ca8009130 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ca800c410 con 0x7f1ca8009130 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 1641220222) 0x7f1cc0008070 con 0x7f1ca8009130 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1ca8009130 legacy=0x7f1ca80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.915+0000 7f1cc6ffd640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1ca8009130 legacy=0x7f1ca80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9787, 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:43.935 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:43.936 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:43.938 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:43.939 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.940 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:43.941 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.942 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.943 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:43.945 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.946 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.947 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.948 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.949 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:43.950 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.951 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.952 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:43.953 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 10 librados: watch_flush enter 2024-08-19T10:05:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 10 librados: watch_flush exit 2024-08-19T10:05:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 20 client.204137.objecter shutdown clearing up homeless session... 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 10 client.204137.objecter successfully canceled tick 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1cb40418e0 legacy=0x7f1cb4043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 10 monclient: shutdown 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 >> v1:172.21.15.1:6793/0 conn(0x7f1cc81cb000 legacy=0x7f1cc81d6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cce9cf640 1 -- 172.21.15.1:0/3407560534 reap_dead start 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cce9cf640 1 -- 172.21.15.1:0/3407560534 reap_dead start 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 shutdown_connections 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 >> 172.21.15.1:0/3407560534 conn(0x7f1cc80ac5d0 msgr2=0x7f1cc8142ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 shutdown_connections 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 -- 172.21.15.1:0/3407560534 wait complete. 2024-08-19T10:05:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:43.916+0000 7f1cd0432640 1 librados: shutdown 2024-08-19T10:05:43.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:43.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:05:43.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:05:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.056+0000 7fc0fd660640 10 monclient: get_monmap_and_config 2024-08-19T10:05:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.056+0000 7fc0fd660640 10 monclient: build_initial_monmap 2024-08-19T10:05:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.056+0000 7fc0fd660640 10 monclient: monmap: 2024-08-19T10:05:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:44.057223+0000 2024-08-19T10:05:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:44.057223+0000 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:44.062 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.056+0000 7fc0fd660640 1 Processor -- start 2024-08-19T10:05:44.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.056+0000 7fc0fd660640 1 -- start start 2024-08-19T10:05:44.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.056+0000 7fc0fd660640 10 monclient: init 2024-08-19T10:05:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:44.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient: _add_conns ranks=[3,2,1,6,4,0,8,5,7] 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient(hunting): picked mon.noname-g con 0x7fc0f80a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient(hunting): picked mon.noname-b con 0x7fc0f80a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient(hunting): picked mon.noname-f con 0x7fc0f813f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f80a3b40 con 0x7fc0f813f5c0 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f80a8130 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f806ede0 con 0x7fc0f80a67c0 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0fd660640 10 monclient(hunting): authenticate will time out at 1600.133438s 2024-08-19T10:05:44.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0f67fc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc0f80a6ba0 0x7fc0f80a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51258/0 (socket says 172.21.15.1:51258) 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.057+0000 7fc0f67fc640 1 -- 172.21.15.1:0/799108872 learned_addr learned my addr 172.21.15.1:0/799108872 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f67fc640 1 -- 172.21.15.1:0/799108872 >> v1:172.21.15.1:6790/0 conn(0x7fc0f80a6ba0 legacy=0x7fc0f80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555816940 0 0) 0x7fc0f80a8130 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f5ffb640 10 monclient(hunting): my global_id is 184556 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0e4003520 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.059+0000 7fc0f77fe640 1 -- 172.21.15.1:0/799108872 >> v1:172.21.15.82:6789/0 conn(0x7fc0f813f5c0 legacy=0x7fc0f81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f6ffd640 1 -- 172.21.15.1:0/799108872 >> v1:172.21.15.82:6790/0 conn(0x7fc0f80a67c0 legacy=0x7fc0f80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 143653946 0 0) 0x7fc0e4003520 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 >> v1:172.21.15.82:6790/0 conn(0x7fc0f80a67c0 legacy=0x7fc0f80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 >> v1:172.21.15.82:6789/0 conn(0x7fc0f813f5c0 legacy=0x7fc0f81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 monclient: found mon.noname-b 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f80a92a0 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: _finish_auth 0 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 5 monclient: authenticate success, global_id 184556 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e0001ae0 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: dump: 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0e00048a0 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 1 -- 172.21.15.1:0/799108872 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e0004dd0 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 4 set_mon_vals no callback set 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0f5ffb640 10 monclient: dump: 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 10 monclient: shutdown 2024-08-19T10:05:44.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 1 -- 172.21.15.1:0/799108872 >> v1:172.21.15.1:6790/0 conn(0x7fc0f80a6ba0 legacy=0x7fc0f80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 1 -- 172.21.15.1:0/799108872 shutdown_connections 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 1 -- 172.21.15.1:0/799108872 >> 172.21.15.1:0/799108872 conn(0x7fc0f80ac4b0 msgr2=0x7fc0f80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 1 -- 172.21.15.1:0/799108872 shutdown_connections 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.060+0000 7fc0fd660640 1 -- 172.21.15.1:0/799108872 wait complete. 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 10 monclient: build_initial_monmap 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 10 monclient: monmap: 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.080 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 1 librados: starting msgr at 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 1 librados: starting objecter 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 1 Processor -- start 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 1 -- start start 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 1 librados: setting wanted keys 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 1 librados: calling monclient init 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.061+0000 7fc0fd660640 10 monclient: init 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient: _add_conns ranks=[0,1,5,3,2,4,6,7,8] 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient(hunting): picked mon.noname-a con 0x7fc0f80a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient(hunting): picked mon.noname-f con 0x7fc0f80a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:44.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient(hunting): picked mon.noname-h con 0x7fc0f813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f813af70 con 0x7fc0f80a67c0 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f813e650 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f813b540 con 0x7fc0f813f5c0 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0fd660640 10 monclient(hunting): authenticate will time out at 1600.138439s 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0f6ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc0f80a67c0 0x7fc0f813c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60322/0 (socket says 172.21.15.1:60322) 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.062+0000 7fc0f6ffd640 1 -- 172.21.15.1:0/2449025465 learned_addr learned my addr 172.21.15.1:0/2449025465 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0f6ffd640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.1:6789/0 conn(0x7fc0f80a67c0 legacy=0x7fc0f813c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc0f80a67c0 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0ef7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0f67fc640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0f80a6ba0 legacy=0x7fc0f813c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc0f813f5c0 2024-08-19T10:05:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0ef7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc0f80a6ba0 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0ef7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.064+0000 7fc0f77fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6791/0 conn(0x7fc0f813f5c0 legacy=0x7fc0f81d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3705232371 0 0) 0x7fc0f813e650 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): my global_id is 174871 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0d8004eb0 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212563839 0 0) 0x7fc0f813af70 con 0x7fc0f80a67c0 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): my global_id is 204174 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0f813e650 con 0x7fc0f80a67c0 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 194165590 0 0) 0x7fc0f813e650 con 0x7fc0f80a67c0 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0f813af70 con 0x7fc0f80a67c0 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e8003990 con 0x7fc0f80a67c0 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): dump: 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 422231166 0 0) 0x7fc0d8004eb0 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0f813e650 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e0004d90 con 0x7fc0f80a6ba0 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient(hunting): dump: 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 587450729 0 0) 0x7fc0f813af70 con 0x7fc0f80a67c0 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6791/0 conn(0x7fc0f813f5c0 legacy=0x7fc0f81d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0f80a6ba0 legacy=0x7fc0f813c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 monclient: found mon.a 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f80a2840 con 0x7fc0f80a67c0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: _finish_auth 0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 5 monclient: authenticate success, global_id 204174 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 10 monclient: _renew_subs 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f813d090 con 0x7fc0f80a67c0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 10 client.204174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 10 monclient: _renew_subs 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0f813d470 con 0x7fc0f80a67c0 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0fd660640 1 librados: init done 2024-08-19T10:05:44.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0e8003ec0 con 0x7fc0f80a67c0 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e80043b0 con 0x7fc0f80a67c0 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0f57fa640 4 set_mon_vals no callback set 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.065+0000 7fc0ef7fe640 10 monclient: dump: 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0fd660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc0dc0065c0 con 0x7fc0dc002860 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc0e8004640 con 0x7fc0f80a67c0 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(571..571 src has 299..571) ==== 5931+0+0 (unknown 491063308 0 0) 0x7fc0e804e830 con 0x7fc0f80a67c0 2024-08-19T10:05:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0ef7fe640 10 client.204174.objecter ms_dispatch 0x7fc0f8000b90 osd_map(571..571 src has 299..571) 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0ef7fe640 3 client.204174.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0ef7fe640 3 client.204174.objecter handle_osd_map decoding full epoch 571 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0ef7fe640 20 client.204174.objecter dump_active .. 0 homeless 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.066+0000 7fc0f77fe640 10 monclient: get_auth_request con 0x7fc0d8041550 auth_method 0 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0f67fc640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0dc002860 legacy=0x7fc0dc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 10 client.204174.objecter ms_handle_connect 0x7fc0dc002860 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 10 client.204174.objecter resend_mon_ops 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919757570 0 0) 0x7fc0dc0065c0 con 0x7fc0dc002860 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 10 monclient(hunting): my global_id is 174880 2024-08-19T10:05:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0f813af70 con 0x7fc0dc002860 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.068+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 80066966 0 0) 0x7fc0f813af70 con 0x7fc0dc002860 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0dc0065c0 con 0x7fc0dc002860 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e00048a0 con 0x7fc0dc002860 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 10 monclient: dump: 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 10 client.204174.objecter ms_handle_connect 0x7fc0d8041550 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3025108914 0 0) 0x7fc0dc0065c0 con 0x7fc0dc002860 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0dc00a160 con 0x7fc0dc002860 2024-08-19T10:05:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc0e00054e0 con 0x7fc0dc002860 2024-08-19T10:05:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0dc002860 legacy=0x7fc0dc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.069+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0dc002860 legacy=0x7fc0dc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.075+0000 7fc0fd660640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.075+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc0dc009e70 con 0x7fc0dc009130 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0f67fc640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0dc009130 legacy=0x7fc0dc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 client.204174.objecter ms_handle_connect 0x7fc0dc009130 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 client.204174.objecter resend_mon_ops 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550131948 0 0) 0x7fc0dc009e70 con 0x7fc0dc009130 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient(hunting): my global_id is 174889 2024-08-19T10:05:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0dc0065c0 con 0x7fc0dc009130 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 277585209 0 0) 0x7fc0dc0065c0 con 0x7fc0dc009130 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0dc009e70 con 0x7fc0dc009130 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0e0005d60 con 0x7fc0dc009130 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.078+0000 7fc0ef7fe640 10 monclient: dump: 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1797165733 0 0) 0x7fc0dc009e70 con 0x7fc0dc009130 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0dc00c3f0 con 0x7fc0dc009130 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3106348760) 0x7fc0e0008060 con 0x7fc0dc009130 2024-08-19T10:05:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0dc009130 legacy=0x7fc0dc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0ef7fe640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6789/0 conn(0x7fc0dc009130 legacy=0x7fc0dc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:44.100 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 394137, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:05:44.101 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.102 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.103 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:44.104 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:44.106 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.107 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.108 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.109 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.110 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.111 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:44.115 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.118 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.119 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:44.120 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 10 librados: watch_flush enter 2024-08-19T10:05:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 10 librados: watch_flush exit 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 20 client.204174.objecter shutdown clearing up homeless session... 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 10 client.204174.objecter successfully canceled tick 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc0d8041550 legacy=0x7fc0d8043a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 10 monclient: shutdown 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 >> v1:172.21.15.1:6789/0 conn(0x7fc0f80a67c0 legacy=0x7fc0f813c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0f77fe640 1 -- 172.21.15.1:0/2449025465 reap_dead start 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0f77fe640 1 -- 172.21.15.1:0/2449025465 reap_dead start 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 shutdown_connections 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 >> 172.21.15.1:0/2449025465 conn(0x7fc0f80ac4b0 msgr2=0x7fc0f813ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 shutdown_connections 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 -- 172.21.15.1:0/2449025465 wait complete. 2024-08-19T10:05:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.079+0000 7fc0fd660640 1 librados: shutdown 2024-08-19T10:05:44.124 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:44.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:05:44.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.238+0000 7fe7b9714640 10 monclient: get_monmap_and_config 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.238+0000 7fe7b9714640 10 monclient: build_initial_monmap 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.238+0000 7fe7b9714640 10 monclient: monmap: 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:44.239113+0000 2024-08-19T10:05:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:44.239113+0000 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:44.240 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.238+0000 7fe7b9714640 1 Processor -- start 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.238+0000 7fe7b9714640 1 -- start start 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.238+0000 7fe7b9714640 10 monclient: init 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient: _add_conns ranks=[8,4,6,1,0,3,7,5,2] 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient(hunting): picked mon.noname-e con 0x7fe7b40a6830 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b40694b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient(hunting): picked mon.noname-d con 0x7fe7b40a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:44.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b40a3c70 con 0x7fe7b40694b0 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b40a33d0 con 0x7fe7b40a46a0 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b40a36b0 con 0x7fe7b40a6830 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b9714640 10 monclient(hunting): authenticate will time out at 1600.315440s 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b2ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe7b40a6830 0x7fe7b40a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54700/0 (socket says 172.21.15.1:54700) 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b2ffd640 1 -- 172.21.15.1:0/2972805855 learned_addr learned my addr 172.21.15.1:0/2972805855 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.239+0000 7fe7b27fc640 1 --1- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6791/0 conn(0x7fe7b40694b0 0x7fe7b40a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56298/0 (socket says 172.21.15.1:56298) 2024-08-19T10:05:44.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.241+0000 7fe7b37fe640 1 -- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6792/0 conn(0x7fe7b40a46a0 legacy=0x7fe7b4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.241+0000 7fe7b27fc640 1 -- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6791/0 conn(0x7fe7b40694b0 legacy=0x7fe7b40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.241+0000 7fe7b2ffd640 1 -- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6793/0 conn(0x7fe7b40a6830 legacy=0x7fe7b40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 492707916 0 0) 0x7fe7b40a3c70 con 0x7fe7b40694b0 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): my global_id is 194368 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7a0003520 con 0x7fe7b40694b0 2024-08-19T10:05:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106966755 0 0) 0x7fe7b40a36b0 con 0x7fe7b40a6830 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): my global_id is 204164 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7b40a3c70 con 0x7fe7b40a6830 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622429009 0 0) 0x7fe7b40a33d0 con 0x7fe7b40a46a0 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient(hunting): my global_id is 204144 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7b40a36b0 con 0x7fe7b40a46a0 2024-08-19T10:05:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4120348353 0 0) 0x7fe7b40a3c70 con 0x7fe7b40a6830 2024-08-19T10:05:44.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b37fe640 1 --1- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6792/0 conn(0x7fe7b40a46a0 0x7fe7b4141850 :-1 s=OPENED pgs=3481 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fe7a4001e10 mon_map magic: 0 2024-08-19T10:05:44.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6792/0 conn(0x7fe7b40a46a0 legacy=0x7fe7b4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6791/0 conn(0x7fe7b40694b0 legacy=0x7fe7b40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:05:44.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 monclient: found mon.noname-e 2024-08-19T10:05:44.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b406dbb0 con 0x7fe7b40a6830 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: _finish_auth 0 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe7ac0019f0 con 0x7fe7b40a6830 2024-08-19T10:05:44.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b1ffb640 10 monclient: dump: 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b9714640 5 monclient: authenticate success, global_id 204164 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.677+0000 7fe7b9714640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7ac0048a0 con 0x7fe7b40a6830 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b1ffb640 1 -- 172.21.15.1:0/2972805855 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe7ac001d40 con 0x7fe7b40a6830 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b9714640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b9714640 4 set_mon_vals no callback set 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.678+0000 7fe7b1ffb640 10 monclient: dump: 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.691 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.679+0000 7fe7b9714640 10 monclient: shutdown 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.679+0000 7fe7b9714640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.679+0000 7fe7b9714640 1 -- 172.21.15.1:0/2972805855 >> v1:172.21.15.1:6793/0 conn(0x7fe7b40a6830 legacy=0x7fe7b40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.679+0000 7fe7b9714640 1 -- 172.21.15.1:0/2972805855 shutdown_connections 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.679+0000 7fe7b9714640 1 -- 172.21.15.1:0/2972805855 >> 172.21.15.1:0/2972805855 conn(0x7fe7b40a9020 msgr2=0x7fe7b4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.680+0000 7fe7b9714640 1 -- 172.21.15.1:0/2972805855 shutdown_connections 2024-08-19T10:05:44.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.680+0000 7fe7b9714640 1 -- 172.21.15.1:0/2972805855 wait complete. 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.681+0000 7fe7b9714640 10 monclient: build_initial_monmap 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.681+0000 7fe7b9714640 10 monclient: monmap: 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:44.693 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.681+0000 7fe7b9714640 1 librados: starting msgr at 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.681+0000 7fe7b9714640 1 librados: starting objecter 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.682+0000 7fe7b9714640 1 Processor -- start 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.682+0000 7fe7b9714640 1 -- start start 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.682+0000 7fe7b9714640 1 librados: setting wanted keys 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.682+0000 7fe7b9714640 1 librados: calling monclient init 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.682+0000 7fe7b9714640 10 monclient: init 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:44.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient: _add_conns ranks=[6,3,8,2,0,5,4,1,7] 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient(hunting): picked mon.noname-d con 0x7fe7b40a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient(hunting): picked mon.noname-g con 0x7fe7b40a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient(hunting): picked mon.noname-e con 0x7fe7b41375f0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b4068b00 con 0x7fe7b40a6830 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b41cf220 con 0x7fe7b40a46a0 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b37fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe7b41375f0 0x7fe7b41d6660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54712/0 (socket says 172.21.15.1:54712) 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b37fe640 1 -- 172.21.15.1:0/3735683416 learned_addr learned my addr 172.21.15.1:0/3735683416 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b4137f80 con 0x7fe7b41375f0 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:44.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.683+0000 7fe7b9714640 10 monclient(hunting): authenticate will time out at 1600.759723s 2024-08-19T10:05:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7b27fc640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b40a6830 legacy=0x7fe7b41ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fe7b40a6830 2024-08-19T10:05:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fe7b41375f0 2024-08-19T10:05:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7b37fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.1:6793/0 conn(0x7fe7b41375f0 legacy=0x7fe7b41d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.685+0000 7fe7b2ffd640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.1:6792/0 conn(0x7fe7b40a46a0 legacy=0x7fe7b41ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fe7b40a46a0 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529534409 0 0) 0x7fe7b4137f80 con 0x7fe7b41375f0 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): my global_id is 204173 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe794004eb0 con 0x7fe7b41375f0 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719314793 0 0) 0x7fe7b41cf220 con 0x7fe7b40a46a0 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): my global_id is 204153 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7b4137f80 con 0x7fe7b40a46a0 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941483411 0 0) 0x7fe79c0015e0 con 0x7fe7b40a6830 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): my global_id is 184593 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7b41cf220 con 0x7fe7b40a6830 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4237446338 0 0) 0x7fe794004eb0 con 0x7fe7b41375f0 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe79c0015e0 con 0x7fe7b41375f0 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe7a4003cd0 con 0x7fe7b41375f0 2024-08-19T10:05:44.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): dump: 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.686+0000 7fe7ab7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3010919348 0 0) 0x7fe7b41cf220 con 0x7fe7b40a6830 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe794004eb0 con 0x7fe7b40a6830 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 55847806 0 0) 0x7fe79c0015e0 con 0x7fe7b41375f0 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.1:6792/0 conn(0x7fe7b40a46a0 legacy=0x7fe7b41ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b40a6830 legacy=0x7fe7b41ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 monclient: found mon.e 2024-08-19T10:05:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b4138df0 con 0x7fe7b41375f0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: _finish_auth 0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 5 monclient: authenticate success, global_id 204173 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 10 monclient: _renew_subs 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b4138550 con 0x7fe7b41375f0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 10 client.204173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 10 monclient: _renew_subs 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7b4136b10 con 0x7fe7b41375f0 2024-08-19T10:05:44.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7b9714640 1 librados: init done 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7a4003e70 con 0x7fe7b41375f0 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe7a40043e0 con 0x7fe7b41375f0 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.687+0000 7fe7ab7fe640 10 monclient: dump: 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.688+0000 7fe7b17fa640 4 set_mon_vals no callback set 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.688+0000 7fe7b9714640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe7a4004580 con 0x7fe7b41375f0 2024-08-19T10:05:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(571..571 src has 299..571) ==== 5931+0+0 (unknown 491063308 0 0) 0x7fe7a404e700 con 0x7fe7b41375f0 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7ab7fe640 10 client.204173.objecter ms_dispatch 0x7fe7b4000b90 osd_map(571..571 src has 299..571) 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7ab7fe640 3 client.204173.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7ab7fe640 3 client.204173.objecter handle_osd_map decoding full epoch 571 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe7b40a33d0 con 0x7fe7b4143180 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7ab7fe640 20 client.204173.objecter dump_active .. 0 homeless 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.689+0000 7fe7b2ffd640 10 monclient: get_auth_request con 0x7fe7940497e0 auth_method 0 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 client.204173.objecter ms_handle_connect 0x7fe7b4143180 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 client.204173.objecter resend_mon_ops 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7b27fc640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b4143180 legacy=0x7fe7b409e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 client.204173.objecter ms_handle_connect 0x7fe7940497e0 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1830366025 0 0) 0x7fe79c001b10 con 0x7fe7b4143180 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient(hunting): my global_id is 184602 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe79c0015e0 con 0x7fe7b4143180 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2033460299 0 0) 0x7fe79c0015e0 con 0x7fe7b4143180 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe79c001b10 con 0x7fe7b4143180 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe79c0051f0 con 0x7fe7b4143180 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 10 monclient: dump: 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2342911826 0 0) 0x7fe79c001b10 con 0x7fe7b4143180 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.692+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7b4137a40 con 0x7fe7b4143180 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.693+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe79c007570 con 0x7fe7b4143180 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.693+0000 7fe7ab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.693+0000 7fe7ab7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.693+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b4143180 legacy=0x7fe7b409e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.693+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b4143180 legacy=0x7fe7b409e6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.700+0000 7fe7b9714640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.700+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe7b4072fb0 con 0x7fe7b402e1d0 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7b27fc640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b402e1d0 legacy=0x7fe7b409ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 client.204173.objecter ms_handle_connect 0x7fe7b402e1d0 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 client.204173.objecter resend_mon_ops 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202998567 0 0) 0x7fe7b4072fb0 con 0x7fe7b402e1d0 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient(hunting): my global_id is 184611 2024-08-19T10:05:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe79c001b10 con 0x7fe7b402e1d0 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3146239134 0 0) 0x7fe79c001b10 con 0x7fe7b402e1d0 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7b4072fb0 con 0x7fe7b402e1d0 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe79c005cb0 con 0x7fe7b402e1d0 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 10 monclient: dump: 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2848040449 0 0) 0x7fe7b4072fb0 con 0x7fe7b402e1d0 2024-08-19T10:05:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.703+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7b402e5b0 con 0x7fe7b402e1d0 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2251597878) 0x7fe79c007070 con 0x7fe7b402e1d0 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7ab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7ab7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b402e1d0 legacy=0x7fe7b409ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7ab7fe640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6790/0 conn(0x7fe7b402e1d0 legacy=0x7fe7b409ede0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:44.716 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 261957, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.718 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.719 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:44.720 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.721 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:44.722 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.723 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:44.726 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.731 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.732 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.732 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:44.733 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:44.734 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:44.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.736 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:44.737 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 10 librados: watch_flush enter 2024-08-19T10:05:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 10 librados: watch_flush exit 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 20 client.204173.objecter shutdown clearing up homeless session... 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 10 client.204173.objecter successfully canceled tick 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe7940497e0 legacy=0x7fe79404bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 10 monclient: shutdown 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 >> v1:172.21.15.1:6793/0 conn(0x7fe7b41375f0 legacy=0x7fe7b41d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b37fe640 1 -- 172.21.15.1:0/3735683416 reap_dead start 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b37fe640 1 -- 172.21.15.1:0/3735683416 reap_dead start 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 shutdown_connections 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 >> 172.21.15.1:0/3735683416 conn(0x7fe7b40a9020 msgr2=0x7fe7b40a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 shutdown_connections 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 -- 172.21.15.1:0/3735683416 wait complete. 2024-08-19T10:05:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.704+0000 7fe7b9714640 1 librados: shutdown 2024-08-19T10:05:44.741 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:44.741 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:05:44.741 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:05:44.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.841+0000 7fd7f673a640 10 monclient: get_monmap_and_config 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.841+0000 7fd7f673a640 10 monclient: build_initial_monmap 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.841+0000 7fd7f673a640 10 monclient: monmap: 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:44.842541+0000 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:44.842541+0000 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:44.844 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 1 Processor -- start 2024-08-19T10:05:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 1 -- start start 2024-08-19T10:05:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient: init 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient: _add_conns ranks=[5,7,3,0,2,6,4,8,1] 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient(hunting): picked mon.noname-h con 0x7fd7f00a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient(hunting): picked mon.noname-i con 0x7fd7f00694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient(hunting): picked mon.noname-g con 0x7fd7f00a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f00a3c70 con 0x7fd7f00a46a0 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f00a33d0 con 0x7fd7f00a6830 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f00a36b0 con 0x7fd7f00694b0 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.842+0000 7fd7f673a640 10 monclient(hunting): authenticate will time out at 1600.918973s 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.843+0000 7fd7f4cd7640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fd7f00a46a0 0x7fd7f0141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:51964/0 (socket says 172.21.15.1:51964) 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.843+0000 7fd7f4cd7640 1 -- 172.21.15.1:0/2444064344 learned_addr learned my addr 172.21.15.1:0/2444064344 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.843+0000 7fd7effff640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fd7f00a6830 0x7fd7f00a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:44710/0 (socket says 172.21.15.1:44710) 2024-08-19T10:05:44.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.845+0000 7fd7ef7fe640 1 -- 172.21.15.1:0/2444064344 >> v1:172.21.15.82:6792/0 conn(0x7fd7f00694b0 legacy=0x7fd7f00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.845+0000 7fd7effff640 1 -- 172.21.15.1:0/2444064344 >> v1:172.21.15.82:6791/0 conn(0x7fd7f00a6830 legacy=0x7fd7f00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.845+0000 7fd7f4cd7640 1 -- 172.21.15.1:0/2444064344 >> v1:172.21.15.82:6790/0 conn(0x7fd7f00a46a0 legacy=0x7fd7f0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.854+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324863257 0 0) 0x7fd7f00a33d0 con 0x7fd7f00a6830 2024-08-19T10:05:44.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.854+0000 7fd7eeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.854+0000 7fd7eeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.854+0000 7fd7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.854+0000 7fd7eeffd640 10 monclient(hunting): my global_id is 27833 2024-08-19T10:05:44.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.854+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d8003520 con 0x7fd7f00a6830 2024-08-19T10:05:44.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.857+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2797648529 0 0) 0x7fd7f00a36b0 con 0x7fd7f00694b0 2024-08-19T10:05:44.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.857+0000 7fd7eeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.857+0000 7fd7eeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.857+0000 7fd7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.857+0000 7fd7eeffd640 10 monclient(hunting): my global_id is 204145 2024-08-19T10:05:44.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.857+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f00a33d0 con 0x7fd7f00694b0 2024-08-19T10:05:44.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.858+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215209259 0 0) 0x7fd7f00a3c70 con 0x7fd7f00a46a0 2024-08-19T10:05:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.858+0000 7fd7eeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.858+0000 7fd7eeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.858+0000 7fd7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.858+0000 7fd7eeffd640 10 monclient(hunting): my global_id is 184620 2024-08-19T10:05:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.858+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f00a36b0 con 0x7fd7f00a46a0 2024-08-19T10:05:44.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1796183368 0 0) 0x7fd7d8003520 con 0x7fd7f00a6830 2024-08-19T10:05:44.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 >> v1:172.21.15.82:6792/0 conn(0x7fd7f00694b0 legacy=0x7fd7f00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 >> v1:172.21.15.82:6790/0 conn(0x7fd7f00a46a0 legacy=0x7fd7f0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 monclient: found mon.noname-h 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f006dbb0 con 0x7fd7f00a6830 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e0001a10 con 0x7fd7f00a6830 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: dump: 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7f673a640 5 monclient: authenticate success, global_id 27833 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7f673a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7e0001ce0 con 0x7fd7f00a6830 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 1 -- 172.21.15.1:0/2444064344 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e0004b20 con 0x7fd7f00a6830 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7eeffd640 10 monclient: dump: 2024-08-19T10:05:44.871 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7f673a640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.865+0000 7fd7f673a640 4 set_mon_vals no callback set 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 10 monclient: shutdown 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 1 -- 172.21.15.1:0/2444064344 >> v1:172.21.15.82:6791/0 conn(0x7fd7f00a6830 legacy=0x7fd7f00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 1 -- 172.21.15.1:0/2444064344 shutdown_connections 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 1 -- 172.21.15.1:0/2444064344 >> 172.21.15.1:0/2444064344 conn(0x7fd7f00a9020 msgr2=0x7fd7f0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 1 -- 172.21.15.1:0/2444064344 shutdown_connections 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.866+0000 7fd7f673a640 1 -- 172.21.15.1:0/2444064344 wait complete. 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 10 monclient: build_initial_monmap 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 10 monclient: monmap: 2024-08-19T10:05:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 1 librados: starting msgr at 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 1 librados: starting objecter 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 1 Processor -- start 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 1 -- start start 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 1 librados: setting wanted keys 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 1 librados: calling monclient init 2024-08-19T10:05:44.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.867+0000 7fd7f673a640 10 monclient: init 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient: _add_conns ranks=[0,7,1,8,6,2,4,5,3] 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f00694b0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient(hunting): picked mon.noname-i con 0x7fd7f00a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient(hunting): picked mon.noname-f con 0x7fd7f00a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f013ae80 con 0x7fd7f00694b0 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f0137880 con 0x7fd7f00a6830 2024-08-19T10:05:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f00a9f50 con 0x7fd7f00a46a0 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7f673a640 10 monclient(hunting): authenticate will time out at 1600.944451s 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7effff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7f00694b0 0x7fd7f0136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60346/0 (socket says 172.21.15.1:60346) 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.868+0000 7fd7effff640 1 -- 172.21.15.1:0/1721968323 learned_addr learned my addr 172.21.15.1:0/1721968323 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7effff640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.1:6789/0 conn(0x7fd7f00694b0 legacy=0x7fd7f0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 client.?.objecter ms_handle_connect 0x7fd7f00694b0 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 client.?.objecter ms_handle_connect 0x7fd7f00a46a0 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 client.?.objecter ms_handle_connect 0x7fd7f00a6830 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ef7fe640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6792/0 conn(0x7fd7f00a46a0 legacy=0x7fd7f0137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7f4cd7640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6789/0 conn(0x7fd7f00a6830 legacy=0x7fd7f01d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7effff640 1 --1- 172.21.15.1:0/1721968323 >> v1:172.21.15.1:6789/0 conn(0x7fd7f00694b0 0x7fd7f0136a60 :-1 s=OPENED pgs=688 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fd7f013ae80 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920920446 0 0) 0x7fd7f00a9f50 con 0x7fd7f00a46a0 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): my global_id is 204154 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7dc004eb0 con 0x7fd7f00a46a0 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781629447 0 0) 0x7fd7f0137880 con 0x7fd7f00a6830 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 10 monclient(hunting): my global_id is 174898 2024-08-19T10:05:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.870+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f00a9f50 con 0x7fd7f00a6830 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2441511140 0 0) 0x7fd7dc004eb0 con 0x7fd7f00a46a0 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7f0137880 con 0x7fd7f00a46a0 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e4001cf0 con 0x7fd7f00a46a0 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): dump: 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 499791405 0 0) 0x7fd7f00a9f50 con 0x7fd7f00a6830 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7dc004eb0 con 0x7fd7f00a6830 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e80020c0 con 0x7fd7f00a6830 2024-08-19T10:05:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient(hunting): dump: 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 115670305 0 0) 0x7fd7f0137880 con 0x7fd7f00a46a0 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6789/0 conn(0x7fd7f00a6830 legacy=0x7fd7f01d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.1:6789/0 conn(0x7fd7f00694b0 legacy=0x7fd7f0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: _finish_hunting 0 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 monclient: found mon.i 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f0139fe0 con 0x7fd7f00a46a0 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: _finish_auth 0 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: _check_auth_tickets 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 5 monclient: authenticate success, global_id 204154 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634890907 0 0) 0x7fd7f013ae80 con 0x7fd7f00694b0 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 10 monclient: _renew_subs 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f013aba0 con 0x7fd7f00a46a0 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 10 client.204154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 10 monclient: _renew_subs 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7f013b330 con 0x7fd7f00a46a0 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7f673a640 1 librados: init done 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7e4003a60 con 0x7fd7f00a46a0 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e4003f90 con 0x7fd7f00a46a0 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ee7fc640 4 set_mon_vals no callback set 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:44.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.871+0000 7fd7ecff9640 10 monclient: dump: 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.891 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7f673a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd7d00065c0 con 0x7fd7d0002860 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd7e4004230 con 0x7fd7f00a46a0 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(572..572 src has 299..572) ==== 5923+0+0 (unknown 3066676271 0 0) 0x7fd7e404e360 con 0x7fd7f00a46a0 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7ecff9640 10 client.204154.objecter ms_dispatch 0x7fd7f0000b90 osd_map(572..572 src has 299..572) 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7ecff9640 3 client.204154.objecter handle_osd_map got epochs [572,572] > 0 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.872+0000 7fd7ecff9640 3 client.204154.objecter handle_osd_map decoding full epoch 572 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.873+0000 7fd7f4cd7640 10 monclient: get_auth_request con 0x7fd7dc041460 auth_method 0 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.873+0000 7fd7ecff9640 20 client.204154.objecter dump_active .. 0 homeless 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7effff640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6791/0 conn(0x7fd7d0002860 legacy=0x7fd7d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 10 client.204154.objecter ms_handle_connect 0x7fd7d0002860 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 10 client.204154.objecter resend_mon_ops 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539506699 0 0) 0x7fd7d00065c0 con 0x7fd7d0002860 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 10 monclient(hunting): my global_id is 27842 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.874+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f013ae80 con 0x7fd7d0002860 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3363053077 0 0) 0x7fd7f013ae80 con 0x7fd7d0002860 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7d00065c0 con 0x7fd7d0002860 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e00017f0 con 0x7fd7d0002860 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 10 monclient: dump: 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 10 client.204154.objecter ms_handle_connect 0x7fd7dc041460 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1822716148 0 0) 0x7fd7d00065c0 con 0x7fd7d0002860 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7d000a160 con 0x7fd7d0002860 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd7e0001a10 con 0x7fd7d0002860 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6791/0 conn(0x7fd7d0002860 legacy=0x7fd7d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.875+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6791/0 conn(0x7fd7d0002860 legacy=0x7fd7d0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.882+0000 7fd7f673a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.882+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd7d0009e70 con 0x7fd7d0009130 2024-08-19T10:05:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7effff640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6791/0 conn(0x7fd7d0009130 legacy=0x7fd7d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 client.204154.objecter ms_handle_connect 0x7fd7d0009130 2024-08-19T10:05:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 client.204154.objecter resend_mon_ops 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813861369 0 0) 0x7fd7d0009e70 con 0x7fd7d0009130 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient(hunting): my global_id is 27851 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d00065c0 con 0x7fd7d0009130 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3864875061 0 0) 0x7fd7d00065c0 con 0x7fd7d0009130 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7d0009e70 con 0x7fd7d0009130 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7e0004e60 con 0x7fd7d0009130 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 10 monclient: dump: 2024-08-19T10:05:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 97978305 0 0) 0x7fd7d0009e70 con 0x7fd7d0009130 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.884+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7d000c410 con 0x7fd7d0009130 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 4114302689) 0x7fd7e00073c0 con 0x7fd7d0009130 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7ecff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6791/0 conn(0x7fd7d0009130 legacy=0x7fd7d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7ecff9640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6791/0 conn(0x7fd7d0009130 legacy=0x7fd7d00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:44.900 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 951151, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:44.901 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:44.902 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.903 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:44.904 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:44.904 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.904 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.905 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:44.906 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:44.907 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.911 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:44.912 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.915 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:44.916 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:44.917 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.921 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:44.922 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:44.926 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:44.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:44.929 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:44.929 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 10 librados: watch_flush enter 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 10 librados: watch_flush exit 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 20 client.204154.objecter shutdown clearing up homeless session... 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 10 client.204154.objecter successfully canceled tick 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd7dc041460 legacy=0x7fd7dc043920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 10 monclient: shutdown 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 >> v1:172.21.15.82:6792/0 conn(0x7fd7f00a46a0 legacy=0x7fd7f0137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f4cd7640 1 -- 172.21.15.1:0/1721968323 reap_dead start 2024-08-19T10:05:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f4cd7640 1 -- 172.21.15.1:0/1721968323 reap_dead start 2024-08-19T10:05:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 shutdown_connections 2024-08-19T10:05:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 >> 172.21.15.1:0/1721968323 conn(0x7fd7f00a9020 msgr2=0x7fd7f00a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 shutdown_connections 2024-08-19T10:05:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 -- 172.21.15.1:0/1721968323 wait complete. 2024-08-19T10:05:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:44.885+0000 7fd7f673a640 1 librados: shutdown 2024-08-19T10:05:44.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:44.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:05:44.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:05:45.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.032+0000 7f2570298640 10 monclient: get_monmap_and_config 2024-08-19T10:05:45.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.032+0000 7f2570298640 10 monclient: build_initial_monmap 2024-08-19T10:05:45.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.032+0000 7f2570298640 10 monclient: monmap: 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:45.033310+0000 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:45.033310+0000 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:45.034 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:45.035 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:45.035 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:45.035 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.032+0000 7f2570298640 1 Processor -- start 2024-08-19T10:05:45.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 1 -- start start 2024-08-19T10:05:45.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient: init 2024-08-19T10:05:45.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:45.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient: _add_conns ranks=[5,4,8,6,3,7,0,1,2] 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient(hunting): picked mon.noname-h con 0x7f25680a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient(hunting): picked mon.noname-c con 0x7f25680a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient(hunting): picked mon.noname-e con 0x7f256813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25680a3b40 con 0x7f25680a6ba0 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25680a8130 con 0x7f25680a67c0 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256806ede0 con 0x7f256813f5c0 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f2570298640 10 monclient(hunting): authenticate will time out at 1601.109794s 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f256d833640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f25680a6ba0 0x7f25680a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56300/0 (socket says 172.21.15.1:56300) 2024-08-19T10:05:45.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.033+0000 7f256d833640 1 -- 172.21.15.1:0/3114875488 learned_addr learned my addr 172.21.15.1:0/3114875488 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256e835640 1 -- 172.21.15.1:0/3114875488 >> v1:172.21.15.1:6793/0 conn(0x7f256813f5c0 legacy=0x7f25681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d833640 1 -- 172.21.15.1:0/3114875488 >> v1:172.21.15.1:6791/0 conn(0x7f25680a6ba0 legacy=0x7f25680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256e034640 1 -- 172.21.15.1:0/3114875488 >> v1:172.21.15.82:6791/0 conn(0x7f25680a67c0 legacy=0x7f25680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594652135 0 0) 0x7f256806ede0 con 0x7f256813f5c0 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): my global_id is 204182 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2554003520 con 0x7f256813f5c0 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866592887 0 0) 0x7f25680a8130 con 0x7f25680a67c0 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient(hunting): my global_id is 27860 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f256806ede0 con 0x7f25680a67c0 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2336445888 0 0) 0x7f2554003520 con 0x7f256813f5c0 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 >> v1:172.21.15.82:6791/0 conn(0x7f25680a67c0 legacy=0x7f25680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 >> v1:172.21.15.1:6791/0 conn(0x7f25680a6ba0 legacy=0x7f25680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: _finish_hunting 0 2024-08-19T10:05:45.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 monclient: found mon.noname-e 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25680a92a0 con 0x7f256813f5c0 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: _finish_auth 0 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: _check_auth_tickets 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2560001ea0 con 0x7f256813f5c0 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f2570298640 5 monclient: authenticate success, global_id 204182 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f2570298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: dump: 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.048 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f2570298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2560003990 con 0x7f256813f5c0 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 1 -- 172.21.15.1:0/3114875488 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2560003f00 con 0x7f256813f5c0 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f2570298640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f2570298640 4 set_mon_vals no callback set 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.036+0000 7f256d032640 10 monclient: dump: 2024-08-19T10:05:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 10 monclient: shutdown 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 1 -- 172.21.15.1:0/3114875488 >> v1:172.21.15.1:6793/0 conn(0x7f256813f5c0 legacy=0x7f25681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 1 -- 172.21.15.1:0/3114875488 shutdown_connections 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 1 -- 172.21.15.1:0/3114875488 >> 172.21.15.1:0/3114875488 conn(0x7f25680ac4b0 msgr2=0x7f25680a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 1 -- 172.21.15.1:0/3114875488 shutdown_connections 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 1 -- 172.21.15.1:0/3114875488 wait complete. 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 10 monclient: build_initial_monmap 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.037+0000 7f2570298640 10 monclient: monmap: 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:45.051 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:45.052 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 librados: starting msgr at 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 librados: starting objecter 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 Processor -- start 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 -- start start 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 librados: setting wanted keys 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 librados: calling monclient init 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient: init 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient: _add_conns ranks=[4,3,2,5,7,1,6,0,8] 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient(hunting): picked mon.noname-c con 0x7f25680a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient(hunting): picked mon.noname-g con 0x7f25680a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient(hunting): picked mon.noname-b con 0x7f256813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:45.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256813a8d0 con 0x7f256813f5c0 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25680a9d30 con 0x7f25680a6ba0 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25680a16b0 con 0x7f25680a67c0 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f2570298640 10 monclient(hunting): authenticate will time out at 1601.114694s 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f256e835640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f256813f5c0 0x7f25680a0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51260/0 (socket says 172.21.15.1:51260) 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.038+0000 7f256e835640 1 -- 172.21.15.1:0/850388095 learned_addr learned my addr 172.21.15.1:0/850388095 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.040+0000 7f256e034640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.1:6791/0 conn(0x7f25680a67c0 legacy=0x7f256809fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.040+0000 7f256e835640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.1:6790/0 conn(0x7f256813f5c0 legacy=0x7f25680a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.040+0000 7f255effd640 10 client.?.objecter ms_handle_connect 0x7f25680a67c0 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.040+0000 7f255effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:45.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.040+0000 7f255effd640 10 client.?.objecter ms_handle_connect 0x7f256813f5c0 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.040+0000 7f255effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2523720053 0 0) 0x7f25680a16b0 con 0x7f25680a67c0 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): my global_id is 194377 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f254c004eb0 con 0x7f25680a67c0 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1017576823 0 0) 0x7f256813a8d0 con 0x7f256813f5c0 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): my global_id is 184565 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25680a16b0 con 0x7f256813f5c0 2024-08-19T10:05:45.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 client.?.objecter ms_handle_connect 0x7f25680a6ba0 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f256d833640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6790/0 conn(0x7f25680a6ba0 legacy=0x7f25680a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1901224692 0 0) 0x7f25680a16b0 con 0x7f256813f5c0 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f256813a8d0 con 0x7f256813f5c0 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2560003e80 con 0x7f256813f5c0 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): dump: 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 867541376 0 0) 0x7f254c004eb0 con 0x7f25680a67c0 2024-08-19T10:05:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25680a16b0 con 0x7f25680a67c0 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914842843 0 0) 0x7f25680a9d30 con 0x7f25680a6ba0 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): my global_id is 184629 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f254c004eb0 con 0x7f25680a6ba0 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2564003610 con 0x7f25680a67c0 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient(hunting): dump: 2024-08-19T10:05:45.058 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 837832121 0 0) 0x7f256813a8d0 con 0x7f256813f5c0 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.1:6791/0 conn(0x7f25680a67c0 legacy=0x7f256809fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6790/0 conn(0x7f25680a6ba0 legacy=0x7f25680a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 monclient: found mon.b 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256813a5f0 con 0x7f256813f5c0 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: _finish_auth 0 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 5 monclient: authenticate success, global_id 184565 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 10 monclient: _renew_subs 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25680a2210 con 0x7f256813f5c0 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 10 client.184565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 10 monclient: _renew_subs 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:05:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f25681d7440 con 0x7f256813f5c0 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f2570298640 1 librados: init done 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2560003990 con 0x7f256813f5c0 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f25600046c0 con 0x7f256813f5c0 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f256c831640 4 set_mon_vals no callback set 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.041+0000 7f255effd640 10 monclient: dump: 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f2570298640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f25500065c0 con 0x7f2550002860 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2560011e00 con 0x7f256813f5c0 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(572..572 src has 299..572) ==== 5923+0+0 (unknown 3066676271 0 0) 0x7f256004e5f0 con 0x7f256813f5c0 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f255effd640 10 client.184565.objecter ms_dispatch 0x7f2568000b90 osd_map(572..572 src has 299..572) 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f255effd640 3 client.184565.objecter handle_osd_map got epochs [572,572] > 0 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f255effd640 3 client.184565.objecter handle_osd_map decoding full epoch 572 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f255effd640 20 client.184565.objecter dump_active .. 0 homeless 2024-08-19T10:05:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.042+0000 7f256d833640 10 monclient: get_auth_request con 0x7f254c0417a0 auth_method 0 2024-08-19T10:05:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f256e034640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6792/0 conn(0x7f2550002860 legacy=0x7f2550004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 10 client.184565.objecter ms_handle_connect 0x7f2550002860 2024-08-19T10:05:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 10 client.184565.objecter resend_mon_ops 2024-08-19T10:05:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896725769 0 0) 0x7f25500065c0 con 0x7f2550002860 2024-08-19T10:05:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 10 monclient(hunting): my global_id is 204163 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.044+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f256813a8d0 con 0x7f2550002860 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1717954264 0 0) 0x7f256813a8d0 con 0x7f2550002860 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25500065c0 con 0x7f2550002860 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2564003da0 con 0x7f2550002860 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 10 monclient: dump: 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.069 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 10 client.184565.objecter ms_handle_connect 0x7f254c0417a0 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 994954533 0 0) 0x7f25500065c0 con 0x7f2550002860 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f255000a160 con 0x7f2550002860 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2564004110 con 0x7f2550002860 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6792/0 conn(0x7f2550002860 legacy=0x7f2550004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.045+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6792/0 conn(0x7f2550002860 legacy=0x7f2550004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.051+0000 7f2570298640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.051+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2550009e50 con 0x7f2550009110 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f256e034640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6792/0 conn(0x7f2550009110 legacy=0x7f25500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 client.184565.objecter ms_handle_connect 0x7f2550009110 2024-08-19T10:05:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 client.184565.objecter resend_mon_ops 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4233830155 0 0) 0x7f2550009e50 con 0x7f2550009110 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient(hunting): my global_id is 204172 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25500065c0 con 0x7f2550009110 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2454999553 0 0) 0x7f25500065c0 con 0x7f2550009110 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2550009e50 con 0x7f2550009110 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2564005b00 con 0x7f2550009110 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 10 monclient: dump: 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2648049151 0 0) 0x7f2550009e50 con 0x7f2550009110 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.054+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f255000c3a0 con 0x7f2550009110 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 2744785806) 0x7f2564005e10 con 0x7f2550009110 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f255effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f255effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6792/0 conn(0x7f2550009110 legacy=0x7f25500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f255effd640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6792/0 conn(0x7f2550009110 legacy=0x7f25500095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:45.073 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 10924, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:45.076 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:45.077 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:45.078 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.079 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.080 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.081 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.083 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.084 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.087 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.089 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.091 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:45.091 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:45.091 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.091 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.091 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.092 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:45.092 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:45.092 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:45.092 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:45.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:45.095 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:45.096 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:45.096 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:45.096 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:45.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:45.096 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:45.096 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:45.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 10 librados: watch_flush enter 2024-08-19T10:05:45.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 10 librados: watch_flush exit 2024-08-19T10:05:45.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 20 client.184565.objecter shutdown clearing up homeless session... 2024-08-19T10:05:45.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 10 client.184565.objecter successfully canceled tick 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.82:6812/2549104415 conn(0x7f254c0417a0 legacy=0x7f254c043c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 10 monclient: shutdown 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 >> v1:172.21.15.1:6790/0 conn(0x7f256813f5c0 legacy=0x7f25680a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f256e835640 1 -- 172.21.15.1:0/850388095 reap_dead start 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f256e835640 1 -- 172.21.15.1:0/850388095 reap_dead start 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 shutdown_connections 2024-08-19T10:05:45.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 >> 172.21.15.1:0/850388095 conn(0x7f25680ac4b0 msgr2=0x7f25680a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:45.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 shutdown_connections 2024-08-19T10:05:45.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 -- 172.21.15.1:0/850388095 wait complete. 2024-08-19T10:05:45.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.055+0000 7f2570298640 1 librados: shutdown 2024-08-19T10:05:45.100 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:45.101 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T10:05:45.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.200+0000 7f177dd5c640 10 monclient: get_monmap_and_config 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.200+0000 7f177dd5c640 10 monclient: build_initial_monmap 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.200+0000 7f177dd5c640 10 monclient: monmap: 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:45.201066+0000 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:45.201066+0000 2024-08-19T10:05:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:45.202 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.200+0000 7f177dd5c640 1 Processor -- start 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.200+0000 7f177dd5c640 1 -- start start 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.200+0000 7f177dd5c640 10 monclient: init 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient: _add_conns ranks=[1,8,3,0,4,7,5,6,2] 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient(hunting): picked mon.noname-f con 0x7f177809eb60 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient(hunting): picked mon.noname-e con 0x7f177809fc00 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient(hunting): picked mon.noname-g con 0x7f177809f650 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17780a4070 con 0x7f177809eb60 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17780a3df0 con 0x7f177809f650 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17780a0760 con 0x7f177809fc00 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f177dd5c640 10 monclient(hunting): authenticate will time out at 1601.277426s 2024-08-19T10:05:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f1776ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f177809fc00 0x7f177813b6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54724/0 (socket says 172.21.15.1:54724) 2024-08-19T10:05:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.201+0000 7f1776ffd640 1 -- 172.21.15.1:0/3759676578 learned_addr learned my addr 172.21.15.1:0/3759676578 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f1776ffd640 1 -- 172.21.15.1:0/3759676578 >> v1:172.21.15.1:6793/0 conn(0x7f177809fc00 legacy=0x7f177813b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711547213 0 0) 0x7f17780a0760 con 0x7f177809fc00 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 10 monclient(hunting): my global_id is 204191 2024-08-19T10:05:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1760003520 con 0x7f177809fc00 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17777fe640 1 -- 172.21.15.1:0/3759676578 >> v1:172.21.15.82:6789/0 conn(0x7f177809eb60 legacy=0x7f177809ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f1777fff640 1 -- 172.21.15.1:0/3759676578 >> v1:172.21.15.82:6790/0 conn(0x7f177809f650 legacy=0x7f177813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.203+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418358016 0 0) 0x7f17780a4070 con 0x7f177809eb60 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient(hunting): my global_id is 174907 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17780a0760 con 0x7f177809eb60 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2188159811 0 0) 0x7f1760003520 con 0x7f177809fc00 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 >> v1:172.21.15.82:6790/0 conn(0x7f177809f650 legacy=0x7f177813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 >> v1:172.21.15.82:6789/0 conn(0x7f177809eb60 legacy=0x7f177809ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 monclient: found mon.noname-e 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17780a0a40 con 0x7f177809fc00 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: _finish_auth 0 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f176c001ae0 con 0x7f177809fc00 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 5 monclient: authenticate success, global_id 204191 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: dump: 2024-08-19T10:05:45.210 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f176c0048a0 con 0x7f177809fc00 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 1 -- 172.21.15.1:0/3759676578 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f176c004e10 con 0x7f177809fc00 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f17767fc640 10 monclient: dump: 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 4 set_mon_vals no callback set 2024-08-19T10:05:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 10 monclient: shutdown 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 1 -- 172.21.15.1:0/3759676578 >> v1:172.21.15.1:6793/0 conn(0x7f177809fc00 legacy=0x7f177813b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 1 -- 172.21.15.1:0/3759676578 shutdown_connections 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 1 -- 172.21.15.1:0/3759676578 >> 172.21.15.1:0/3759676578 conn(0x7f1778068650 msgr2=0x7f1778068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 1 -- 172.21.15.1:0/3759676578 shutdown_connections 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.204+0000 7f177dd5c640 1 -- 172.21.15.1:0/3759676578 wait complete. 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 10 monclient: build_initial_monmap 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 10 monclient: monmap: 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:45.215 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 1 librados: starting msgr at 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 1 librados: starting objecter 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 1 Processor -- start 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 1 -- start start 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 1 librados: setting wanted keys 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 1 librados: calling monclient init 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.205+0000 7f177dd5c640 10 monclient: init 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient: _add_conns ranks=[3,1,4,7,0,8,2,6,5] 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient(hunting): picked mon.noname-g con 0x7f177809eb60 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient(hunting): picked mon.noname-f con 0x7f177809f650 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient(hunting): picked mon.noname-c con 0x7f177809fc00 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1778121440 con 0x7f177809f650 2024-08-19T10:05:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17780a2fc0 con 0x7f177809eb60 2024-08-19T10:05:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1778121a10 con 0x7f177809fc00 2024-08-19T10:05:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f177dd5c640 10 monclient(hunting): authenticate will time out at 1601.282296s 2024-08-19T10:05:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f1777fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f177809fc00 0x7f17781d21f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56332/0 (socket says 172.21.15.1:56332) 2024-08-19T10:05:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.206+0000 7f1777fff640 1 -- 172.21.15.1:0/1625970311 learned_addr learned my addr 172.21.15.1:0/1625970311 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:45.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f1777fff640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.1:6791/0 conn(0x7f177809fc00 legacy=0x7f17781d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 client.?.objecter ms_handle_connect 0x7f177809fc00 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 client.?.objecter ms_handle_connect 0x7f177809eb60 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f17777fe640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.82:6790/0 conn(0x7f177809eb60 legacy=0x7f17781363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 client.?.objecter ms_handle_connect 0x7f177809f650 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f1776ffd640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.82:6789/0 conn(0x7f177809f650 legacy=0x7f17781ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423808603 0 0) 0x7f1778121a10 con 0x7f177809fc00 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): my global_id is 194386 2024-08-19T10:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1764004ee0 con 0x7f177809fc00 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353174379 0 0) 0x7f17780a2fc0 con 0x7f177809eb60 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): my global_id is 184647 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1778121a10 con 0x7f177809eb60 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2441161296 0 0) 0x7f1778121440 con 0x7f177809f650 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): my global_id is 174916 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17780a2fc0 con 0x7f177809f650 2024-08-19T10:05:45.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4262041363 0 0) 0x7f1764004ee0 con 0x7f177809fc00 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1778121440 con 0x7f177809fc00 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1770003610 con 0x7f177809fc00 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): dump: 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.208+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 151822463 0 0) 0x7f17780a2fc0 con 0x7f177809f650 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1764004ee0 con 0x7f177809f650 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f176c001d30 con 0x7f177809f650 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient(hunting): dump: 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1814394965 0 0) 0x7f1778121a10 con 0x7f177809eb60 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17780a2fc0 con 0x7f177809eb60 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1768003990 con 0x7f177809eb60 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient(hunting): dump: 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3658880819 0 0) 0x7f1778121440 con 0x7f177809fc00 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.82:6790/0 conn(0x7f177809eb60 legacy=0x7f17781363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.82:6789/0 conn(0x7f177809f650 legacy=0x7f17781ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 monclient: found mon.c 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17781237b0 con 0x7f177809fc00 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: _finish_auth 0 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 5 monclient: authenticate success, global_id 194386 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 10 monclient: _renew_subs 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:45.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1778120f80 con 0x7f177809fc00 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 10 client.194386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 10 monclient: _renew_subs 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f1778136ca0 con 0x7f177809fc00 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f177dd5c640 1 librados: init done 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1770003b40 con 0x7f177809fc00 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f1770004030 con 0x7f177809fc00 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f1775ffb640 4 set_mon_vals no callback set 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.209+0000 7f175ffff640 10 monclient: dump: 2024-08-19T10:05:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f17700042d0 con 0x7f177809fc00 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f177dd5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f177dd5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f177dd5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1758005080 con 0x7f177809fc00 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(572..572 src has 299..572) ==== 5923+0+0 (unknown 3066676271 0 0) 0x7f177004e4e0 con 0x7f177809fc00 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f175ffff640 10 client.194386.objecter ms_dispatch 0x7f1778000b90 osd_map(572..572 src has 299..572) 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f175ffff640 3 client.194386.objecter handle_osd_map got epochs [572,572] > 0 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f175ffff640 3 client.194386.objecter handle_osd_map decoding full epoch 572 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f175ffff640 20 client.194386.objecter dump_active .. 0 homeless 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.210+0000 7f17777fe640 10 monclient: get_auth_request con 0x7f1764041b40 auth_method 0 2024-08-19T10:05:45.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.212+0000 7f175ffff640 10 client.194386.objecter ms_handle_connect 0x7f1764041b40 2024-08-19T10:05:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.214+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f17700176a0 con 0x7f177809fc00 2024-08-19T10:05:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.214+0000 7f175ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.214+0000 7f175ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.468+0000 7f177dd5c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T10:05:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.468+0000 7f177dd5c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T10:05:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.468+0000 7f177dd5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.468+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1758005c40 con 0x7f177809fc00 2024-08-19T10:05:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f175ffff640 1 -- 172.21.15.1:0/1625970311 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f1770016fa0 con 0x7f177809fc00 2024-08-19T10:05:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f175ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T10:05:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f175ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:45.475 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.475 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.475 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=453850954,config=2586702516,health=2536881673,kv=1455257465,logm=272930389}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=1665023395,config=2586702516,health=2536881673,kv=1455257465,logm=1378465239}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=30,config=1,health=12,kv=11,logm=46} crc {auth=1553821719,config=1228700967,health=2536881673,kv=487780811,logm=1931963576}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.470+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=453850954,config=2586702516,health=2536881673,kv=1063813278,logm=231182343}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.476 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=3663123454,config=2586702516,health=2091408699,kv=1455257465,logm=1180492150}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=857662835,config=2586702516,health=2536881673,kv=2401233588,logm=3991145146}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=3005404073,config=2586702516,health=721959230,kv=3098392845,logm=277195460}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=901309655,config=2586702516,health=2479353110,kv=1455257465,logm=86999839}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.471+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=453850954,config=2586702516,health=2536881673,kv=3798231245,logm=1147015055}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683342898}) 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.477 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314280313}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3884685798}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2029526300}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.472+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4098238991}) 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.478 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3660067279}) 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2100856800}) 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145580716}) 2024-08-19T10:05:45.479 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1819657111}) 2024-08-19T10:05:45.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 10 librados: watch_flush enter 2024-08-19T10:05:45.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 10 librados: watch_flush exit 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 20 client.194386.objecter shutdown clearing up homeless session... 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 10 client.194386.objecter successfully canceled tick 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.82:6812/2549104415 conn(0x7f1764041b40 legacy=0x7f1764044000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 10 monclient: shutdown 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.473+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 >> v1:172.21.15.1:6791/0 conn(0x7f177809fc00 legacy=0x7f17781d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 shutdown_connections 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 >> 172.21.15.1:0/1625970311 conn(0x7f1778068650 msgr2=0x7f17780a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 shutdown_connections 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f177dd5c640 1 -- 172.21.15.1:0/1625970311 wait complete. 2024-08-19T10:05:45.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f177dd5c640 1 librados: shutdown 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859212710}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810326795}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1121491278}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.486 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2978180318}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1999086199}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3762515851}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=501641424}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527146879}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.474+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3054362595}) 2024-08-19T10:05:45.487 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355132135}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1784431691}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3872953183}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4024818803}) 2024-08-19T10:05:45.488 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3911225783}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3257591279}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2344464448}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667141836}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.476+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=752112208}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.489 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2477882578}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1220334533}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1392042000}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1790531650}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3788202019}) 2024-08-19T10:05:45.490 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=381029018}) 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=738893762}) 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311348252}) 2024-08-19T10:05:45.491 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.478+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2751615233}) 2024-08-19T10:05:45.492 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2425290706}) 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2347030099}) 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.502 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2211469146}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1013754696}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=687139086}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=993528818}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.503 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4102079232}) 2024-08-19T10:05:45.504 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.504 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150699676}) 2024-08-19T10:05:45.504 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.491+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1319079832}) 2024-08-19T10:05:45.504 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.504 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377435299}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=761246223}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=186142490}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2929403111}) 2024-08-19T10:05:45.506 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=730768777}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1278669806}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=641528858}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395786572}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.492+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1213491228}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.507 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609702314}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701512394}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2812202455}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=995938148}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3389404355}) 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.508 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3168794602}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2560508903}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276661963}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.494+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2994308089}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245504537}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.509 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149154004}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1503841958}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1501008464}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2994458195}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1986413067}) 2024-08-19T10:05:45.510 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3237398923}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406024168}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.495+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2155648277}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640071478}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681447697}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.511 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=329121544}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3551451659}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=194738955}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3873412418}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2076920432}) 2024-08-19T10:05:45.512 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.513 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932734034}) 2024-08-19T10:05:45.513 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.496+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2875253498}) 2024-08-19T10:05:45.862 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.859+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.859+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.859+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810195319}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.859+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328848444}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=262537173}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2092522352}) 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=233561522}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2313074815}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4145827817}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425532359}) 2024-08-19T10:05:45.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1413967363}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.864+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.864+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446309714}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2489122618}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3306371503}) 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.867 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=939121946}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3233258976}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2189560016}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=987173989}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583668534}) 2024-08-19T10:05:45.868 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.865+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1811052047}) 2024-08-19T10:05:45.883 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.883 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=240026475}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088156623}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4159120103}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1403276057}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.884 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3326004124}) 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4136035843}) 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1185021802}) 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311797440}) 2024-08-19T10:05:45.885 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:45.882+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=753665161}) 2024-08-19T10:05:46.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684104708}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185746075}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3966803382}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4109961946}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1686042847}) 2024-08-19T10:05:46.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3413485587}) 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=212702272}) 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613232062}) 2024-08-19T10:05:46.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2291697363}) 2024-08-19T10:05:46.493 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:05:46.494 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:05:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.603+0000 7fd9e10e5640 10 monclient: get_monmap_and_config 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.603+0000 7fd9e10e5640 10 monclient: build_initial_monmap 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.603+0000 7fd9e10e5640 10 monclient: monmap: 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:46.604751+0000 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:46.604751+0000 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.604+0000 7fd9e10e5640 1 Processor -- start 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.604+0000 7fd9e10e5640 1 -- start start 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.604+0000 7fd9e10e5640 10 monclient: init 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient: _add_conns ranks=[5,0,8,7,6,4,3,2,1] 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient(hunting): picked mon.noname-h con 0x7fd9dc0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient(hunting): picked mon.noname-a con 0x7fd9dc0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient(hunting): picked mon.noname-e con 0x7fd9dc13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc0a3b40 con 0x7fd9dc0a6ba0 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc0a8130 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc06ede0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9e10e5640 10 monclient(hunting): authenticate will time out at 1602.681224s 2024-08-19T10:05:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9d3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9dc0a6ba0 0x7fd9dc0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60368/0 (socket says 172.21.15.1:60368) 2024-08-19T10:05:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9db59e640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd9dc13f5c0 0x7fd9dc1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54726/0 (socket says 172.21.15.1:54726) 2024-08-19T10:05:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.605+0000 7fd9d3fff640 1 -- 172.21.15.1:0/3373506196 learned_addr learned my addr 172.21.15.1:0/3373506196 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:46.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9d3fff640 1 -- 172.21.15.1:0/3373506196 >> v1:172.21.15.1:6789/0 conn(0x7fd9dc0a6ba0 legacy=0x7fd9dc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:46.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9db59e640 1 -- 172.21.15.1:0/3373506196 >> v1:172.21.15.1:6793/0 conn(0x7fd9dc13f5c0 legacy=0x7fd9dc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:46.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9dad9d640 1 -- 172.21.15.1:0/3373506196 >> v1:172.21.15.82:6791/0 conn(0x7fd9dc0a67c0 legacy=0x7fd9dc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994164630 0 0) 0x7fd9dc0a3b40 con 0x7fd9dc0a6ba0 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _negotiate 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): my global_id is 204192 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c8003520 con 0x7fd9dc0a6ba0 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501031547 0 0) 0x7fd9dc06ede0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _negotiate 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): my global_id is 204200 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9dc0a3b40 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172902346 0 0) 0x7fd9dc0a8130 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _negotiate 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 10 monclient(hunting): my global_id is 27869 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.607+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9dc06ede0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3533155150 0 0) 0x7fd9dc0a3b40 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 >> v1:172.21.15.82:6791/0 conn(0x7fd9dc0a67c0 legacy=0x7fd9dc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 >> v1:172.21.15.1:6789/0 conn(0x7fd9dc0a6ba0 legacy=0x7fd9dc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: _finish_hunting 0 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 monclient: found mon.noname-e 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9dc0a92a0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: _finish_auth 0 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: _check_auth_tickets 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 5 monclient: authenticate success, global_id 204200 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd9d4001dd0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:46.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: dump: 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9d40020a0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 1 -- 172.21.15.1:0/3373506196 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd9d4004ea0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9da59c640 10 monclient: dump: 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 4 set_mon_vals no callback set 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 10 monclient: shutdown 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 1 -- 172.21.15.1:0/3373506196 >> v1:172.21.15.1:6793/0 conn(0x7fd9dc13f5c0 legacy=0x7fd9dc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 1 -- 172.21.15.1:0/3373506196 shutdown_connections 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 1 -- 172.21.15.1:0/3373506196 >> 172.21.15.1:0/3373506196 conn(0x7fd9dc0ac4b0 msgr2=0x7fd9dc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 1 -- 172.21.15.1:0/3373506196 shutdown_connections 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.608+0000 7fd9e10e5640 1 -- 172.21.15.1:0/3373506196 wait complete. 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.609+0000 7fd9e10e5640 10 monclient: build_initial_monmap 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.609+0000 7fd9e10e5640 10 monclient: monmap: 2024-08-19T10:05:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.609+0000 7fd9e10e5640 1 librados: starting msgr at 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.609+0000 7fd9e10e5640 1 librados: starting objecter 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 Processor -- start 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 -- start start 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 librados: setting wanted keys 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 librados: calling monclient init 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient: init 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient: _add_conns ranks=[6,1,8,5,3,2,0,4,7] 2024-08-19T10:05:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient(hunting): picked mon.noname-d con 0x7fd9dc0a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient(hunting): picked mon.noname-f con 0x7fd9dc0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient(hunting): picked mon.noname-e con 0x7fd9dc13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc13c660 con 0x7fd9dc0a6ba0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc0a9cc0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc13c380 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9e10e5640 10 monclient(hunting): authenticate will time out at 1602.686573s 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9db59e640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd9dc13f5c0 0x7fd9dc1d6340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:54730/0 (socket says 172.21.15.1:54730) 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9dad9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fd9dc0a67c0 0x7fd9dc139de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45158/0 (socket says 172.21.15.1:45158) 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.610+0000 7fd9db59e640 1 -- 172.21.15.1:0/2379591193 learned_addr learned my addr 172.21.15.1:0/2379591193 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9dad9d640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.1:6792/0 conn(0x7fd9dc0a67c0 legacy=0x7fd9dc139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d37fe640 10 client.?.objecter ms_handle_connect 0x7fd9dc0a67c0 2024-08-19T10:05:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d37fe640 10 client.?.objecter ms_handle_connect 0x7fd9dc13f5c0 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9db59e640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.1:6793/0 conn(0x7fd9dc13f5c0 legacy=0x7fd9dc1d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d3fff640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6789/0 conn(0x7fd9dc0a6ba0 legacy=0x7fd9dc1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d37fe640 10 client.?.objecter ms_handle_connect 0x7fd9dc0a6ba0 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.612+0000 7fd9d3fff640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6789/0 conn(0x7fd9dc0a6ba0 legacy=0x7fd9dc1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d3fff640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6789/0 conn(0x7fd9dc0a6ba0 legacy=0x7fd9dc1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d3fff640 1 --1- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6789/0 conn(0x7fd9dc0a6ba0 0x7fd9dc1d2c10 :-1 s=OPENED pgs=3729 cs=1 l=1).handle_message read tag failed 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d3fff640 1 --1- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6789/0 conn(0x7fd9dc0a6ba0 0x7fd9dc1d2c10 :-1 s=OPENED pgs=3729 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6789/0 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412366462 0 0) 0x7fd9dc0a9cc0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): my global_id is 204162 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9bc004ee0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308982734 0 0) 0x7fd9dc13c380 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): my global_id is 204209 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9dc0a9cc0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3634829202 0 0) 0x7fd9bc004ee0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9dc13c380 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd9cc003610 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): dump: 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3215216553 0 0) 0x7fd9dc0a9cc0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9bc004ee0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd9d4004ea0 con 0x7fd9dc13f5c0 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient(hunting): dump: 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:46.631 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2301035764 0 0) 0x7fd9dc13c380 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.1:6793/0 conn(0x7fd9dc13f5c0 legacy=0x7fd9dc1d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6789/0 conn(0x7fd9dc0a6ba0 legacy=0x7fd9dc1d2c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 monclient: found mon.d 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9dc13b130 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: _finish_auth 0 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 5 monclient: authenticate success, global_id 204162 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 10 monclient: _renew_subs 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9dc13a4f0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 10 client.204162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 10 monclient: _renew_subs 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9dc0a1c30 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9e10e5640 1 librados: init done 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9cc003b60 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd9cc0040d0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:46.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d9d9b640 4 set_mon_vals no callback set 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.613+0000 7fd9d37fe640 10 monclient: dump: 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd9cc004270 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(573..573 src has 299..573) ==== 5550+0+0 (unknown 3486788203 0 0) 0x7fd9cc04e260 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d37fe640 10 client.204162.objecter ms_dispatch 0x7fd9dc000b90 osd_map(573..573 src has 299..573) 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d37fe640 3 client.204162.objecter handle_osd_map got epochs [573,573] > 0 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d37fe640 3 client.204162.objecter handle_osd_map decoding full epoch 573 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9e10e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9e10e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:46.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9e10e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c0005080 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d37fe640 20 client.204162.objecter dump_active .. 0 homeless 2024-08-19T10:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.614+0000 7fd9d3fff640 10 monclient: get_auth_request con 0x7fd9bc0413e0 auth_method 0 2024-08-19T10:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.617+0000 7fd9d37fe640 10 client.204162.objecter ms_handle_connect 0x7fd9bc0413e0 2024-08-19T10:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.619+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fd9cc0175c0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.619+0000 7fd9d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:46.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.619+0000 7fd9d37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:46.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3079873794}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515843826}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=992310901}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=593308623}) 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3998825243}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1548530598}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=122215107}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354118176}) 2024-08-19T10:05:46.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.674+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2353649194}) 2024-08-19T10:05:46.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2272420248}) 2024-08-19T10:05:46.704 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885604715}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3368034589}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4102775770}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2368157853}) 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.705 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2865347301}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3170638027}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357939940}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.675+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2210227582}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863246726}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.706 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329024282}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2283737274}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4070786255}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3113796991}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2701586974}) 2024-08-19T10:05:46.707 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2648865551}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506891866}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.677+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3003318073}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425112715}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206609499}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.708 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1954868288}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3184472826}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2390313587}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1755327890}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2179061234}) 2024-08-19T10:05:46.709 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601030665}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.678+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3372679161}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4217249157}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=879000385}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3603628762}) 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.710 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1588169337}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2708199057}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=941919141}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2897192787}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878877209}) 2024-08-19T10:05:46.711 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.679+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4147649906}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3823964709}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712889200}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3761694391}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3075178674}) 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.712 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3794803076}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3954973187}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3185466048}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669467999}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.681+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300503768}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.713 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887172785}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492407733}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1255469213}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3523146155}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.714 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3735085080}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2934480984}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1627314093}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272316407}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.682+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4235623618}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205611171}) 2024-08-19T10:05:46.715 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465285783}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3610172111}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2995858118}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3937000149}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.716 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1545427803}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=887306370}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755023623}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.684+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=103760195}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490536324}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081506975}) 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.717 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2173775649}) 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=671547909}) 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3908292143}) 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.718 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2141071978}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3890277755}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494674922}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.685+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3765815238}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494358002}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002742061}) 2024-08-19T10:05:46.719 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4287384881}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=178822118}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1271603483}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.686+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1742686186}) 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.720 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3234279865}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692393745}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.687+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1666505786}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557138645}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669256784}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2952078693}) 2024-08-19T10:05:46.721 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1561978079}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3298844936}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3487666584}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3180802025}) 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.722 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024756392}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.688+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3220235555}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1735905981}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689751599}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=561543983}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1129183414}) 2024-08-19T10:05:46.723 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3946073196}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2578081168}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3259982550}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954142191}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.689+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=870657784}) 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.724 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1762712725}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1276706495}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2160488377}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=667052906}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3693774774}) 2024-08-19T10:05:46.725 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2924266623}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2485899516}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076318191}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.691+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=795009352}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=308603288}) 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.726 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3614791381}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3516112786}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1227343257}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2378672197}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=285378732}) 2024-08-19T10:05:46.727 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=839238968}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285895669}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.692+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375229922}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386430979}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612663286}) 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.728 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=195103661}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=144662244}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2491284787}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3671636343}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2678545861}) 2024-08-19T10:05:46.729 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121228906}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.693+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4190323518}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469192446}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626616989}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1169626916}) 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.730 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1290393396}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1412027461}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=130374268}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=782523528}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732114069}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.695+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1569444989}) 2024-08-19T10:05:46.731 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1904144464}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064279667}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1164169571}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1115068768}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.732 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3766520758}) 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=464360162}) 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3582233170}) 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241760113}) 2024-08-19T10:05:46.733 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.696+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2298705280}) 2024-08-19T10:05:46.876 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.876 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.876 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690514460}) 2024-08-19T10:05:46.876 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.876 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.876 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955236120}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2655116184}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2027376543}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3407959686}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3672874151}) 2024-08-19T10:05:46.877 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.878 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.878 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1545198745}) 2024-08-19T10:05:46.878 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.878 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496329820}) 2024-08-19T10:05:46.878 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.873+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=230886359}) 2024-08-19T10:05:46.894 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.891+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.894 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.891+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.894 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.891+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115993604}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.891+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1911436119}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2759981786}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1158372849}) 2024-08-19T10:05:46.895 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2352766122}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3687394502}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3682036860}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816685159}) 2024-08-19T10:05:46.896 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.892+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4004743103}) 2024-08-19T10:05:46.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116910902}) 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1317863054}) 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3571999574}) 2024-08-19T10:05:46.905 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1975884551}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=654935819}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4015412280}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2962432626}) 2024-08-19T10:05:46.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642748555}) 2024-08-19T10:05:46.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3977630439}) 2024-08-19T10:05:46.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.904+0000 7fd9e10e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.904+0000 7fd9e10e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.904+0000 7fd9e10e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.904+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd9c0005370 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.905+0000 7fd9d37fe640 1 -- 172.21.15.1:0/2379591193 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 2858254651) 0x7fd9cc016ec0 con 0x7fd9dc0a67c0 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.905+0000 7fd9d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.905+0000 7fd9d37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:46.908 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":130,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:05:46.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209789861}) 2024-08-19T10:05:46.910 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892028579}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=498179260}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2380720161}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2908080381}) 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.911 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1757927368}) 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2021052244}) 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967704547}) 2024-08-19T10:05:46.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:46.906+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4178022986}) 2024-08-19T10:05:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 10 librados: watch_flush enter 2024-08-19T10:05:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 10 librados: watch_flush exit 2024-08-19T10:05:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 20 client.204162.objecter shutdown clearing up homeless session... 2024-08-19T10:05:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 10 client.204162.objecter successfully canceled tick 2024-08-19T10:05:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd9bc0413e0 legacy=0x7fd9bc0438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 10 monclient: shutdown 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 >> v1:172.21.15.1:6792/0 conn(0x7fd9dc0a67c0 legacy=0x7fd9dc139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 shutdown_connections 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 >> 172.21.15.1:0/2379591193 conn(0x7fd9dc0ac4b0 msgr2=0x7fd9dc0aa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 shutdown_connections 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 -- 172.21.15.1:0/2379591193 wait complete. 2024-08-19T10:05:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:46.908+0000 7fd9e10e5640 1 librados: shutdown 2024-08-19T10:05:46.920 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:05:46.920 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T10:05:46.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:05:46.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:05:47.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.018+0000 7fc125d55640 10 monclient: get_monmap_and_config 2024-08-19T10:05:47.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.018+0000 7fc125d55640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.018+0000 7fc125d55640 10 monclient: monmap: 2024-08-19T10:05:47.019 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:47.019141+0000 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:47.019141+0000 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.020 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.021 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.021 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.021 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.021 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.021 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.021 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.018+0000 7fc125d55640 1 Processor -- start 2024-08-19T10:05:47.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.018+0000 7fc125d55640 1 -- start start 2024-08-19T10:05:47.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.018+0000 7fc125d55640 10 monclient: init 2024-08-19T10:05:47.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient: _add_conns ranks=[4,7,3,2,0,1,8,6,5] 2024-08-19T10:05:47.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient(hunting): picked mon.noname-c con 0x7fc1200a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient(hunting): picked mon.noname-i con 0x7fc1200694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient(hunting): picked mon.noname-g con 0x7fc1200a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1200a3c70 con 0x7fc1200a46a0 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1200a33d0 con 0x7fc1200a6830 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1200a36b0 con 0x7fc1200694b0 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc125d55640 10 monclient(hunting): authenticate will time out at 1603.095462s 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc11f7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fc1200a6830 0x7fc1200a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50776/0 (socket says 172.21.15.1:50776) 2024-08-19T10:05:47.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.019+0000 7fc11f7fe640 1 -- 172.21.15.1:0/801225339 learned_addr learned my addr 172.21.15.1:0/801225339 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.021+0000 7fc11f7fe640 1 -- 172.21.15.1:0/801225339 >> v1:172.21.15.1:6791/0 conn(0x7fc1200a6830 legacy=0x7fc1200a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.021+0000 7fc11ffff640 1 -- 172.21.15.1:0/801225339 >> v1:172.21.15.82:6790/0 conn(0x7fc1200a46a0 legacy=0x7fc120141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.021+0000 7fc11effd640 1 -- 172.21.15.1:0/801225339 >> v1:172.21.15.82:6792/0 conn(0x7fc1200694b0 legacy=0x7fc1200a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.021+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377083051 0 0) 0x7fc1200a33d0 con 0x7fc1200a6830 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): my global_id is 194395 2024-08-19T10:05:47.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc108003520 con 0x7fc1200a6830 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 483177114 0 0) 0x7fc1200a3c70 con 0x7fc1200a46a0 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): my global_id is 184656 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1200a33d0 con 0x7fc1200a46a0 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032128310 0 0) 0x7fc1200a36b0 con 0x7fc1200694b0 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient(hunting): my global_id is 204181 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1200a3c70 con 0x7fc1200694b0 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 915690597 0 0) 0x7fc108003520 con 0x7fc1200a6830 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 >> v1:172.21.15.82:6792/0 conn(0x7fc1200694b0 legacy=0x7fc1200a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 >> v1:172.21.15.82:6790/0 conn(0x7fc1200a46a0 legacy=0x7fc120141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 monclient: found mon.noname-c 2024-08-19T10:05:47.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12006dbb0 con 0x7fc1200a6830 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc110001e60 con 0x7fc1200a6830 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 5 monclient: authenticate success, global_id 194395 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: dump: 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.042 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc110004c20 con 0x7fc1200a6830 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 1 -- 172.21.15.1:0/801225339 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc110005190 con 0x7fc1200a6830 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 4 set_mon_vals no callback set 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc11e7fc640 10 monclient: dump: 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.044 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 10 monclient: shutdown 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 1 -- 172.21.15.1:0/801225339 >> v1:172.21.15.1:6791/0 conn(0x7fc1200a6830 legacy=0x7fc1200a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 1 -- 172.21.15.1:0/801225339 shutdown_connections 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 1 -- 172.21.15.1:0/801225339 >> 172.21.15.1:0/801225339 conn(0x7fc1200a9020 msgr2=0x7fc120068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 1 -- 172.21.15.1:0/801225339 shutdown_connections 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.022+0000 7fc125d55640 1 -- 172.21.15.1:0/801225339 wait complete. 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.023+0000 7fc125d55640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.023+0000 7fc125d55640 10 monclient: monmap: 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:47.047 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.023+0000 7fc125d55640 1 librados: starting msgr at 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.023+0000 7fc125d55640 1 librados: starting objecter 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.023+0000 7fc125d55640 1 Processor -- start 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 1 -- start start 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 1 librados: setting wanted keys 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 1 librados: calling monclient init 2024-08-19T10:05:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient: init 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient: _add_conns ranks=[1,0,7,5,8,4,6,2,3] 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient(hunting): picked mon.noname-f con 0x7fc1200694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient(hunting): picked mon.noname-a con 0x7fc1200a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient(hunting): picked mon.noname-i con 0x7fc1200a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12013af10 con 0x7fc1200a46a0 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12013b1f0 con 0x7fc1200694b0 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1200a9ff0 con 0x7fc1200a6830 2024-08-19T10:05:47.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc125d55640 10 monclient(hunting): authenticate will time out at 1603.100506s 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc11effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc1200a46a0 0x7fc1201d3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58700/0 (socket says 172.21.15.1:58700) 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.024+0000 7fc11effd640 1 -- 172.21.15.1:0/4015507741 learned_addr learned my addr 172.21.15.1:0/4015507741 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc11effd640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc1200a46a0 legacy=0x7fc1201d3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 client.?.objecter ms_handle_connect 0x7fc1200a46a0 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 client.?.objecter ms_handle_connect 0x7fc1200694b0 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc11f7fe640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.82:6789/0 conn(0x7fc1200694b0 legacy=0x7fc1201328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 client.?.objecter ms_handle_connect 0x7fc1200a6830 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc11ffff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.82:6792/0 conn(0x7fc1200a6830 legacy=0x7fc1201d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1209533189 0 0) 0x7fc12013af10 con 0x7fc1200a46a0 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): my global_id is 204201 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc10c004eb0 con 0x7fc1200a46a0 2024-08-19T10:05:47.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987431104 0 0) 0x7fc12013b1f0 con 0x7fc1200694b0 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 10 monclient(hunting): my global_id is 174925 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.026+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc12013af10 con 0x7fc1200694b0 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936794817 0 0) 0x7fc1200a9ff0 con 0x7fc1200a6830 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): my global_id is 204190 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc12013b1f0 con 0x7fc1200a6830 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1322670718 0 0) 0x7fc10c004eb0 con 0x7fc1200a46a0 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1200a9ff0 con 0x7fc1200a46a0 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc114003610 con 0x7fc1200a46a0 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): dump: 2024-08-19T10:05:47.054 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.055 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2130043300 0 0) 0x7fc12013af10 con 0x7fc1200694b0 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc10c004eb0 con 0x7fc1200694b0 2024-08-19T10:05:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc1100020b0 con 0x7fc1200694b0 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient(hunting): dump: 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.057 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 617483909 0 0) 0x7fc1200a9ff0 con 0x7fc1200a46a0 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.82:6792/0 conn(0x7fc1200a6830 legacy=0x7fc1201d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.82:6789/0 conn(0x7fc1200694b0 legacy=0x7fc1201328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 monclient: found mon.a 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1200a9c30 con 0x7fc1200a46a0 2024-08-19T10:05:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 5 monclient: authenticate success, global_id 204201 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 10 monclient: _renew_subs 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc12013ac30 con 0x7fc1200a46a0 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 10 client.204201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 10 monclient: _renew_subs 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc120137ad0 con 0x7fc1200a46a0 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc125d55640 1 librados: init done 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc114003bd0 con 0x7fc1200a46a0 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc114004140 con 0x7fc1200a46a0 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc11dffb640 4 set_mon_vals no callback set 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.027+0000 7fc107fff640 10 monclient: dump: 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.061 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc125d55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc1000065c0 con 0x7fc100002860 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc1140042e0 con 0x7fc1200a46a0 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(574..574 src has 299..574) ==== 5923+0+0 (unknown 649153396 0 0) 0x7fc11404e490 con 0x7fc1200a46a0 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc107fff640 10 client.204201.objecter ms_dispatch 0x7fc120000b90 osd_map(574..574 src has 299..574) 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc107fff640 3 client.204201.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T10:05:47.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc107fff640 3 client.204201.objecter handle_osd_map decoding full epoch 574 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc107fff640 20 client.204201.objecter dump_active .. 0 homeless 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.028+0000 7fc11ffff640 10 monclient: get_auth_request con 0x7fc10c0415b0 auth_method 0 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc11f7fe640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc100002860 legacy=0x7fc100004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 client.204201.objecter ms_handle_connect 0x7fc100002860 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 client.204201.objecter resend_mon_ops 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835274281 0 0) 0x7fc1000065c0 con 0x7fc100002860 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient(hunting): my global_id is 204210 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1200a9ff0 con 0x7fc100002860 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1119398024 0 0) 0x7fc1200a9ff0 con 0x7fc100002860 2024-08-19T10:05:47.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1000065c0 con 0x7fc100002860 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc110001900 con 0x7fc100002860 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 10 monclient: dump: 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 629728725 0 0) 0x7fc1000065c0 con 0x7fc100002860 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.030+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc10000a160 con 0x7fc100002860 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.031+0000 7fc107fff640 10 client.204201.objecter ms_handle_connect 0x7fc10c0415b0 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.031+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc110005450 con 0x7fc100002860 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.031+0000 7fc107fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.031+0000 7fc107fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:47.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.031+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc100002860 legacy=0x7fc100004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.031+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc100002860 legacy=0x7fc100004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.037+0000 7fc125d55640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:47.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.037+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc10000c4f0 con 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc11f7fe640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc100009110 legacy=0x7fc100009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 10 client.204201.objecter ms_handle_connect 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 10 client.204201.objecter resend_mon_ops 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3245928491 0 0) 0x7fc10000c4f0 con 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 10 monclient(hunting): my global_id is 204219 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.039+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1000065c0 con 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1945016925 0 0) 0x7fc1000065c0 con 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc10000c4f0 con 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc110005d00 con 0x7fc100009110 2024-08-19T10:05:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 10 monclient: dump: 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3038083105 0 0) 0x7fc10000c4f0 con 0x7fc100009110 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc100009ce0 con 0x7fc100009110 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7519 (unknown 0 0 3340845864) 0x7fc110008040 con 0x7fc100009110 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc100009110 legacy=0x7fc100009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc107fff640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc100009110 legacy=0x7fc100009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 12911, 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:47.071 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:47.072 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.073 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.075 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.076 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.077 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.078 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:47.079 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.080 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.081 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:47.082 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.083 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.084 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.085 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.092 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:47.092 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 10 librados: watch_flush enter 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 10 librados: watch_flush exit 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 20 client.204201.objecter shutdown clearing up homeless session... 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 10 client.204201.objecter successfully canceled tick 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc10c0415b0 legacy=0x7fc10c043a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 10 monclient: shutdown 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 >> v1:172.21.15.1:6789/0 conn(0x7fc1200a46a0 legacy=0x7fc1201d3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc11ffff640 1 -- 172.21.15.1:0/4015507741 reap_dead start 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc11ffff640 1 -- 172.21.15.1:0/4015507741 reap_dead start 2024-08-19T10:05:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 shutdown_connections 2024-08-19T10:05:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.040+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 >> 172.21.15.1:0/4015507741 conn(0x7fc1200a9020 msgr2=0x7fc1200a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.041+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 shutdown_connections 2024-08-19T10:05:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.041+0000 7fc125d55640 1 -- 172.21.15.1:0/4015507741 wait complete. 2024-08-19T10:05:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.041+0000 7fc125d55640 1 librados: shutdown 2024-08-19T10:05:47.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:47.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:05:47.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:05:47.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033692156}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239700444}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2492283341}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4237636349}) 2024-08-19T10:05:47.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1199441197}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3775289887}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3340242861}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144107368}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.168+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=885962456}) 2024-08-19T10:05:47.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3676255962}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243265778}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=435726828}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155079657}) 2024-08-19T10:05:47.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=198575167}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1656474577}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1879709660}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789206706}) 2024-08-19T10:05:47.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3803181618}) 2024-08-19T10:05:47.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.196+0000 7fd1a3a6e640 10 monclient: get_monmap_and_config 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.196+0000 7fd1a3a6e640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.196+0000 7fd1a3a6e640 10 monclient: monmap: 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:47.197650+0000 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:47.197650+0000 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.198 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.199 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.199 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.199 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.199 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.199 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 1 Processor -- start 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 1 -- start start 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient: init 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient: _add_conns ranks=[3,7,8,4,2,6,5,1,0] 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient(hunting): picked mon.noname-g con 0x7fd19c0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient(hunting): picked mon.noname-i con 0x7fd19c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient(hunting): picked mon.noname-e con 0x7fd19c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:47.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c0a3b40 con 0x7fd19c0a67c0 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c0a8130 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c06ede0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.197+0000 7fd1a3a6e640 10 monclient(hunting): authenticate will time out at 1603.273991s 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.198+0000 7fd1a200b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd19c13f5c0 0x7fd19c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48122/0 (socket says 172.21.15.1:48122) 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.198+0000 7fd1a200b640 1 -- 172.21.15.1:0/3507018789 learned_addr learned my addr 172.21.15.1:0/3507018789 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a200b640 1 -- 172.21.15.1:0/3507018789 >> v1:172.21.15.1:6793/0 conn(0x7fd19c13f5c0 legacy=0x7fd19c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a1009640 1 -- 172.21.15.1:0/3507018789 >> v1:172.21.15.82:6792/0 conn(0x7fd19c0a6ba0 legacy=0x7fd19c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a180a640 1 -- 172.21.15.1:0/3507018789 >> v1:172.21.15.82:6790/0 conn(0x7fd19c0a67c0 legacy=0x7fd19c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282498072 0 0) 0x7fd19c06ede0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): my global_id is 204218 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd188003520 con 0x7fd19c13f5c0 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581191486 0 0) 0x7fd19c0a8130 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): my global_id is 204199 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c06ede0 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595043540 0 0) 0x7fd19c0a3b40 con 0x7fd19c0a67c0 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient(hunting): my global_id is 184665 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c0a8130 con 0x7fd19c0a67c0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1454477555 0 0) 0x7fd188003520 con 0x7fd19c13f5c0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 >> v1:172.21.15.82:6792/0 conn(0x7fd19c0a6ba0 legacy=0x7fd19c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 >> v1:172.21.15.82:6790/0 conn(0x7fd19c0a67c0 legacy=0x7fd19c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 monclient: found mon.noname-e 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c0a92a0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd194001e50 con 0x7fd19c13f5c0 2024-08-19T10:05:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a3a6e640 5 monclient: authenticate success, global_id 204218 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a3a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: dump: 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.209 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a3a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd194004c20 con 0x7fd19c13f5c0 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a3a6e640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 1 -- 172.21.15.1:0/3507018789 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd194005190 con 0x7fd19c13f5c0 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a3a6e640 4 set_mon_vals no callback set 2024-08-19T10:05:47.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.200+0000 7fd1a0808640 10 monclient: dump: 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.211 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 10 monclient: shutdown 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/3507018789 >> v1:172.21.15.1:6793/0 conn(0x7fd19c13f5c0 legacy=0x7fd19c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/3507018789 shutdown_connections 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/3507018789 >> 172.21.15.1:0/3507018789 conn(0x7fd19c0ac4b0 msgr2=0x7fd19c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/3507018789 shutdown_connections 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/3507018789 wait complete. 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.201+0000 7fd1a3a6e640 10 monclient: monmap: 2024-08-19T10:05:47.212 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.213 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 librados: starting msgr at 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 librados: starting objecter 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 Processor -- start 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 -- start start 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 librados: setting wanted keys 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 librados: calling monclient init 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient: init 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient: _add_conns ranks=[5,0,1,6,4,7,2,8,3] 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient(hunting): picked mon.noname-h con 0x7fd19c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient(hunting): picked mon.noname-a con 0x7fd19c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient(hunting): picked mon.noname-f con 0x7fd19c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c13ae50 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c0a9bc0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c13bcc0 con 0x7fd19c0a67c0 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a3a6e640 10 monclient(hunting): authenticate will time out at 1603.278784s 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a1009640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd19c0a6ba0 0x7fd19c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58722/0 (socket says 172.21.15.1:58722) 2024-08-19T10:05:47.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.202+0000 7fd1a1009640 1 -- 172.21.15.1:0/4116071389 learned_addr learned my addr 172.21.15.1:0/4116071389 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a1009640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6789/0 conn(0x7fd19c0a6ba0 legacy=0x7fd19c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 client.?.objecter ms_handle_connect 0x7fd19c0a6ba0 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 client.?.objecter ms_handle_connect 0x7fd19c0a67c0 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 client.?.objecter ms_handle_connect 0x7fd19c13f5c0 2024-08-19T10:05:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a180a640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.82:6791/0 conn(0x7fd19c0a67c0 legacy=0x7fd19c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a200b640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.82:6789/0 conn(0x7fd19c13f5c0 legacy=0x7fd19c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355903978 0 0) 0x7fd19c13ae50 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): my global_id is 204228 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd180004eb0 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1746960116 0 0) 0x7fd19c13bcc0 con 0x7fd19c0a67c0 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): my global_id is 27878 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c13ae50 con 0x7fd19c0a67c0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2043209969 0 0) 0x7fd19c0a92a0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): my global_id is 174934 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c13bcc0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 851808425 0 0) 0x7fd180004eb0 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd19c0a92a0 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd18c003610 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:47.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): dump: 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2340386752 0 0) 0x7fd19c13ae50 con 0x7fd19c0a67c0 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd180004eb0 con 0x7fd19c0a67c0 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd198003610 con 0x7fd19c0a67c0 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): dump: 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2952718429 0 0) 0x7fd19c13bcc0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd19c13ae50 con 0x7fd19c13f5c0 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd1940016a0 con 0x7fd19c13f5c0 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient(hunting): dump: 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2278698347 0 0) 0x7fd19c0a92a0 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.82:6791/0 conn(0x7fd19c0a67c0 legacy=0x7fd19c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.82:6789/0 conn(0x7fd19c13f5c0 legacy=0x7fd19c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 monclient: found mon.a 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c1cf020 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 5 monclient: authenticate success, global_id 204228 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 10 monclient: _renew_subs 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd19c13b420 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 10 client.204228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 10 monclient: _renew_subs 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd19c13d0f0 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1a3a6e640 1 librados: init done 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd18c003b90 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd18c004080 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd193fff640 4 set_mon_vals no callback set 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.205+0000 7fd1927fc640 10 monclient: dump: 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.232 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1a3a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd1840065c0 con 0x7fd184002860 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd18c004220 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(574..574 src has 299..574) ==== 5923+0+0 (unknown 649153396 0 0) 0x7fd18c04e490 con 0x7fd19c0a6ba0 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1927fc640 10 client.204228.objecter ms_dispatch 0x7fd19c000b90 osd_map(574..574 src has 299..574) 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1927fc640 3 client.204228.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1927fc640 3 client.204228.objecter handle_osd_map decoding full epoch 574 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1927fc640 20 client.204228.objecter dump_active .. 0 homeless 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.206+0000 7fd1a200b640 10 monclient: get_auth_request con 0x7fd180041b30 auth_method 0 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1a180a640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6790/0 conn(0x7fd184002860 legacy=0x7fd184004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 10 client.204228.objecter ms_handle_connect 0x7fd184002860 2024-08-19T10:05:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 10 client.204228.objecter resend_mon_ops 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2219203237 0 0) 0x7fd1840065c0 con 0x7fd184002860 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 10 monclient(hunting): my global_id is 184574 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.208+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c0a92a0 con 0x7fd184002860 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1622304396 0 0) 0x7fd19c0a92a0 con 0x7fd184002860 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1840065c0 con 0x7fd184002860 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd198003e90 con 0x7fd184002860 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 10 monclient: dump: 2024-08-19T10:05:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2021773656 0 0) 0x7fd1840065c0 con 0x7fd184002860 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd18400a160 con 0x7fd184002860 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 10 client.204228.objecter ms_handle_connect 0x7fd180041b30 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd198004180 con 0x7fd184002860 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6790/0 conn(0x7fd184002860 legacy=0x7fd184004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.209+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6790/0 conn(0x7fd184002860 legacy=0x7fd184004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.215+0000 7fd1a3a6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.215+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd184009e50 con 0x7fd184009110 2024-08-19T10:05:47.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1a180a640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6790/0 conn(0x7fd184009110 legacy=0x7fd1840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 client.204228.objecter ms_handle_connect 0x7fd184009110 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 client.204228.objecter resend_mon_ops 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745622984 0 0) 0x7fd184009e50 con 0x7fd184009110 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient(hunting): my global_id is 184583 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1840065c0 con 0x7fd184009110 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2151367028 0 0) 0x7fd1840065c0 con 0x7fd184009110 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd184009e50 con 0x7fd184009110 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd198005b00 con 0x7fd184009110 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient: dump: 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2494809778 0 0) 0x7fd184009e50 con 0x7fd184009110 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd18400c3a0 con 0x7fd184009110 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2879882056) 0x7fd198006ff0 con 0x7fd184009110 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6790/0 conn(0x7fd184009110 legacy=0x7fd1840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1927fc640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6790/0 conn(0x7fd184009110 legacy=0x7fd1840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 264466, 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:47.240 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.242 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.243 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:47.244 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.245 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.246 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.247 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.252 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.253 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.254 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.255 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.257 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.258 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.259 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.260 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.261 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:47.261 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:47.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.218+0000 7fd1a3a6e640 10 librados: watch_flush enter 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 10 librados: watch_flush exit 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 20 client.204228.objecter shutdown clearing up homeless session... 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 10 client.204228.objecter successfully canceled tick 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd180041b30 legacy=0x7fd180043ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 10 monclient: shutdown 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 >> v1:172.21.15.1:6789/0 conn(0x7fd19c0a6ba0 legacy=0x7fd19c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a200b640 1 -- 172.21.15.1:0/4116071389 reap_dead start 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a200b640 1 -- 172.21.15.1:0/4116071389 reap_dead start 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 shutdown_connections 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 >> 172.21.15.1:0/4116071389 conn(0x7fd19c0ac4b0 msgr2=0x7fd19c141c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 shutdown_connections 2024-08-19T10:05:47.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 -- 172.21.15.1:0/4116071389 wait complete. 2024-08-19T10:05:47.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.219+0000 7fd1a3a6e640 1 librados: shutdown 2024-08-19T10:05:47.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:47.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:05:47.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432645046}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371436007}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3637933629}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.285 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3419466446}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3402798690}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1558040892}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=833266732}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254064071}) 2024-08-19T10:05:47.286 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.282+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1854541525}) 2024-08-19T10:05:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.364+0000 7f9f484cc640 10 monclient: get_monmap_and_config 2024-08-19T10:05:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.364+0000 7f9f484cc640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.364+0000 7f9f484cc640 10 monclient: monmap: 2024-08-19T10:05:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:47.365771+0000 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:47.365771+0000 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.367 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.365+0000 7f9f484cc640 1 Processor -- start 2024-08-19T10:05:47.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.365+0000 7f9f484cc640 1 -- start start 2024-08-19T10:05:47.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.365+0000 7f9f484cc640 10 monclient: init 2024-08-19T10:05:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient: _add_conns ranks=[7,5,0,1,6,8,3,2,4] 2024-08-19T10:05:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient(hunting): picked mon.noname-i con 0x7f9f400a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient(hunting): picked mon.noname-h con 0x7f9f400a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f400a3b40 con 0x7f9f4013f5c0 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f400a8130 con 0x7f9f400a6ba0 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f4006ede0 con 0x7f9f400a67c0 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f484cc640 10 monclient(hunting): authenticate will time out at 1603.442367s 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f46a69640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9f4013f5c0 0x7f9f401419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58738/0 (socket says 172.21.15.1:58738) 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f46a69640 1 -- 172.21.15.1:0/667292063 learned_addr learned my addr 172.21.15.1:0/667292063 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f45a67640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f9f400a6ba0 0x7f9f400a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:52734/0 (socket says 172.21.15.1:52734) 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.366+0000 7f9f46268640 1 --1- 172.21.15.1:0/667292063 >> v1:172.21.15.82:6792/0 conn(0x7f9f400a67c0 0x7f9f400a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:36698/0 (socket says 172.21.15.1:36698) 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f46a69640 1 -- 172.21.15.1:0/667292063 >> v1:172.21.15.1:6789/0 conn(0x7f9f4013f5c0 legacy=0x7f9f401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45a67640 1 -- 172.21.15.1:0/667292063 >> v1:172.21.15.82:6791/0 conn(0x7f9f400a6ba0 legacy=0x7f9f400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f46268640 1 -- 172.21.15.1:0/667292063 >> v1:172.21.15.82:6792/0 conn(0x7f9f400a67c0 legacy=0x7f9f400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879628219 0 0) 0x7f9f400a3b40 con 0x7f9f4013f5c0 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): my global_id is 204237 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f2c003520 con 0x7f9f4013f5c0 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421631523 0 0) 0x7f9f400a8130 con 0x7f9f400a6ba0 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): my global_id is 27887 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f400a3b40 con 0x7f9f400a6ba0 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017212586 0 0) 0x7f9f4006ede0 con 0x7f9f400a67c0 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient(hunting): my global_id is 204208 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f400a8130 con 0x7f9f400a67c0 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617559019 0 0) 0x7f9f2c003520 con 0x7f9f4013f5c0 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 >> v1:172.21.15.82:6792/0 conn(0x7f9f400a67c0 legacy=0x7f9f400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 >> v1:172.21.15.82:6791/0 conn(0x7f9f400a6ba0 legacy=0x7f9f400a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 monclient: found mon.noname-a 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f400a92a0 con 0x7f9f4013f5c0 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f38001e60 con 0x7f9f4013f5c0 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f484cc640 5 monclient: authenticate success, global_id 204237 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.368+0000 7f9f484cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: dump: 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f38004c20 con 0x7f9f4013f5c0 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 1 -- 172.21.15.1:0/667292063 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f38005190 con 0x7f9f4013f5c0 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 4 set_mon_vals no callback set 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f45266640 10 monclient: dump: 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 10 monclient: shutdown 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 1 -- 172.21.15.1:0/667292063 >> v1:172.21.15.1:6789/0 conn(0x7f9f4013f5c0 legacy=0x7f9f401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 1 -- 172.21.15.1:0/667292063 shutdown_connections 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 1 -- 172.21.15.1:0/667292063 >> 172.21.15.1:0/667292063 conn(0x7f9f400ac4b0 msgr2=0x7f9f400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 1 -- 172.21.15.1:0/667292063 shutdown_connections 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.369+0000 7f9f484cc640 1 -- 172.21.15.1:0/667292063 wait complete. 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 10 monclient: monmap: 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 1 librados: starting msgr at 2024-08-19T10:05:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 1 librados: starting objecter 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 1 Processor -- start 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 1 -- start start 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 1 librados: setting wanted keys 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 1 librados: calling monclient init 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.370+0000 7f9f484cc640 10 monclient: init 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient: _add_conns ranks=[1,4,6,8,7,2,5,0,3] 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient(hunting): picked mon.noname-f con 0x7f9f400a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient(hunting): picked mon.noname-c con 0x7f9f400a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient(hunting): picked mon.noname-d con 0x7f9f4013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f4013d310 con 0x7f9f400a67c0 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f401cf020 con 0x7f9f400a6ba0 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f4013ae60 con 0x7f9f4013f5c0 2024-08-19T10:05:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f484cc640 10 monclient(hunting): authenticate will time out at 1603.447248s 2024-08-19T10:05:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f45a67640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f9f400a6ba0 0x7f9f401ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50788/0 (socket says 172.21.15.1:50788) 2024-08-19T10:05:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.371+0000 7f9f45a67640 1 -- 172.21.15.1:0/1562662613 learned_addr learned my addr 172.21.15.1:0/1562662613 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f46a69640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6792/0 conn(0x7f9f4013f5c0 legacy=0x7f9f401d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f36ffd640 10 client.?.objecter ms_handle_connect 0x7f9f4013f5c0 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f36ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f36ffd640 10 client.?.objecter ms_handle_connect 0x7f9f400a6ba0 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f36ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f45a67640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f400a6ba0 legacy=0x7f9f401ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f46268640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.82:6789/0 conn(0x7f9f400a67c0 legacy=0x7f9f401ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.373+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 client.?.objecter ms_handle_connect 0x7f9f400a67c0 2024-08-19T10:05:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 274912581 0 0) 0x7f9f401cf020 con 0x7f9f400a6ba0 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): my global_id is 194404 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f24004eb0 con 0x7f9f400a6ba0 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287659029 0 0) 0x7f9f4013ae60 con 0x7f9f4013f5c0 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): my global_id is 204171 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f401cf020 con 0x7f9f4013f5c0 2024-08-19T10:05:47.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482861126 0 0) 0x7f9f400a8130 con 0x7f9f400a67c0 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 10 monclient(hunting): my global_id is 174943 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f4013ae60 con 0x7f9f400a67c0 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3140790772 0 0) 0x7f9f24004eb0 con 0x7f9f400a6ba0 2024-08-19T10:05:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.375+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f400a8130 con 0x7f9f400a6ba0 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3581391242 0 0) 0x7f9f401cf020 con 0x7f9f4013f5c0 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f24004eb0 con 0x7f9f4013f5c0 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f300047e0 con 0x7f9f400a6ba0 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.392 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f380020b0 con 0x7f9f4013f5c0 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1478455951 0 0) 0x7f9f4013ae60 con 0x7f9f400a67c0 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f401cf020 con 0x7f9f400a67c0 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f3c001d10 con 0x7f9f400a67c0 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient(hunting): dump: 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2511765802 0 0) 0x7f9f400a8130 con 0x7f9f400a6ba0 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6792/0 conn(0x7f9f4013f5c0 legacy=0x7f9f401d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.82:6789/0 conn(0x7f9f400a67c0 legacy=0x7f9f401ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 monclient: found mon.c 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f40139970 con 0x7f9f400a6ba0 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 5 monclient: authenticate success, global_id 194404 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 10 monclient: _renew_subs 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4013bcd0 con 0x7f9f400a6ba0 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 10 client.194404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 10 monclient: _renew_subs 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f4013e540 con 0x7f9f400a6ba0 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f484cc640 1 librados: init done 2024-08-19T10:05:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f30004ca0 con 0x7f9f400a6ba0 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f30005190 con 0x7f9f400a6ba0 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f44a65640 4 set_mon_vals no callback set 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.376+0000 7f9f36ffd640 10 monclient: dump: 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9f30005330 con 0x7f9f400a6ba0 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f484cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f280065c0 con 0x7f9f28002860 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(574..574 src has 299..574) ==== 5923+0+0 (unknown 649153396 0 0) 0x7f9f3004e410 con 0x7f9f400a6ba0 2024-08-19T10:05:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f36ffd640 10 client.194404.objecter ms_dispatch 0x7f9f40000b90 osd_map(574..574 src has 299..574) 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f36ffd640 3 client.194404.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f36ffd640 3 client.194404.objecter handle_osd_map decoding full epoch 574 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f36ffd640 20 client.194404.objecter dump_active .. 0 homeless 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.377+0000 7f9f46a69640 10 monclient: get_auth_request con 0x7f9f24041a50 auth_method 0 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f46268640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28002860 legacy=0x7f9f28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 client.194404.objecter ms_handle_connect 0x7f9f28002860 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 client.194404.objecter resend_mon_ops 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570185305 0 0) 0x7f9f280065c0 con 0x7f9f28002860 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient(hunting): my global_id is 194413 2024-08-19T10:05:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f400a8130 con 0x7f9f28002860 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2225733289 0 0) 0x7f9f400a8130 con 0x7f9f28002860 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f280065c0 con 0x7f9f28002860 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f3c004d90 con 0x7f9f28002860 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 monclient: dump: 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 10 client.194404.objecter ms_handle_connect 0x7f9f24041a50 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4235208037 0 0) 0x7f9f280065c0 con 0x7f9f28002860 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.379+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f2800a160 con 0x7f9f28002860 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.380+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9f3c005120 con 0x7f9f28002860 2024-08-19T10:05:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.380+0000 7f9f36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.380+0000 7f9f36ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.380+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28002860 legacy=0x7f9f28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.380+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28002860 legacy=0x7f9f28004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.386+0000 7f9f484cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.386+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f28009e50 con 0x7f9f28009110 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f46268640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 legacy=0x7f9f280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 client.194404.objecter ms_handle_connect 0x7f9f28009110 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 client.194404.objecter resend_mon_ops 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294045112 0 0) 0x7f9f28009e50 con 0x7f9f28009110 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient(hunting): my global_id is 194422 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f280065c0 con 0x7f9f28009110 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 344898333 0 0) 0x7f9f280065c0 con 0x7f9f28009110 2024-08-19T10:05:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f28009e50 con 0x7f9f28009110 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f3c005640 con 0x7f9f28009110 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 10 monclient: dump: 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1768595214 0 0) 0x7f9f28009e50 con 0x7f9f28009110 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f2800c3f0 con 0x7f9f28009110 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f46268640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 legacy=0x7f9f280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f46268640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 legacy=0x7f9f280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f46268640 1 --1- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 0x7f9f280095a0 :-1 s=OPENED pgs=3775 cs=1 l=1).handle_message read tag failed 2024-08-19T10:05:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f46268640 1 --1- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 0x7f9f280095a0 :-1 s=OPENED pgs=3775 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 legacy=0x7f9f280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f28009110 legacy=0x7f9f280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.389+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f28009e50 con 0x7f9f2404cae0 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f46268640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f2404cae0 legacy=0x7f9f2404b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 10 client.194404.objecter ms_handle_connect 0x7f9f2404cae0 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 10 client.194404.objecter resend_mon_ops 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070223899 0 0) 0x7f9f28009e50 con 0x7f9f2404cae0 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.391+0000 7f9f36ffd640 10 monclient(hunting): my global_id is 194431 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f280065c0 con 0x7f9f2404cae0 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 198943566 0 0) 0x7f9f280065c0 con 0x7f9f2404cae0 2024-08-19T10:05:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f28009e50 con 0x7f9f2404cae0 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f3c0063a0 con 0x7f9f2404cae0 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 10 monclient: dump: 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3403869403 0 0) 0x7f9f28009e50 con 0x7f9f2404cae0 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f3c005640 con 0x7f9f2404cae0 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2272972824) 0x7f9f3c007850 con 0x7f9f2404cae0 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f2404cae0 legacy=0x7f9f2404b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f36ffd640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f2404cae0 legacy=0x7f9f2404b470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f46a69640 1 -- 172.21.15.1:0/1562662613 reap_dead start 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 135079, 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:47.413 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:47.414 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.415 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.416 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:47.417 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.418 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:47.433 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:47.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 10 librados: watch_flush enter 2024-08-19T10:05:47.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 10 librados: watch_flush exit 2024-08-19T10:05:47.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 20 client.194404.objecter shutdown clearing up homeless session... 2024-08-19T10:05:47.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 10 client.194404.objecter successfully canceled tick 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9f24041a50 legacy=0x7f9f24043f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 10 monclient: shutdown 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 >> v1:172.21.15.1:6791/0 conn(0x7f9f400a6ba0 legacy=0x7f9f401ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 shutdown_connections 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.392+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 >> 172.21.15.1:0/1562662613 conn(0x7f9f400ac4b0 msgr2=0x7f9f4013edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.393+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 shutdown_connections 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.393+0000 7f9f484cc640 1 -- 172.21.15.1:0/1562662613 wait complete. 2024-08-19T10:05:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.393+0000 7f9f484cc640 1 librados: shutdown 2024-08-19T10:05:47.436 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:47.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:05:47.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:05:47.568 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806192786}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2287125231}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1145603451}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2817894692}) 2024-08-19T10:05:47.569 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1169599832}) 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=951502836}) 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3386200700}) 2024-08-19T10:05:47.570 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845768500}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.554+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=311905412}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1947404224}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3527676710}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.571 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1128468929}) 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2510985753}) 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2190079928}) 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.572 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4047697248}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4084339555}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680404844}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.555+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=584859802}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4261000356}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.573 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874545143}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3883645134}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147478585}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1409233086}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.556+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3316377700}) 2024-08-19T10:05:47.574 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.557+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.557+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.557+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1259917649}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.557+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.557+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494331627}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.557+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2039937419}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325992189}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730272142}) 2024-08-19T10:05:47.575 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3942474317}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4247108035}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3149861619}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2911014247}) 2024-08-19T10:05:47.576 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1251486154}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8322998}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.558+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1822888153}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3012027709}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223957446}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.577 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533528535}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3390106844}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2541745667}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1245161309}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.578 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2734661128}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938674495}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.559+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4248880287}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.560+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.560+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879240937}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931863245}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1879502806}) 2024-08-19T10:05:47.579 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1486445256}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1699551962}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1313289623}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.580 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2524008344}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395518829}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.561+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=775035824}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511093899}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615206347}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.581 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3177526771}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2166260881}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2689561928}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326360187}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3545104628}) 2024-08-19T10:05:47.582 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.583 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647332}) 2024-08-19T10:05:47.583 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.562+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1565090493}) 2024-08-19T10:05:47.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.579+0000 7f59f12c5640 10 monclient: get_monmap_and_config 2024-08-19T10:05:47.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.579+0000 7f59f12c5640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.579+0000 7f59f12c5640 10 monclient: monmap: 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:47.580005+0000 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:47.580005+0000 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.579+0000 7f59f12c5640 1 Processor -- start 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.579+0000 7f59f12c5640 1 -- start start 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.579+0000 7f59f12c5640 10 monclient: init 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient: _add_conns ranks=[3,8,1,5,4,0,6,7,2] 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient(hunting): picked mon.noname-g con 0x7f59ec0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient(hunting): picked mon.noname-e con 0x7f59ec0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient(hunting): picked mon.noname-f con 0x7f59ec0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec0a3c70 con 0x7f59ec0a46a0 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec0a33d0 con 0x7f59ec0a6830 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec0a36b0 con 0x7f59ec0694b0 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59f12c5640 10 monclient(hunting): authenticate will time out at 1603.656192s 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59ea7fc640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f59ec0694b0 0x7f59ec0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48136/0 (socket says 172.21.15.1:48136) 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.580+0000 7f59ea7fc640 1 -- 172.21.15.1:0/936969807 learned_addr learned my addr 172.21.15.1:0/936969807 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59ea7fc640 1 -- 172.21.15.1:0/936969807 >> v1:172.21.15.1:6793/0 conn(0x7f59ec0694b0 legacy=0x7f59ec0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4176031801 0 0) 0x7f59ec0a36b0 con 0x7f59ec0694b0 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient(hunting): my global_id is 204227 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59eaffd640 1 -- 172.21.15.1:0/936969807 >> v1:172.21.15.82:6790/0 conn(0x7f59ec0a6830 legacy=0x7f59ec0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d8003520 con 0x7f59ec0694b0 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59eb7fe640 1 -- 172.21.15.1:0/936969807 >> v1:172.21.15.82:6789/0 conn(0x7f59ec0a46a0 legacy=0x7f59ec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2474560398 0 0) 0x7f59d8003520 con 0x7f59ec0694b0 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 >> v1:172.21.15.82:6790/0 conn(0x7f59ec0a6830 legacy=0x7f59ec0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 >> v1:172.21.15.82:6789/0 conn(0x7f59ec0a46a0 legacy=0x7f59ec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 monclient: found mon.noname-e 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec06dbb0 con 0x7f59ec0694b0 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59d4001ae0 con 0x7f59ec0694b0 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: dump: 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59f12c5640 5 monclient: authenticate success, global_id 204227 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59f12c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59d40048a0 con 0x7f59ec0694b0 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 1 -- 172.21.15.1:0/936969807 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59d4004e10 con 0x7f59ec0694b0 2024-08-19T10:05:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59f12c5640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59f12c5640 4 set_mon_vals no callback set 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.582+0000 7f59e9ffb640 10 monclient: dump: 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 10 monclient: shutdown 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 1 -- 172.21.15.1:0/936969807 >> v1:172.21.15.1:6793/0 conn(0x7f59ec0694b0 legacy=0x7f59ec0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 1 -- 172.21.15.1:0/936969807 shutdown_connections 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 1 -- 172.21.15.1:0/936969807 >> 172.21.15.1:0/936969807 conn(0x7f59ec0a9020 msgr2=0x7f59ec068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 1 -- 172.21.15.1:0/936969807 shutdown_connections 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.583+0000 7f59f12c5640 1 -- 172.21.15.1:0/936969807 wait complete. 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 10 monclient: monmap: 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.604 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 1 librados: starting msgr at 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 1 librados: starting objecter 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 1 Processor -- start 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 1 -- start start 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 1 librados: setting wanted keys 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 1 librados: calling monclient init 2024-08-19T10:05:47.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.584+0000 7f59f12c5640 10 monclient: init 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient: _add_conns ranks=[7,4,3,2,1,0,8,6,5] 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient(hunting): picked mon.noname-i con 0x7f59ec0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient(hunting): picked mon.noname-c con 0x7f59ec0a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient(hunting): picked mon.noname-g con 0x7f59ec0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec142a40 con 0x7f59ec0a6830 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec138660 con 0x7f59ec0a46a0 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec138940 con 0x7f59ec0694b0 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59ea7fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f59ec0a46a0 0x7f59ec1ceac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50828/0 (socket says 172.21.15.1:50828) 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59ea7fc640 1 -- 172.21.15.1:0/846199733 learned_addr learned my addr 172.21.15.1:0/846199733 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.585+0000 7f59f12c5640 10 monclient(hunting): authenticate will time out at 1603.661215s 2024-08-19T10:05:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59ea7fc640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6791/0 conn(0x7f59ec0a46a0 legacy=0x7f59ec1ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 client.?.objecter ms_handle_connect 0x7f59ec0a46a0 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 client.?.objecter ms_handle_connect 0x7f59ec0694b0 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59eaffd640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.82:6792/0 conn(0x7f59ec0694b0 legacy=0x7f59ec1ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59eb7fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.82:6790/0 conn(0x7f59ec0a6830 legacy=0x7f59ec1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 client.?.objecter ms_handle_connect 0x7f59ec0a6830 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1463183471 0 0) 0x7f59ec138660 con 0x7f59ec0a46a0 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): my global_id is 194440 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59cc004eb0 con 0x7f59ec0a46a0 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651678628 0 0) 0x7f59ec138940 con 0x7f59ec0694b0 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): my global_id is 204217 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ec138660 con 0x7f59ec0694b0 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2293869083 0 0) 0x7f59dc001560 con 0x7f59ec0a6830 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): my global_id is 184683 2024-08-19T10:05:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ec138940 con 0x7f59ec0a6830 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2359950221 0 0) 0x7f59cc004eb0 con 0x7f59ec0a46a0 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59dc001560 con 0x7f59ec0a46a0 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59d4004d90 con 0x7f59ec0a46a0 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): dump: 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 680860208 0 0) 0x7f59ec138940 con 0x7f59ec0a6830 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59cc004eb0 con 0x7f59ec0a6830 2024-08-19T10:05:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3629487097 0 0) 0x7f59ec138660 con 0x7f59ec0694b0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59ec138940 con 0x7f59ec0694b0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3236579742 0 0) 0x7f59dc001560 con 0x7f59ec0a46a0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.82:6792/0 conn(0x7f59ec0694b0 legacy=0x7f59ec1ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.587+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.82:6790/0 conn(0x7f59ec0a6830 legacy=0x7f59ec1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 1 monclient: found mon.c 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec1cf1d0 con 0x7f59ec0a46a0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 5 monclient: authenticate success, global_id 194440 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 10 monclient: _renew_subs 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59ec0a9f70 con 0x7f59ec0a46a0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 10 client.194440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 10 monclient: _renew_subs 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f59ec138380 con 0x7f59ec0a46a0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 1 librados: init done 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59d4001d30 con 0x7f59ec0a46a0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59d4001b00 con 0x7f59ec0a46a0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e97fa640 4 set_mon_vals no callback set 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 10 monclient: dump: 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59d00065c0 con 0x7f59d0002860 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.588+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f59d4011a00 con 0x7f59ec0a46a0 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.589+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(574..574 src has 299..574) ==== 5923+0+0 (unknown 649153396 0 0) 0x7f59d404e430 con 0x7f59ec0a46a0 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.589+0000 7f59e37fe640 10 client.194440.objecter ms_dispatch 0x7f59ec000b90 osd_map(574..574 src has 299..574) 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.589+0000 7f59e37fe640 3 client.194440.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T10:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.589+0000 7f59e37fe640 3 client.194440.objecter handle_osd_map decoding full epoch 574 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.589+0000 7f59e37fe640 20 client.194440.objecter dump_active .. 0 homeless 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.589+0000 7f59eb7fe640 10 monclient: get_auth_request con 0x7f59cc0413b0 auth_method 0 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 client.194440.objecter ms_handle_connect 0x7f59d0002860 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59eaffd640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6792/0 conn(0x7f59d0002860 legacy=0x7f59d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 client.194440.objecter resend_mon_ops 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047178230 0 0) 0x7f59d00065c0 con 0x7f59d0002860 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient(hunting): my global_id is 204180 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59dc001560 con 0x7f59d0002860 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4025792920 0 0) 0x7f59dc001560 con 0x7f59d0002860 2024-08-19T10:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59d00065c0 con 0x7f59d0002860 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59e4003f80 con 0x7f59d0002860 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient: dump: 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 448196854 0 0) 0x7f59d00065c0 con 0x7f59d0002860 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59d000a160 con 0x7f59d0002860 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 client.194440.objecter ms_handle_connect 0x7f59cc0413b0 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f59e4005b40 con 0x7f59d0002860 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6792/0 conn(0x7f59d0002860 legacy=0x7f59d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.591+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6792/0 conn(0x7f59d0002860 legacy=0x7f59d0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.598+0000 7f59f12c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.598+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59d0009e70 con 0x7f59d0009130 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59eaffd640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6792/0 conn(0x7f59d0009130 legacy=0x7f59d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 client.194440.objecter ms_handle_connect 0x7f59d0009130 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 client.194440.objecter resend_mon_ops 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 852073606 0 0) 0x7f59d0009e70 con 0x7f59d0009130 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient(hunting): my global_id is 204189 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d00065c0 con 0x7f59d0009130 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3005167554 0 0) 0x7f59d00065c0 con 0x7f59d0009130 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59d0009e70 con 0x7f59d0009130 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f59e4005db0 con 0x7f59d0009130 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 10 monclient: dump: 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1333217990 0 0) 0x7f59d0009e70 con 0x7f59d0009130 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.600+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59d000c410 con 0x7f59d0009130 2024-08-19T10:05:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 1142134339) 0x7f59e4008030 con 0x7f59d0009130 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59e37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59e37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6792/0 conn(0x7f59d0009130 legacy=0x7f59d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59e37fe640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6792/0 conn(0x7f59d0009130 legacy=0x7f59d00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:47.623 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13470, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.624 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.625 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.626 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:47.627 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.629 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.630 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.633 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.634 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.637 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:47.639 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.642 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:47.643 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:47.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 10 librados: watch_flush enter 2024-08-19T10:05:47.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 10 librados: watch_flush exit 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 20 client.194440.objecter shutdown clearing up homeless session... 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 10 client.194440.objecter successfully canceled tick 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.82:6812/2549104415 conn(0x7f59cc0413b0 legacy=0x7f59cc043870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 10 monclient: shutdown 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 >> v1:172.21.15.1:6791/0 conn(0x7f59ec0a46a0 legacy=0x7f59ec1ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59eb7fe640 1 -- 172.21.15.1:0/846199733 reap_dead start 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59eb7fe640 1 -- 172.21.15.1:0/846199733 reap_dead start 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 shutdown_connections 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 >> 172.21.15.1:0/846199733 conn(0x7f59ec0a9020 msgr2=0x7f59ec0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 shutdown_connections 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 -- 172.21.15.1:0/846199733 wait complete. 2024-08-19T10:05:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.601+0000 7f59f12c5640 1 librados: shutdown 2024-08-19T10:05:47.647 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:47.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:05:47.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.746+0000 7ff2ea6c5640 10 monclient: get_monmap_and_config 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.746+0000 7ff2ea6c5640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.746+0000 7ff2ea6c5640 10 monclient: monmap: 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:47.747026+0000 2024-08-19T10:05:47.747 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:47.747026+0000 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.748 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.746+0000 7ff2ea6c5640 1 Processor -- start 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.746+0000 7ff2ea6c5640 1 -- start start 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.746+0000 7ff2ea6c5640 10 monclient: init 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient: _add_conns ranks=[8,2,7,6,3,1,5,4,0] 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient(hunting): picked mon.noname-e con 0x7ff2e40a6830 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e40694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient(hunting): picked mon.noname-i con 0x7ff2e40a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e40a3c70 con 0x7ff2e40694b0 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e40a33d0 con 0x7ff2e40a46a0 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e40a36b0 con 0x7ff2e40a6830 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2ea6c5640 10 monclient(hunting): authenticate will time out at 1603.823371s 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2e37fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7ff2e40694b0 0x7ff2e40a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33068/0 (socket says 172.21.15.1:33068) 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2e3fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 0x7ff2e40a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48150/0 (socket says 172.21.15.1:48150) 2024-08-19T10:05:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.747+0000 7ff2e37fe640 1 -- 172.21.15.1:0/2745206570 learned_addr learned my addr 172.21.15.1:0/2745206570 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e37fe640 1 -- 172.21.15.1:0/2745206570 >> v1:172.21.15.1:6790/0 conn(0x7ff2e40694b0 legacy=0x7ff2e40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e3fff640 1 -- 172.21.15.1:0/2745206570 >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 legacy=0x7ff2e40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e8c62640 1 -- 172.21.15.1:0/2745206570 >> v1:172.21.15.82:6792/0 conn(0x7ff2e40a46a0 legacy=0x7ff2e4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533932716 0 0) 0x7ff2e40a3c70 con 0x7ff2e40694b0 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): my global_id is 184592 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2cc003520 con 0x7ff2e40694b0 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997771769 0 0) 0x7ff2e40a36b0 con 0x7ff2e40a6830 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): my global_id is 204236 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e40a3c70 con 0x7ff2e40a6830 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1656889343 0 0) 0x7ff2e40a33d0 con 0x7ff2e40a46a0 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 10 monclient(hunting): my global_id is 204226 2024-08-19T10:05:47.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.749+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e40a36b0 con 0x7ff2e40a46a0 2024-08-19T10:05:47.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e3fff640 1 --1- 172.21.15.1:0/2745206570 >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 0x7ff2e40a6c10 :-1 s=OPENED pgs=3623 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7ff2d4001ac0 mon_map magic: 0 2024-08-19T10:05:47.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3487077547 0 0) 0x7ff2cc003520 con 0x7ff2e40694b0 2024-08-19T10:05:47.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 legacy=0x7ff2e40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 >> v1:172.21.15.82:6792/0 conn(0x7ff2e40a46a0 legacy=0x7ff2e4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 monclient: found mon.noname-b 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e406dbb0 con 0x7ff2e40694b0 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2d8001ae0 con 0x7ff2e40694b0 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 5 monclient: authenticate success, global_id 184592 2024-08-19T10:05:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: dump: 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.767 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2d4001ac0 con 0x7ff2e40a6830 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2d80048a0 con 0x7ff2e40694b0 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/2745206570 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2d8004e10 con 0x7ff2e40694b0 2024-08-19T10:05:47.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 4 set_mon_vals no callback set 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2e2ffd640 10 monclient: dump: 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.769 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 10 monclient: shutdown 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/2745206570 >> v1:172.21.15.1:6790/0 conn(0x7ff2e40694b0 legacy=0x7ff2e40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/2745206570 shutdown_connections 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/2745206570 >> 172.21.15.1:0/2745206570 conn(0x7ff2e40a9020 msgr2=0x7ff2e4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/2745206570 shutdown_connections 2024-08-19T10:05:47.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.750+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/2745206570 wait complete. 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 10 monclient: monmap: 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.771 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 1 librados: starting msgr at 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 1 librados: starting objecter 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 1 Processor -- start 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 1 -- start start 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 1 librados: setting wanted keys 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 1 librados: calling monclient init 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.751+0000 7ff2ea6c5640 10 monclient: init 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient: _add_conns ranks=[3,1,8,7,4,6,5,0,2] 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient(hunting): picked mon.noname-g con 0x7ff2e40694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient(hunting): picked mon.noname-f con 0x7ff2e40a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient(hunting): picked mon.noname-e con 0x7ff2e40a6830 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e413b930 con 0x7ff2e40a46a0 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e41caeb0 con 0x7ff2e40694b0 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e41395d0 con 0x7ff2e40a6830 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2ea6c5640 10 monclient(hunting): authenticate will time out at 1603.828490s 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2e8c62640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 0x7ff2e41d6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48166/0 (socket says 172.21.15.1:48166) 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.752+0000 7ff2e8c62640 1 -- 172.21.15.1:0/1181021668 learned_addr learned my addr 172.21.15.1:0/1181021668 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e8c62640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 legacy=0x7ff2e41d6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 client.?.objecter ms_handle_connect 0x7ff2e40a6830 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e3fff640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.82:6790/0 conn(0x7ff2e40694b0 legacy=0x7ff2e4141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 client.?.objecter ms_handle_connect 0x7ff2e40694b0 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846988224 0 0) 0x7ff2e41395d0 con 0x7ff2e40a6830 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): my global_id is 204245 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e37fe640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.82:6789/0 conn(0x7ff2e40a46a0 legacy=0x7ff2e41d2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d0004eb0 con 0x7ff2e40a6830 2024-08-19T10:05:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 client.?.objecter ms_handle_connect 0x7ff2e40a46a0 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 48849959 0 0) 0x7ff2e41caeb0 con 0x7ff2e40694b0 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 10 monclient(hunting): my global_id is 184692 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.754+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e41395d0 con 0x7ff2e40694b0 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215701343 0 0) 0x7ff2e413b930 con 0x7ff2e40a46a0 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): my global_id is 174961 2024-08-19T10:05:47.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e41caeb0 con 0x7ff2e40a46a0 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1294071879 0 0) 0x7ff2d0004eb0 con 0x7ff2e40a6830 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2e413b930 con 0x7ff2e40a6830 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2dc003cd0 con 0x7ff2e40a6830 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): dump: 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 638427793 0 0) 0x7ff2e413b930 con 0x7ff2e40a6830 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.82:6790/0 conn(0x7ff2e40694b0 legacy=0x7ff2e4141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.82:6789/0 conn(0x7ff2e40a46a0 legacy=0x7ff2e41d2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 monclient: found mon.e 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e413b090 con 0x7ff2e40a6830 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 5 monclient: authenticate success, global_id 204245 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 10 monclient: _renew_subs 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e413b370 con 0x7ff2e40a6830 2024-08-19T10:05:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 10 client.204245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:47.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 10 monclient: _renew_subs 2024-08-19T10:05:47.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:47.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2e413a130 con 0x7ff2e40a6830 2024-08-19T10:05:47.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2ea6c5640 1 librados: init done 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2dc003e70 con 0x7ff2e40a6830 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2dc0043e0 con 0x7ff2e40a6830 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e27fc640 4 set_mon_vals no callback set 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.755+0000 7ff2e0ff9640 10 monclient: dump: 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2ea6c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff2c40065c0 con 0x7ff2c4002860 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff2dc004680 con 0x7ff2e40a6830 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(574..574 src has 299..574) ==== 5923+0+0 (unknown 649153396 0 0) 0x7ff2dc04e7e0 con 0x7ff2e40a6830 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e0ff9640 10 client.204245.objecter ms_dispatch 0x7ff2e4000b90 osd_map(574..574 src has 299..574) 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e0ff9640 3 client.204245.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e0ff9640 3 client.204245.objecter handle_osd_map decoding full epoch 574 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e0ff9640 20 client.204245.objecter dump_active .. 0 homeless 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.756+0000 7ff2e37fe640 10 monclient: get_auth_request con 0x7ff2d0041140 auth_method 0 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e3fff640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2c4002860 legacy=0x7ff2c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 client.204245.objecter ms_handle_connect 0x7ff2c4002860 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 client.204245.objecter resend_mon_ops 2024-08-19T10:05:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3388644596 0 0) 0x7ff2c40065c0 con 0x7ff2c4002860 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient(hunting): my global_id is 204254 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e413b930 con 0x7ff2c4002860 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1199914100 0 0) 0x7ff2e413b930 con 0x7ff2c4002860 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2c40065c0 con 0x7ff2c4002860 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2d40047e0 con 0x7ff2c4002860 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 10 monclient: dump: 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.784 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 392767624 0 0) 0x7ff2c40065c0 con 0x7ff2c4002860 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.758+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2c400a160 con 0x7ff2c4002860 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.759+0000 7ff2e0ff9640 10 client.204245.objecter ms_handle_connect 0x7ff2d0041140 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.759+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff2d4004910 con 0x7ff2c4002860 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.759+0000 7ff2e0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.759+0000 7ff2e0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.759+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2c4002860 legacy=0x7ff2c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.759+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2c4002860 legacy=0x7ff2c4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.765+0000 7ff2ea6c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.765+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff2c4009e70 con 0x7ff2c4009130 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e3fff640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2c4009130 legacy=0x7ff2c40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 client.204245.objecter ms_handle_connect 0x7ff2c4009130 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 client.204245.objecter resend_mon_ops 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3296835858 0 0) 0x7ff2c4009e70 con 0x7ff2c4009130 2024-08-19T10:05:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient(hunting): my global_id is 204263 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c40065c0 con 0x7ff2c4009130 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2269207042 0 0) 0x7ff2c40065c0 con 0x7ff2c4009130 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2c4009e70 con 0x7ff2c4009130 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff2d4005100 con 0x7ff2c4009130 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.767+0000 7ff2e0ff9640 10 monclient: dump: 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.789 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3164046706 0 0) 0x7ff2c4009e70 con 0x7ff2c4009130 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff2c400c410 con 0x7ff2c4009130 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3540656216) 0x7ff2d4007460 con 0x7ff2c4009130 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2c4009130 legacy=0x7ff2c40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2c4009130 legacy=0x7ff2c40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13640, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:47.791 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.792 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.794 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.794 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.794 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.794 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:47.795 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:47.796 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.797 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.798 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.799 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.802 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.803 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:47.804 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:47.805 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.806 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:47.808 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:47.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.813 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:47.813 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 10 librados: watch_flush enter 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 10 librados: watch_flush exit 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 20 client.204245.objecter shutdown clearing up homeless session... 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 10 client.204245.objecter successfully canceled tick 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff2d0041140 legacy=0x7ff2d0043600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 10 monclient: shutdown 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 >> v1:172.21.15.1:6793/0 conn(0x7ff2e40a6830 legacy=0x7ff2e41d6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e8c62640 1 -- 172.21.15.1:0/1181021668 reap_dead start 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2e8c62640 1 -- 172.21.15.1:0/1181021668 reap_dead start 2024-08-19T10:05:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 shutdown_connections 2024-08-19T10:05:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 >> 172.21.15.1:0/1181021668 conn(0x7ff2e40a9020 msgr2=0x7ff2e40a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 shutdown_connections 2024-08-19T10:05:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 -- 172.21.15.1:0/1181021668 wait complete. 2024-08-19T10:05:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.768+0000 7ff2ea6c5640 1 librados: shutdown 2024-08-19T10:05:47.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:47.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:05:47.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703770368}) 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563357135}) 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.881 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2267095687}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.878+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=337516122}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1522030362}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=779563511}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2107848561}) 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.882 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3359710249}) 2024-08-19T10:05:47.883 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.879+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2041537139}) 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810651791}) 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4072583332}) 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.898 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=180309341}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3150653773}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=106638181}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4201275874}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=855839961}) 2024-08-19T10:05:47.899 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.900 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087847084}) 2024-08-19T10:05:47.900 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.895+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=794711749}) 2024-08-19T10:05:47.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.900+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2183944931}) 2024-08-19T10:05:47.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.902 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121019452}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3548400281}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4269707879}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3278068825}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.903 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=813259638}) 2024-08-19T10:05:47.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4087565235}) 2024-08-19T10:05:47.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218483581}) 2024-08-19T10:05:47.904 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1576626747}) 2024-08-19T10:05:47.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1496415379}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4033633580}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4191948857}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=44876873}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3439742616}) 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=426286885}) 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1999202398}) 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761538141}) 2024-08-19T10:05:47.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:47.913+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1922864287}) 2024-08-19T10:05:47.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.933+0000 7f01eefba640 10 monclient: get_monmap_and_config 2024-08-19T10:05:47.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.933+0000 7f01eefba640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.933+0000 7f01eefba640 10 monclient: monmap: 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:47.934551+0000 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:47.934551+0000 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.935 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.936 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 1 Processor -- start 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 1 -- start start 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient: init 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient: _add_conns ranks=[6,3,2,4,8,1,7,5,0] 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient(hunting): picked mon.noname-d con 0x7f01e80a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient(hunting): picked mon.noname-g con 0x7f01e80694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient(hunting): picked mon.noname-b con 0x7f01e80a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80a3c70 con 0x7f01e80a46a0 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80a33d0 con 0x7f01e80694b0 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80a36b0 con 0x7f01e80a6830 2024-08-19T10:05:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.934+0000 7f01eefba640 10 monclient(hunting): authenticate will time out at 1604.010931s 2024-08-19T10:05:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.935+0000 7f01ecd56640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f01e80a6830 0x7f01e80a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:41936/0 (socket says 172.21.15.1:41936) 2024-08-19T10:05:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.935+0000 7f01ed557640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f01e80a46a0 0x7f01e8141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33092/0 (socket says 172.21.15.1:33092) 2024-08-19T10:05:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.935+0000 7f01ecd56640 1 -- 172.21.15.1:0/258930443 learned_addr learned my addr 172.21.15.1:0/258930443 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01ecd56640 1 -- 172.21.15.1:0/258930443 >> v1:172.21.15.1:6792/0 conn(0x7f01e80a6830 legacy=0x7f01e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01ed557640 1 -- 172.21.15.1:0/258930443 >> v1:172.21.15.1:6790/0 conn(0x7f01e80a46a0 legacy=0x7f01e8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01dffff640 1 -- 172.21.15.1:0/258930443 >> v1:172.21.15.82:6790/0 conn(0x7f01e80694b0 legacy=0x7f01e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455962259 0 0) 0x7f01e80a36b0 con 0x7f01e80a6830 2024-08-19T10:05:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): my global_id is 204198 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d0003520 con 0x7f01e80a6830 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745801943 0 0) 0x7f01e80a3c70 con 0x7f01e80a46a0 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): my global_id is 184601 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e80a36b0 con 0x7f01e80a46a0 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473830173 0 0) 0x7f01e80a33d0 con 0x7f01e80694b0 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient(hunting): my global_id is 184701 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e80a3c70 con 0x7f01e80694b0 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 757203651 0 0) 0x7f01e80a36b0 con 0x7f01e80a46a0 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 >> v1:172.21.15.1:6792/0 conn(0x7f01e80a6830 legacy=0x7f01e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 >> v1:172.21.15.82:6790/0 conn(0x7f01e80694b0 legacy=0x7f01e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 monclient: found mon.noname-b 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e806dbb0 con 0x7f01e80a46a0 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e0001e40 con 0x7f01e80a46a0 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01eefba640 5 monclient: authenticate success, global_id 184601 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: dump: 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01eefba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01e0004b60 con 0x7f01e80a46a0 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 1 -- 172.21.15.1:0/258930443 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e00050d0 con 0x7f01e80a46a0 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01df7fe640 10 monclient: dump: 2024-08-19T10:05:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01eefba640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.937+0000 7f01eefba640 4 set_mon_vals no callback set 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 10 monclient: shutdown 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 1 -- 172.21.15.1:0/258930443 >> v1:172.21.15.1:6790/0 conn(0x7f01e80a46a0 legacy=0x7f01e8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 1 -- 172.21.15.1:0/258930443 shutdown_connections 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 1 -- 172.21.15.1:0/258930443 >> 172.21.15.1:0/258930443 conn(0x7f01e80a9020 msgr2=0x7f01e8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 1 -- 172.21.15.1:0/258930443 shutdown_connections 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.938+0000 7f01eefba640 1 -- 172.21.15.1:0/258930443 wait complete. 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 10 monclient: build_initial_monmap 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 10 monclient: monmap: 2024-08-19T10:05:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 1 librados: starting msgr at 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 1 librados: starting objecter 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 1 Processor -- start 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 1 -- start start 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 1 librados: setting wanted keys 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 1 librados: calling monclient init 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 10 monclient: init 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 10 monclient: _add_conns ranks=[2,7,6,3,8,5,1,0,4] 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.939+0000 7f01eefba640 10 monclient(hunting): picked mon.noname-b con 0x7f01e80694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 10 monclient(hunting): picked mon.noname-i con 0x7f01e80a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 10 monclient(hunting): picked mon.noname-d con 0x7f01e80a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e8138e00 con 0x7f01e80694b0 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e8135e10 con 0x7f01e80a6830 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80a9d30 con 0x7f01e80a46a0 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01eefba640 10 monclient(hunting): authenticate will time out at 1604.016093s 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01ecd56640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f01e80694b0 0x7f01e8134ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33106/0 (socket says 172.21.15.1:33106) 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.940+0000 7f01ecd56640 1 -- 172.21.15.1:0/4265436179 learned_addr learned my addr 172.21.15.1:0/4265436179 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01ecd56640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.1:6790/0 conn(0x7f01e80694b0 legacy=0x7f01e8134ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 client.?.objecter ms_handle_connect 0x7f01e80694b0 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 client.?.objecter ms_handle_connect 0x7f01e80a6830 2024-08-19T10:05:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01ed557640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.1:6792/0 conn(0x7f01e80a6830 legacy=0x7f01e81d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dffff640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6792/0 conn(0x7f01e80a46a0 legacy=0x7f01e8135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659345667 0 0) 0x7f01e8138e00 con 0x7f01e80694b0 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): my global_id is 184610 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d4004eb0 con 0x7f01e80694b0 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 client.?.objecter ms_handle_connect 0x7f01e80a46a0 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3117024588 0 0) 0x7f01e8135e10 con 0x7f01e80a6830 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): my global_id is 204207 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e8138e00 con 0x7f01e80a6830 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476003087 0 0) 0x7f01e80a9d30 con 0x7f01e80a46a0 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): my global_id is 204235 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e8135e10 con 0x7f01e80a46a0 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3466119895 0 0) 0x7f01e8138e00 con 0x7f01e80a6830 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01e80a9d30 con 0x7f01e80a6830 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e0005050 con 0x7f01e80a6830 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): dump: 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.961 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2235336162 0 0) 0x7f01d4004eb0 con 0x7f01e80694b0 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01e8138e00 con 0x7f01e80694b0 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e40019d0 con 0x7f01e80694b0 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient(hunting): dump: 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.962 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 868542444 0 0) 0x7f01e80a9d30 con 0x7f01e80a6830 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6792/0 conn(0x7f01e80a46a0 legacy=0x7f01e8135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.1:6790/0 conn(0x7f01e80694b0 legacy=0x7f01e8134ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 monclient: found mon.d 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e8137f90 con 0x7f01e80a6830 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient: _finish_auth 0 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01eefba640 5 monclient: authenticate success, global_id 204207 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01eefba640 10 monclient: _renew_subs 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01eefba640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.942+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01e8136aa0 con 0x7f01e80a6830 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 10 client.204207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 10 monclient: _renew_subs 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f01e81426e0 con 0x7f01e80a6830 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 1 librados: init done 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01e0004b60 con 0x7f01e80a6830 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e0002090 con 0x7f01e80a6830 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01deffd640 4 set_mon_vals no callback set 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 10 monclient: dump: 2024-08-19T10:05:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01c80065c0 con 0x7f01c8002860 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.943+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f01e0011cc0 con 0x7f01e80a6830 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.944+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7f01e004e5c0 con 0x7f01e80a6830 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.944+0000 7f01dd7fa640 10 client.204207.objecter ms_dispatch 0x7f01e8000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.944+0000 7f01dd7fa640 3 client.204207.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.944+0000 7f01dd7fa640 3 client.204207.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.944+0000 7f01dd7fa640 20 client.204207.objecter dump_active .. 0 homeless 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.944+0000 7f01dffff640 10 monclient: get_auth_request con 0x7f01d4041370 auth_method 0 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 client.204207.objecter ms_handle_connect 0x7f01c8002860 2024-08-19T10:05:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 client.204207.objecter resend_mon_ops 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01ecd56640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6789/0 conn(0x7f01c8002860 legacy=0x7f01c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257539089 0 0) 0x7f01c80065c0 con 0x7f01c8002860 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient(hunting): my global_id is 174970 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e80a9d30 con 0x7f01c8002860 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1531562180 0 0) 0x7f01e80a9d30 con 0x7f01c8002860 2024-08-19T10:05:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01c80065c0 con 0x7f01c8002860 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 client.204207.objecter ms_handle_connect 0x7f01d4041370 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e4005300 con 0x7f01c8002860 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 10 monclient: dump: 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3919665366 0 0) 0x7f01c80065c0 con 0x7f01c8002860 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.946+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01c800a160 con 0x7f01c8002860 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.947+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f01e40075a0 con 0x7f01c8002860 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.947+0000 7f01dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.947+0000 7f01dd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.947+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6789/0 conn(0x7f01c8002860 legacy=0x7f01c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.947+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6789/0 conn(0x7f01c8002860 legacy=0x7f01c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.953+0000 7f01eefba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.953+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01c8009e30 con 0x7f01c8009110 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01ecd56640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6789/0 conn(0x7f01c8009110 legacy=0x7f01c8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 client.204207.objecter ms_handle_connect 0x7f01c8009110 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 client.204207.objecter resend_mon_ops 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617663303 0 0) 0x7f01c8009e30 con 0x7f01c8009110 2024-08-19T10:05:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient(hunting): my global_id is 174979 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c80065c0 con 0x7f01c8009110 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3998307944 0 0) 0x7f01c80065c0 con 0x7f01c8009110 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01c8009e30 con 0x7f01c8009110 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f01e4006e90 con 0x7f01c8009110 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 10 monclient: dump: 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3248781793 0 0) 0x7f01c8009e30 con 0x7f01c8009110 2024-08-19T10:05:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.956+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01c800c3f0 con 0x7f01c8009110 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2265331917) 0x7f01e4006fc0 con 0x7f01c8009110 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01dd7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6789/0 conn(0x7f01c8009110 legacy=0x7f01c8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01dd7fa640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6789/0 conn(0x7f01c8009110 legacy=0x7f01c8009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.976 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 398014, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.978 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:47.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:47.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:47.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:47.998 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:47.998 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 10 librados: watch_flush enter 2024-08-19T10:05:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 10 librados: watch_flush exit 2024-08-19T10:05:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 20 client.204207.objecter shutdown clearing up homeless session... 2024-08-19T10:05:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 10 client.204207.objecter successfully canceled tick 2024-08-19T10:05:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.82:6812/2549104415 conn(0x7f01d4041370 legacy=0x7f01d4043830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 10 monclient: shutdown 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 >> v1:172.21.15.1:6792/0 conn(0x7f01e80a6830 legacy=0x7f01e81d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01ed557640 1 -- 172.21.15.1:0/4265436179 reap_dead start 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01ed557640 1 -- 172.21.15.1:0/4265436179 reap_dead start 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 shutdown_connections 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 >> 172.21.15.1:0/4265436179 conn(0x7f01e80a9020 msgr2=0x7f01e80a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 shutdown_connections 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 -- 172.21.15.1:0/4265436179 wait complete. 2024-08-19T10:05:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:47.957+0000 7f01eefba640 1 librados: shutdown 2024-08-19T10:05:48.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:48.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:05:48.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:05:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.100+0000 7fbbd4c7e640 10 monclient: get_monmap_and_config 2024-08-19T10:05:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.100+0000 7fbbd4c7e640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.100+0000 7fbbd4c7e640 10 monclient: monmap: 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:48.101484+0000 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:48.101484+0000 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.103 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 1 Processor -- start 2024-08-19T10:05:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 1 -- start start 2024-08-19T10:05:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient: init 2024-08-19T10:05:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient: _add_conns ranks=[0,8,5,7,2,4,3,6,1] 2024-08-19T10:05:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient(hunting): picked mon.noname-a con 0x7fbbd00a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient(hunting): picked mon.noname-e con 0x7fbbd00a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient(hunting): picked mon.noname-h con 0x7fbbd013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd00a3b40 con 0x7fbbd00a67c0 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd00a8130 con 0x7fbbd013f5c0 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd006ede0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbd4c7e640 10 monclient(hunting): authenticate will time out at 1604.177787s 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbcdd9b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6ba0 0x7fbbd00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48196/0 (socket says 172.21.15.1:48196) 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbce59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbbd00a67c0 0x7fbbd00a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58760/0 (socket says 172.21.15.1:58760) 2024-08-19T10:05:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.101+0000 7fbbcdd9b640 1 -- 172.21.15.1:0/498931900 learned_addr learned my addr 172.21.15.1:0/498931900 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbce59c640 1 -- 172.21.15.1:0/498931900 >> v1:172.21.15.1:6789/0 conn(0x7fbbd00a67c0 legacy=0x7fbbd00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcdd9b640 1 -- 172.21.15.1:0/498931900 >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6ba0 legacy=0x7fbbd00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10670934 0 0) 0x7fbbd00a3b40 con 0x7fbbd00a67c0 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): my global_id is 204246 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb8003520 con 0x7fbbd00a67c0 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126170824 0 0) 0x7fbbd006ede0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient(hunting): my global_id is 204272 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd00a3b40 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbced9d640 1 -- 172.21.15.1:0/498931900 >> v1:172.21.15.82:6791/0 conn(0x7fbbd013f5c0 legacy=0x7fbbd01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1155885539 0 0) 0x7fbbd00a3b40 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 >> v1:172.21.15.82:6791/0 conn(0x7fbbd013f5c0 legacy=0x7fbbd01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 >> v1:172.21.15.1:6789/0 conn(0x7fbbd00a67c0 legacy=0x7fbbd00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 monclient: found mon.noname-e 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd00a92a0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbd4c7e640 5 monclient: authenticate success, global_id 204272 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc4001940 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbd4c7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: dump: 2024-08-19T10:05:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbd4c7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbc4001c10 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 1 -- 172.21.15.1:0/498931900 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc4004be0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbd4c7e640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbd4c7e640 4 set_mon_vals no callback set 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.104+0000 7fbbcd59a640 10 monclient: dump: 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.114 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 10 monclient: shutdown 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/498931900 >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6ba0 legacy=0x7fbbd00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/498931900 shutdown_connections 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/498931900 >> 172.21.15.1:0/498931900 conn(0x7fbbd00ac4b0 msgr2=0x7fbbd00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/498931900 shutdown_connections 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/498931900 wait complete. 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.105+0000 7fbbd4c7e640 10 monclient: monmap: 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 librados: starting msgr at 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 librados: starting objecter 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 Processor -- start 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 -- start start 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 librados: setting wanted keys 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 librados: calling monclient init 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient: init 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient: _add_conns ranks=[5,8,4,1,3,0,2,7,6] 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient(hunting): picked mon.noname-h con 0x7fbbd00a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient(hunting): picked mon.noname-e con 0x7fbbd00a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient(hunting): picked mon.noname-c con 0x7fbbd013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd0139ce0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd013a830 con 0x7fbbd00a67c0 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbd013f310 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbd4c7e640 10 monclient(hunting): authenticate will time out at 1604.182739s 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbcdd9b640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6ba0 0x7fbbd01d3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48208/0 (socket says 172.21.15.1:48208) 2024-08-19T10:05:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.106+0000 7fbbcdd9b640 1 -- 172.21.15.1:0/1767811229 learned_addr learned my addr 172.21.15.1:0/1767811229 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.108+0000 7fbbcdd9b640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6ba0 legacy=0x7fbbd01d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.108+0000 7fbbced9d640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.1:6791/0 conn(0x7fbbd013f5c0 legacy=0x7fbbd01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.108+0000 7fbbbeffd640 10 client.?.objecter ms_handle_connect 0x7fbbd00a6ba0 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.108+0000 7fbbbeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.108+0000 7fbbbeffd640 10 client.?.objecter ms_handle_connect 0x7fbbd013f5c0 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.108+0000 7fbbbeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448482267 0 0) 0x7fbbd0139ce0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): my global_id is 194449 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbce59c640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6791/0 conn(0x7fbbd00a67c0 legacy=0x7fbbd01ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb0004eb0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 client.?.objecter ms_handle_connect 0x7fbbd00a67c0 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528366639 0 0) 0x7fbbd013f310 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): my global_id is 204281 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd0139ce0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381687360 0 0) 0x7fbbd013a830 con 0x7fbbd00a67c0 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): my global_id is 27905 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd013f310 con 0x7fbbd00a67c0 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 363580119 0 0) 0x7fbbb0004eb0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbd013a830 con 0x7fbbd013f5c0 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc8002040 con 0x7fbbd013f5c0 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): dump: 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.128 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.129 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3839764124 0 0) 0x7fbbd0139ce0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbb0004eb0 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc4004b20 con 0x7fbbd00a6ba0 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): dump: 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.130 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3623823247 0 0) 0x7fbbd013f310 con 0x7fbbd00a67c0 2024-08-19T10:05:48.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbd0139ce0 con 0x7fbbd00a67c0 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc0001320 con 0x7fbbd00a67c0 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient(hunting): dump: 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.132 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2735981206 0 0) 0x7fbbd013a830 con 0x7fbbd013f5c0 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.1:6793/0 conn(0x7fbbd00a6ba0 legacy=0x7fbbd01d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6791/0 conn(0x7fbbd00a67c0 legacy=0x7fbbd01ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 monclient: found mon.c 2024-08-19T10:05:48.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd00a9d30 con 0x7fbbd013f5c0 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 5 monclient: authenticate success, global_id 194449 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 10 monclient: _renew_subs 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd0142bf0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 10 client.194449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 10 monclient: _renew_subs 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbd00a0d30 con 0x7fbbd013f5c0 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbd4c7e640 1 librados: init done 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbc8003c40 con 0x7fbbd013f5c0 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc8004170 con 0x7fbbd013f5c0 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbccd99640 4 set_mon_vals no callback set 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.109+0000 7fbbbeffd640 10 monclient: dump: 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.135 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.110+0000 7fbbd4c7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.110+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbbb40065c0 con 0x7fbbb4002860 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.110+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbbc80043c0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.111+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7fbbc804e5f0 con 0x7fbbd013f5c0 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.111+0000 7fbbbeffd640 10 client.194449.objecter ms_dispatch 0x7fbbd0000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.111+0000 7fbbbeffd640 3 client.194449.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.111+0000 7fbbbeffd640 3 client.194449.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.111+0000 7fbbbeffd640 20 client.194449.objecter dump_active .. 0 homeless 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.111+0000 7fbbcdd9b640 10 monclient: get_auth_request con 0x7fbbb0041a40 auth_method 0 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 client.194449.objecter ms_handle_connect 0x7fbbb4002860 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbce59c640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6790/0 conn(0x7fbbb4002860 legacy=0x7fbbb4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 client.194449.objecter resend_mon_ops 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145369424 0 0) 0x7fbbb40065c0 con 0x7fbbb4002860 2024-08-19T10:05:48.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient(hunting): my global_id is 184710 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd013a830 con 0x7fbbb4002860 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 308452774 0 0) 0x7fbbd013a830 con 0x7fbbb4002860 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbb40065c0 con 0x7fbbb4002860 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 client.194449.objecter ms_handle_connect 0x7fbbb0041a40 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc0004f20 con 0x7fbbb4002860 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 10 monclient: dump: 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.138 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.139 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3045541112 0 0) 0x7fbbb40065c0 con 0x7fbbb4002860 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.113+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbb400a160 con 0x7fbbb4002860 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.114+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbbc0006460 con 0x7fbbb4002860 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.114+0000 7fbbbeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.114+0000 7fbbbeffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.114+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6790/0 conn(0x7fbbb4002860 legacy=0x7fbbb4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.114+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6790/0 conn(0x7fbbb4002860 legacy=0x7fbbb4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.120+0000 7fbbd4c7e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.120+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbbb4009e50 con 0x7fbbb4009110 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.122+0000 7fbbce59c640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6790/0 conn(0x7fbbb4009110 legacy=0x7fbbb40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.122+0000 7fbbbeffd640 10 client.194449.objecter ms_handle_connect 0x7fbbb4009110 2024-08-19T10:05:48.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.122+0000 7fbbbeffd640 10 client.194449.objecter resend_mon_ops 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 906694966 0 0) 0x7fbbb4009e50 con 0x7fbbb4009110 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient(hunting): my global_id is 184719 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb40065c0 con 0x7fbbb4009110 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2021944927 0 0) 0x7fbbb40065c0 con 0x7fbbb4009110 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbb4009e50 con 0x7fbbb4009110 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbbc00054d0 con 0x7fbbb4009110 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 10 monclient: dump: 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.141 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.142 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3631129128 0 0) 0x7fbbb4009e50 con 0x7fbbb4009110 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.123+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbbb400c3f0 con 0x7fbbb4009110 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 2899107604) 0x7fbbc00058c0 con 0x7fbbb4009110 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbbeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbbeffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6790/0 conn(0x7fbbb4009110 legacy=0x7fbbb40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbbeffd640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6790/0 conn(0x7fbbb4009110 legacy=0x7fbbb40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:48.143 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 265377, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:48.144 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.145 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.146 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.147 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:48.148 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.149 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.151 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:48.156 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:48.157 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:48.158 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.159 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.160 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.162 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.166 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:48.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.168 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:48.168 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 10 librados: watch_flush enter 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 10 librados: watch_flush exit 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 20 client.194449.objecter shutdown clearing up homeless session... 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 10 client.194449.objecter successfully canceled tick 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbbb0041a40 legacy=0x7fbbb0043f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 10 monclient: shutdown 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 >> v1:172.21.15.1:6791/0 conn(0x7fbbd013f5c0 legacy=0x7fbbd01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbced9d640 1 -- 172.21.15.1:0/1767811229 reap_dead start 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbced9d640 1 -- 172.21.15.1:0/1767811229 reap_dead start 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 shutdown_connections 2024-08-19T10:05:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 >> 172.21.15.1:0/1767811229 conn(0x7fbbd00ac4b0 msgr2=0x7fbbd00a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 shutdown_connections 2024-08-19T10:05:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 -- 172.21.15.1:0/1767811229 wait complete. 2024-08-19T10:05:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.124+0000 7fbbd4c7e640 1 librados: shutdown 2024-08-19T10:05:48.171 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:48.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:05:48.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:05:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3659760913}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445557224}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993956015}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1644117204}) 2024-08-19T10:05:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2014785289}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1793099245}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2087984666}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636792820}) 2024-08-19T10:05:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3884341208}) 2024-08-19T10:05:48.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.271+0000 7fa684174640 10 monclient: get_monmap_and_config 2024-08-19T10:05:48.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.271+0000 7fa684174640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.271+0000 7fa684174640 10 monclient: monmap: 2024-08-19T10:05:48.272 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.272 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.272 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:48.272370+0000 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:48.272370+0000 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.273 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.271+0000 7fa684174640 1 Processor -- start 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 1 -- start start 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient: init 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient: _add_conns ranks=[4,1,8,0,6,7,3,5,2] 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient(hunting): picked mon.noname-c con 0x7fa67c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient(hunting): picked mon.noname-f con 0x7fa67c0a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient(hunting): picked mon.noname-e con 0x7fa67c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:48.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c0a3b40 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c0a8130 con 0x7fa67c0a67c0 2024-08-19T10:05:48.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c06ede0 con 0x7fa67c13f5c0 2024-08-19T10:05:48.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa684174640 10 monclient(hunting): authenticate will time out at 1604.348683s 2024-08-19T10:05:48.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa681f10640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fa67c0a67c0 0x7fa67c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50846/0 (socket says 172.21.15.1:50846) 2024-08-19T10:05:48.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.272+0000 7fa681f10640 1 -- 172.21.15.1:0/2478694056 learned_addr learned my addr 172.21.15.1:0/2478694056 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.274+0000 7fa682711640 1 -- 172.21.15.1:0/2478694056 >> v1:172.21.15.1:6793/0 conn(0x7fa67c13f5c0 legacy=0x7fa67c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.274+0000 7fa681f10640 1 -- 172.21.15.1:0/2478694056 >> v1:172.21.15.1:6791/0 conn(0x7fa67c0a67c0 legacy=0x7fa67c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa68170f640 1 -- 172.21.15.1:0/2478694056 >> v1:172.21.15.82:6789/0 conn(0x7fa67c0a6ba0 legacy=0x7fa67c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919934642 0 0) 0x7fa67c0a8130 con 0x7fa67c0a67c0 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): my global_id is 194458 2024-08-19T10:05:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa664003520 con 0x7fa67c0a67c0 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057922873 0 0) 0x7fa67c06ede0 con 0x7fa67c13f5c0 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): my global_id is 204290 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa67c0a8130 con 0x7fa67c13f5c0 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 73605762 0 0) 0x7fa67c0a3b40 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient(hunting): my global_id is 174988 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa67c06ede0 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1042862219 0 0) 0x7fa664003520 con 0x7fa67c0a67c0 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 >> v1:172.21.15.1:6793/0 conn(0x7fa67c13f5c0 legacy=0x7fa67c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 >> v1:172.21.15.82:6789/0 conn(0x7fa67c0a6ba0 legacy=0x7fa67c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 monclient: found mon.noname-c 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c0a92a0 con 0x7fa67c0a67c0 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa66c001e60 con 0x7fa67c0a67c0 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa684174640 5 monclient: authenticate success, global_id 194458 2024-08-19T10:05:48.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa684174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: dump: 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.283 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa684174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2398659456 0 0) 0x7fa67c0a8130 con 0x7fa67c13f5c0 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa674001aa0 con 0x7fa67c13f5c0 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa66c004c20 con 0x7fa67c0a67c0 2024-08-19T10:05:48.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 1 -- 172.21.15.1:0/2478694056 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa66c005190 con 0x7fa67c0a67c0 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa684174640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa684174640 4 set_mon_vals no callback set 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.275+0000 7fa680f0e640 10 monclient: dump: 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.285 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 10 monclient: shutdown 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 -- 172.21.15.1:0/2478694056 >> v1:172.21.15.1:6791/0 conn(0x7fa67c0a67c0 legacy=0x7fa67c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 -- 172.21.15.1:0/2478694056 shutdown_connections 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 -- 172.21.15.1:0/2478694056 >> 172.21.15.1:0/2478694056 conn(0x7fa67c0ac4b0 msgr2=0x7fa67c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 -- 172.21.15.1:0/2478694056 shutdown_connections 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 -- 172.21.15.1:0/2478694056 wait complete. 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 10 monclient: monmap: 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.287 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 librados: starting msgr at 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 librados: starting objecter 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.276+0000 7fa684174640 1 Processor -- start 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 1 -- start start 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 1 librados: setting wanted keys 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 1 librados: calling monclient init 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient: init 2024-08-19T10:05:48.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient: _add_conns ranks=[7,8,6,4,3,1,0,5,2] 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient(hunting): picked mon.noname-i con 0x7fa67c0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient(hunting): picked mon.noname-e con 0x7fa67c0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient(hunting): picked mon.noname-d con 0x7fa67c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c0a1870 con 0x7fa67c13f5c0 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c142bd0 con 0x7fa67c0a67c0 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c13a9b0 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa684174640 10 monclient(hunting): authenticate will time out at 1604.353503s 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa68170f640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fa67c0a6ba0 0x7fa67c0a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48230/0 (socket says 172.21.15.1:48230) 2024-08-19T10:05:48.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.277+0000 7fa68170f640 1 -- 172.21.15.1:0/3307077085 learned_addr learned my addr 172.21.15.1:0/3307077085 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa682711640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.1:6792/0 conn(0x7fa67c13f5c0 legacy=0x7fa67c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 client.?.objecter ms_handle_connect 0x7fa67c13f5c0 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa68170f640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.1:6793/0 conn(0x7fa67c0a6ba0 legacy=0x7fa67c0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 client.?.objecter ms_handle_connect 0x7fa67c0a6ba0 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249091355 0 0) 0x7fa67c0a1870 con 0x7fa67c13f5c0 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): my global_id is 204216 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa668004eb0 con 0x7fa67c13f5c0 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397090068 0 0) 0x7fa67c13a9b0 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 monclient(hunting): my global_id is 204299 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa681f10640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6792/0 conn(0x7fa67c0a67c0 legacy=0x7fa67c0a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa67c0a1870 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 client.?.objecter ms_handle_connect 0x7fa67c0a67c0 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.279+0000 7fa67a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8987877 0 0) 0x7fa67c142bd0 con 0x7fa67c0a67c0 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): my global_id is 204244 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa67c13a9b0 con 0x7fa67c0a67c0 2024-08-19T10:05:48.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2749990828 0 0) 0x7fa668004eb0 con 0x7fa67c13f5c0 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa67c142bd0 con 0x7fa67c13f5c0 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1972485099 0 0) 0x7fa67c0a1870 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa668004eb0 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa674003950 con 0x7fa67c13f5c0 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): dump: 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa670003610 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient(hunting): dump: 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.300 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1083229617 0 0) 0x7fa668004eb0 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6792/0 conn(0x7fa67c0a67c0 legacy=0x7fa67c0a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.1:6792/0 conn(0x7fa67c13f5c0 legacy=0x7fa67c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 monclient: found mon.e 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c0a2890 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 5 monclient: authenticate success, global_id 204299 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 10 monclient: _renew_subs 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c13a110 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 10 client.204299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 10 monclient: _renew_subs 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa67c13f290 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa684174640 1 librados: init done 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa670003ba0 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa670004110 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67bfff640 4 set_mon_vals no callback set 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.280+0000 7fa67a7fc640 10 monclient: dump: 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa684174640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa65c0065c0 con 0x7fa65c002860 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa670011a80 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7fa67004e450 con 0x7fa67c0a6ba0 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa67a7fc640 10 client.204299.objecter ms_dispatch 0x7fa67c000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:05:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa67a7fc640 3 client.204299.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa67a7fc640 3 client.204299.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa67a7fc640 20 client.204299.objecter dump_active .. 0 homeless 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.281+0000 7fa682711640 10 monclient: get_auth_request con 0x7fa668041440 auth_method 0 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa681f10640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6791/0 conn(0x7fa65c002860 legacy=0x7fa65c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 10 client.204299.objecter ms_handle_connect 0x7fa65c002860 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 10 client.204299.objecter resend_mon_ops 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371887764 0 0) 0x7fa65c0065c0 con 0x7fa65c002860 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 10 monclient(hunting): my global_id is 27914 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.283+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa668004eb0 con 0x7fa65c002860 2024-08-19T10:05:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 10 client.204299.objecter ms_handle_connect 0x7fa668041440 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 812675569 0 0) 0x7fa668004eb0 con 0x7fa65c002860 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa65c0065c0 con 0x7fa65c002860 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa66c0020b0 con 0x7fa65c002860 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 10 monclient: dump: 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3144220001 0 0) 0x7fa65c0065c0 con 0x7fa65c002860 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa65c00a160 con 0x7fa65c002860 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa66c005810 con 0x7fa65c002860 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6791/0 conn(0x7fa65c002860 legacy=0x7fa65c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.284+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6791/0 conn(0x7fa65c002860 legacy=0x7fa65c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.290+0000 7fa684174640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.290+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa65c009e70 con 0x7fa65c009130 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa681f10640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6791/0 conn(0x7fa65c009130 legacy=0x7fa65c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 client.204299.objecter ms_handle_connect 0x7fa65c009130 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 client.204299.objecter resend_mon_ops 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142260871 0 0) 0x7fa65c009e70 con 0x7fa65c009130 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient(hunting): my global_id is 27923 2024-08-19T10:05:48.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa65c0065c0 con 0x7fa65c009130 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4042924842 0 0) 0x7fa65c0065c0 con 0x7fa65c009130 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa65c009e70 con 0x7fa65c009130 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa66c005f80 con 0x7fa65c009130 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 10 monclient: dump: 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.309 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2492047330 0 0) 0x7fa65c009e70 con 0x7fa65c009130 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.293+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa65c00c410 con 0x7fa65c009130 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 985066832) 0x7fa66c0082a0 con 0x7fa65c009130 2024-08-19T10:05:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa67a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa67a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6791/0 conn(0x7fa65c009130 legacy=0x7fa65c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa67a7fc640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6791/0 conn(0x7fa65c009130 legacy=0x7fa65c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 954558, 2024-08-19T10:05:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:48.332 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 10 librados: watch_flush enter 2024-08-19T10:05:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 10 librados: watch_flush exit 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 20 client.204299.objecter shutdown clearing up homeless session... 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 10 client.204299.objecter successfully canceled tick 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa668041440 legacy=0x7fa668043900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 10 monclient: shutdown 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 >> v1:172.21.15.1:6793/0 conn(0x7fa67c0a6ba0 legacy=0x7fa67c0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa682711640 1 -- 172.21.15.1:0/3307077085 reap_dead start 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa682711640 1 -- 172.21.15.1:0/3307077085 reap_dead start 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 shutdown_connections 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 >> 172.21.15.1:0/3307077085 conn(0x7fa67c0ac4b0 msgr2=0x7fa67c138750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 shutdown_connections 2024-08-19T10:05:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 -- 172.21.15.1:0/3307077085 wait complete. 2024-08-19T10:05:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.294+0000 7fa684174640 1 librados: shutdown 2024-08-19T10:05:48.335 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:48.335 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:05:48.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:05:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.434+0000 7fa0fb59e640 10 monclient: get_monmap_and_config 2024-08-19T10:05:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.434+0000 7fa0fb59e640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.434+0000 7fa0fb59e640 10 monclient: monmap: 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:48.435465+0000 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:48.435465+0000 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.437 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 1 Processor -- start 2024-08-19T10:05:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 1 -- start start 2024-08-19T10:05:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient: init 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient: _add_conns ranks=[3,7,6,0,8,4,1,2,5] 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient(hunting): picked mon.noname-g con 0x7fa0fc06d8b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient(hunting): picked mon.noname-i con 0x7fa0fc0a5740 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient(hunting): picked mon.noname-d con 0x7fa0fc0a5b60 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0fc136750 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0fc09dbf0 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0fc09ded0 con 0x7fa0fc0a5740 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.435+0000 7fa0fb59e640 10 monclient(hunting): authenticate will time out at 1604.511933s 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.436+0000 7fa0fad9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fa0fc0a5b60 0x7fa0fc141890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:41964/0 (socket says 172.21.15.1:41964) 2024-08-19T10:05:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.436+0000 7fa0fad9d640 1 -- 172.21.15.1:0/4119554100 learned_addr learned my addr 172.21.15.1:0/4119554100 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0fad9d640 1 -- 172.21.15.1:0/4119554100 >> v1:172.21.15.1:6792/0 conn(0x7fa0fc0a5b60 legacy=0x7fa0fc141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770614043 0 0) 0x7fa0fc09dbf0 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient(hunting): my global_id is 204225 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0e4003520 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0fa59c640 1 -- 172.21.15.1:0/4119554100 >> v1:172.21.15.82:6790/0 conn(0x7fa0fc06d8b0 legacy=0x7fa0fc136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f9d9b640 1 -- 172.21.15.1:0/4119554100 >> v1:172.21.15.82:6792/0 conn(0x7fa0fc0a5740 legacy=0x7fa0fc13e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3927040261 0 0) 0x7fa0e4003520 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 >> v1:172.21.15.82:6792/0 conn(0x7fa0fc0a5740 legacy=0x7fa0fc13e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 >> v1:172.21.15.82:6790/0 conn(0x7fa0fc06d8b0 legacy=0x7fa0fc136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 monclient: found mon.noname-d 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0fc09e1b0 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0f4001e60 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: dump: 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0fb59e640 5 monclient: authenticate success, global_id 204225 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0fb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa0f4004c20 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 1 -- 172.21.15.1:0/4119554100 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0f4005190 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0fb59e640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0fb59e640 4 set_mon_vals no callback set 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.438+0000 7fa0f959a640 10 monclient: dump: 2024-08-19T10:05:48.447 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 10 monclient: shutdown 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4119554100 >> v1:172.21.15.1:6792/0 conn(0x7fa0fc0a5b60 legacy=0x7fa0fc141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4119554100 shutdown_connections 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4119554100 >> 172.21.15.1:0/4119554100 conn(0x7fa0fc0ac090 msgr2=0x7fa0fc0a8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4119554100 shutdown_connections 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.439+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4119554100 wait complete. 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient: monmap: 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 librados: starting msgr at 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 librados: starting objecter 2024-08-19T10:05:48.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 Processor -- start 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 -- start start 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 librados: setting wanted keys 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 librados: calling monclient init 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient: init 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient: _add_conns ranks=[4,3,7,1,6,8,5,0,2] 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7fa0fc06d8b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient(hunting): picked mon.noname-g con 0x7fa0fc0a5740 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient(hunting): picked mon.noname-i con 0x7fa0fc0a5b60 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0fc0aa5e0 con 0x7fa0fc0a5740 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0fc0a0de0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0fc0a28a0 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fb59e640 10 monclient(hunting): authenticate will time out at 1604.516885s 2024-08-19T10:05:48.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fa59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fa0fc06d8b0 0x7fa0fc1ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:50848/0 (socket says 172.21.15.1:50848) 2024-08-19T10:05:48.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.440+0000 7fa0fa59c640 1 -- 172.21.15.1:0/4263887954 learned_addr learned my addr 172.21.15.1:0/4263887954 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fa59c640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.1:6791/0 conn(0x7fa0fc06d8b0 legacy=0x7fa0fc1ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 client.?.objecter ms_handle_connect 0x7fa0fc06d8b0 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1826436975 0 0) 0x7fa0fc0a0de0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): my global_id is 194467 2024-08-19T10:05:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fad9d640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0fc0a5b60 legacy=0x7fa0fc1d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0f9d9b640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6790/0 conn(0x7fa0fc0a5740 legacy=0x7fa0fc1d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0dc004eb0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 client.?.objecter ms_handle_connect 0x7fa0fc0a5b60 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 client.?.objecter ms_handle_connect 0x7fa0fc0a5740 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698253948 0 0) 0x7fa0fc0a28a0 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): my global_id is 204262 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0fc0a0de0 con 0x7fa0fc0a5b60 2024-08-19T10:05:48.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348446259 0 0) 0x7fa0fc0aa5e0 con 0x7fa0fc0a5740 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): my global_id is 184737 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0fc0a28a0 con 0x7fa0fc0a5740 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2487916413 0 0) 0x7fa0dc004eb0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0fc0aa5e0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0ec003610 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): dump: 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.461 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.462 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3752039434 0 0) 0x7fa0fc0aa5e0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0fc0a5b60 legacy=0x7fa0fc1d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6790/0 conn(0x7fa0fc0a5740 legacy=0x7fa0fc1d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 monclient: found mon.c 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0fc0a2b80 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0f0001c00 con 0x7fa0fc0a5740 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 5 monclient: authenticate success, global_id 194467 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 10 monclient: _renew_subs 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa0fc0a1940 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 10 client.194467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 10 monclient: _renew_subs 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa0fc0a3c70 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0fb59e640 1 librados: init done 2024-08-19T10:05:48.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa0ec003b90 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0ec0040c0 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0ebfff640 4 set_mon_vals no callback set 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.443+0000 7fa0eaffd640 10 monclient: dump: 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.444+0000 7fa0fb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.444+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa0e00065c0 con 0x7fa0e0002860 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.444+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa0ec004260 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.445+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7fa0ec04e490 con 0x7fa0fc06d8b0 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.445+0000 7fa0eaffd640 10 client.194467.objecter ms_dispatch 0x7fa0fc000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.445+0000 7fa0eaffd640 3 client.194467.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.445+0000 7fa0eaffd640 3 client.194467.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.445+0000 7fa0eaffd640 20 client.194467.objecter dump_active .. 0 homeless 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.445+0000 7fa0fad9d640 10 monclient: get_auth_request con 0x7fa0dc040fa0 auth_method 0 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.446+0000 7fa0f9d9b640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0e0002860 legacy=0x7fa0e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.446+0000 7fa0eaffd640 10 client.194467.objecter ms_handle_connect 0x7fa0e0002860 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.446+0000 7fa0eaffd640 10 client.194467.objecter resend_mon_ops 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651809447 0 0) 0x7fa0e00065c0 con 0x7fa0e0002860 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient(hunting): my global_id is 204271 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0fc0aa5e0 con 0x7fa0e0002860 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3674880253 0 0) 0x7fa0fc0aa5e0 con 0x7fa0e0002860 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0e00065c0 con 0x7fa0e0002860 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0f0003cd0 con 0x7fa0e0002860 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient: dump: 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 client.194467.objecter ms_handle_connect 0x7fa0dc040fa0 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 884961865 0 0) 0x7fa0e00065c0 con 0x7fa0e0002860 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0e000a160 con 0x7fa0e0002860 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa0f0003f40 con 0x7fa0e0002860 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0e0002860 legacy=0x7fa0e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.447+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0e0002860 legacy=0x7fa0e0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.454+0000 7fa0fb59e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.454+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa0e0009e50 con 0x7fa0e0009110 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0f9d9b640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0e0009110 legacy=0x7fa0e00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 10 client.194467.objecter ms_handle_connect 0x7fa0e0009110 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 10 client.194467.objecter resend_mon_ops 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1259293898 0 0) 0x7fa0e0009e50 con 0x7fa0e0009110 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 10 monclient(hunting): my global_id is 204280 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.456+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0e00065c0 con 0x7fa0e0009110 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1953615608 0 0) 0x7fa0e00065c0 con 0x7fa0e0009110 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0e0009e50 con 0x7fa0e0009110 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa0f0004470 con 0x7fa0e0009110 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:05:48.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 10 monclient: dump: 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 168570957 0 0) 0x7fa0e0009e50 con 0x7fa0e0009110 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa0e000c3f0 con 0x7fa0e0009110 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 247255170) 0x7fa0f0005de0 con 0x7fa0e0009110 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0e0009110 legacy=0x7fa0e00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0eaffd640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6792/0 conn(0x7fa0e0009110 legacy=0x7fa0e00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14327, 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:05:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:05:48.477 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:05:48.478 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:05:48.479 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.481 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.483 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.484 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.485 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:05:48.491 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:05:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:05:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:05:48.494 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:05:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 10 librados: watch_flush enter 2024-08-19T10:05:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 10 librados: watch_flush exit 2024-08-19T10:05:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 20 client.194467.objecter shutdown clearing up homeless session... 2024-08-19T10:05:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 10 client.194467.objecter successfully canceled tick 2024-08-19T10:05:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa0dc040fa0 legacy=0x7fa0dc043460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 10 monclient: shutdown 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 >> v1:172.21.15.1:6791/0 conn(0x7fa0fc06d8b0 legacy=0x7fa0fc1ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.457+0000 7fa0fad9d640 1 -- 172.21.15.1:0/4263887954 reap_dead start 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.458+0000 7fa0fad9d640 1 -- 172.21.15.1:0/4263887954 reap_dead start 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.458+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 shutdown_connections 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.458+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 >> 172.21.15.1:0/4263887954 conn(0x7fa0fc0ac090 msgr2=0x7fa0fc0ab070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.458+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 shutdown_connections 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.458+0000 7fa0fb59e640 1 -- 172.21.15.1:0/4263887954 wait complete. 2024-08-19T10:05:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.458+0000 7fa0fb59e640 1 librados: shutdown 2024-08-19T10:05:48.497 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:05:48.498 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['h', 'g'] 2024-08-19T10:05:48.498 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-19T10:05:48.498 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-08-19T10:05:48.498 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-08-19T10:05:48.498 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-08-19T10:05:48.498 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:05:48.519 INFO:tasks.ceph.mon.h:Stopped 2024-08-19T10:05:48.519 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-08-19T10:05:48.519 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-08-19T10:05:48.520 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T10:05:48.520 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:05:48.575 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T10:05:48.576 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-08-19T10:05:48.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:05:48.599 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546749453}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=279274101}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3908259599}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=377596440}) 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.600 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3992287177}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=805218144}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3856865022}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271881981}) 2024-08-19T10:05:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:05:48.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1643295322}) 2024-08-19T10:05:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.673+0000 7fdf381b1640 10 monclient: get_monmap_and_config 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.673+0000 7fdf381b1640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.673+0000 7fdf381b1640 10 monclient: monmap: 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:48.674134+0000 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:48.674134+0000 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.673+0000 7fdf381b1640 1 Processor -- start 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.673+0000 7fdf381b1640 1 -- start start 2024-08-19T10:05:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.673+0000 7fdf381b1640 10 monclient: init 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient: _add_conns ranks=[0,7,6,8,2,5,1,4,3] 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient(hunting): picked mon.noname-a con 0x7fdf300a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient(hunting): picked mon.noname-i con 0x7fdf300694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient(hunting): picked mon.noname-d con 0x7fdf300a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf300a3c70 con 0x7fdf300a6830 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf300a33d0 con 0x7fdf300a46a0 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf300a36b0 con 0x7fdf300694b0 2024-08-19T10:05:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf381b1640 10 monclient(hunting): authenticate will time out at 1604.750467s 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf35f4d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdf300a6830 0x7fdf300a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58762/0 (socket says 172.21.15.1:58762) 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf35f4d640 1 -- 172.21.15.1:0/1543758629 learned_addr learned my addr 172.21.15.1:0/1543758629 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.674+0000 7fdf3674e640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdf300a46a0 0x7fdf30141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:41968/0 (socket says 172.21.15.1:41968) 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf3674e640 1 -- 172.21.15.1:0/1543758629 >> v1:172.21.15.1:6792/0 conn(0x7fdf300a46a0 legacy=0x7fdf30141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf35f4d640 1 -- 172.21.15.1:0/1543758629 >> v1:172.21.15.1:6789/0 conn(0x7fdf300a6830 legacy=0x7fdf300a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873779955 0 0) 0x7fdf300a33d0 con 0x7fdf300a46a0 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): my global_id is 204234 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf18003520 con 0x7fdf300a46a0 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286772264 0 0) 0x7fdf300a3c70 con 0x7fdf300a6830 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 10 monclient(hunting): my global_id is 204255 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.676+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf300a33d0 con 0x7fdf300a6830 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf3574c640 1 -- 172.21.15.1:0/1543758629 >> v1:172.21.15.82:6792/0 conn(0x7fdf300694b0 legacy=0x7fdf300a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2899848831 0 0) 0x7fdf300a33d0 con 0x7fdf300a6830 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 >> v1:172.21.15.82:6792/0 conn(0x7fdf300694b0 legacy=0x7fdf300a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 >> v1:172.21.15.1:6792/0 conn(0x7fdf300a46a0 legacy=0x7fdf30141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 monclient: found mon.noname-a 2024-08-19T10:05:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3006dbb0 con 0x7fdf300a6830 2024-08-19T10:05:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf28001ac0 con 0x7fdf300a6830 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 5 monclient: authenticate success, global_id 204255 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: dump: 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf280048a0 con 0x7fdf300a6830 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 1 -- 172.21.15.1:0/1543758629 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf28004dd0 con 0x7fdf300a6830 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf34f4b640 10 monclient: dump: 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 4 set_mon_vals no callback set 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 10 monclient: shutdown 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 1 -- 172.21.15.1:0/1543758629 >> v1:172.21.15.1:6789/0 conn(0x7fdf300a6830 legacy=0x7fdf300a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 1 -- 172.21.15.1:0/1543758629 shutdown_connections 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 1 -- 172.21.15.1:0/1543758629 >> 172.21.15.1:0/1543758629 conn(0x7fdf300a9020 msgr2=0x7fdf30068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.677+0000 7fdf381b1640 1 -- 172.21.15.1:0/1543758629 shutdown_connections 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 -- 172.21.15.1:0/1543758629 wait complete. 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 10 monclient: build_initial_monmap 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 10 monclient: monmap: 2024-08-19T10:05:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 librados: starting msgr at 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 librados: starting objecter 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 Processor -- start 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 -- start start 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 librados: setting wanted keys 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 1 librados: calling monclient init 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.678+0000 7fdf381b1640 10 monclient: init 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient: _add_conns ranks=[6,3,2,4,8,5,0,7,1] 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient(hunting): picked mon.noname-d con 0x7fdf300694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient(hunting): picked mon.noname-g con 0x7fdf300a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient(hunting): picked mon.noname-b con 0x7fdf300a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf300a9f30 con 0x7fdf300a6830 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf30068a70 con 0x7fdf300a46a0 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf301399a0 con 0x7fdf300694b0 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf381b1640 10 monclient(hunting): authenticate will time out at 1604.755313s 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf3574c640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fdf300a46a0 legacy=0x7fdf30138750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf35f4d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdf300694b0 0x7fdf30138040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:41980/0 (socket says 172.21.15.1:41980) 2024-08-19T10:05:48.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.679+0000 7fdf35f4d640 1 -- 172.21.15.1:0/2078056611 learned_addr learned my addr 172.21.15.1:0/2078056611 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf35f4d640 1 -- 172.21.15.1:0/2078056611 >> v1:172.21.15.1:6792/0 conn(0x7fdf300694b0 legacy=0x7fdf30138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 client.?.objecter ms_handle_connect 0x7fdf300694b0 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf3674e640 1 -- 172.21.15.1:0/2078056611 >> v1:172.21.15.1:6790/0 conn(0x7fdf300a6830 legacy=0x7fdf301d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 client.?.objecter ms_handle_connect 0x7fdf300a6830 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 658466390 0 0) 0x7fdf301399a0 con 0x7fdf300694b0 2024-08-19T10:05:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): my global_id is 204243 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf1c004eb0 con 0x7fdf300694b0 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616362860 0 0) 0x7fdf300a9f30 con 0x7fdf300a6830 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 10 monclient(hunting): my global_id is 184619 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.681+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf301399a0 con 0x7fdf300a6830 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1424343982 0 0) 0x7fdf1c004eb0 con 0x7fdf300694b0 2024-08-19T10:05:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf300a9f30 con 0x7fdf300694b0 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf28004d90 con 0x7fdf300694b0 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): dump: 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3690805379 0 0) 0x7fdf301399a0 con 0x7fdf300a6830 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf1c004eb0 con 0x7fdf300a6830 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf200016a0 con 0x7fdf300a6830 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:05:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient(hunting): dump: 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 510786756 0 0) 0x7fdf300a9f30 con 0x7fdf300694b0 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 >> v1:172.21.15.82:6790/0 conn(0x7fdf300a46a0 legacy=0x7fdf30138750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 >> v1:172.21.15.1:6790/0 conn(0x7fdf300a6830 legacy=0x7fdf301d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 monclient: found mon.d 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf30138e60 con 0x7fdf300694b0 2024-08-19T10:05:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 5 monclient: authenticate success, global_id 204243 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 10 monclient: _renew_subs 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3013b460 con 0x7fdf300694b0 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 10 client.204243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 10 monclient: _renew_subs 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf3013a770 con 0x7fdf300694b0 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf381b1640 1 librados: init done 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf280048a0 con 0x7fdf300694b0 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:48.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdf28001ae0 con 0x7fdf300694b0 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2ffff640 4 set_mon_vals no callback set 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.682+0000 7fdf2e7fc640 10 monclient: dump: 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fdf280119c0 con 0x7fdf300694b0 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf381b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf381b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf381b1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf10005080 con 0x7fdf300694b0 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7fdf2804e500 con 0x7fdf300694b0 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf2e7fc640 10 client.204243.objecter ms_dispatch 0x7fdf30000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf2e7fc640 3 client.204243.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf2e7fc640 3 client.204243.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf3574c640 10 monclient: get_auth_request con 0x7fdf1c041690 auth_method 0 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.683+0000 7fdf2e7fc640 20 client.204243.objecter dump_active .. 0 homeless 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.685+0000 7fdf2e7fc640 10 client.204243.objecter ms_handle_connect 0x7fdf1c041690 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.687+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fdf28017790 con 0x7fdf300694b0 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.687+0000 7fdf2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:05:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.687+0000 7fdf2e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:05:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf381b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:05:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf381b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf381b1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:05:48.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdf10005370 con 0x7fdf300694b0 2024-08-19T10:05:48.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf2e7fc640 1 -- 172.21.15.1:0/2078056611 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 2224646842) 0x7fdf28017090 con 0x7fdf300694b0 2024-08-19T10:05:48.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:05:48.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.942+0000 7fdf2e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:05:48.946 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":130,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:05:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 10 librados: watch_flush enter 2024-08-19T10:05:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 10 librados: watch_flush exit 2024-08-19T10:05:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 20 client.204243.objecter shutdown clearing up homeless session... 2024-08-19T10:05:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 10 client.204243.objecter successfully canceled tick 2024-08-19T10:05:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 >> v1:172.21.15.82:6812/2549104415 conn(0x7fdf1c041690 legacy=0x7fdf1c043b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 10 monclient: shutdown 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 >> v1:172.21.15.1:6792/0 conn(0x7fdf300694b0 legacy=0x7fdf30138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 shutdown_connections 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.944+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 >> 172.21.15.1:0/2078056611 conn(0x7fdf300a9020 msgr2=0x7fdf300a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.945+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 shutdown_connections 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.945+0000 7fdf381b1640 1 -- 172.21.15.1:0/2078056611 wait complete. 2024-08-19T10:05:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:48.945+0000 7fdf381b1640 1 librados: shutdown 2024-08-19T10:05:51.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:05:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.066+0000 7fdb335a3640 10 monclient: get_monmap_and_config 2024-08-19T10:05:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.066+0000 7fdb335a3640 10 monclient: build_initial_monmap 2024-08-19T10:05:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.066+0000 7fdb335a3640 10 monclient: monmap: 2024-08-19T10:05:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:05:52.067244+0000 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:05:52.067244+0000 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:52.068 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:52.069 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:52.069 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:52.069 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:52.069 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.066+0000 7fdb335a3640 1 Processor -- start 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 1 -- start start 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient: init 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient: _add_conns ranks=[0,7,6,1,4,2,5,8,3] 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient(hunting): picked mon.noname-a con 0x7fdb2c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient(hunting): picked mon.noname-i con 0x7fdb2c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient(hunting): picked mon.noname-d con 0x7fdb2c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c0a3b40 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c0a8130 con 0x7fdb2c13f5c0 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c06ede0 con 0x7fdb2c0a6ba0 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb335a3640 10 monclient(hunting): authenticate will time out at 1608.143693s 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb3133f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdb2c0a67c0 0x7fdb2c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58772/0 (socket says 172.21.15.1:58772) 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb31b40640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdb2c13f5c0 0x7fdb2c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:41990/0 (socket says 172.21.15.1:41990) 2024-08-19T10:05:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.067+0000 7fdb3133f640 1 -- 172.21.15.1:0/1867468415 learned_addr learned my addr 172.21.15.1:0/1867468415 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.069+0000 7fdb31b40640 1 -- 172.21.15.1:0/1867468415 >> v1:172.21.15.1:6792/0 conn(0x7fdb2c13f5c0 legacy=0x7fdb2c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb3133f640 1 -- 172.21.15.1:0/1867468415 >> v1:172.21.15.1:6789/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235847962 0 0) 0x7fdb2c0a3b40 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient(hunting): _negotiate 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient(hunting): my global_id is 204264 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb30b3e640 1 -- 172.21.15.1:0/1867468415 >> v1:172.21.15.82:6792/0 conn(0x7fdb2c0a6ba0 legacy=0x7fdb2c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb18003520 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4161160163 0 0) 0x7fdb18003520 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 >> v1:172.21.15.82:6792/0 conn(0x7fdb2c0a6ba0 legacy=0x7fdb2c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 >> v1:172.21.15.1:6792/0 conn(0x7fdb2c13f5c0 legacy=0x7fdb2c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: _finish_hunting 0 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 monclient: found mon.noname-a 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c0a92a0 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: _finish_auth 0 2024-08-19T10:05:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: _check_auth_tickets 2024-08-19T10:05:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:05:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdb28001ae0 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb335a3640 5 monclient: authenticate success, global_id 204264 2024-08-19T10:05:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb335a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: dump: 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb335a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:05:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb280048a0 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 1 -- 172.21.15.1:0/1867468415 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdb28004e10 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb23fff640 10 monclient: dump: 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb335a3640 10 monclient: get_monmap_and_config success 2024-08-19T10:05:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.070+0000 7fdb335a3640 4 set_mon_vals no callback set 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 10 monclient: shutdown 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 1 -- 172.21.15.1:0/1867468415 >> v1:172.21.15.1:6789/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 1 -- 172.21.15.1:0/1867468415 shutdown_connections 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 1 -- 172.21.15.1:0/1867468415 >> 172.21.15.1:0/1867468415 conn(0x7fdb2c0ac4b0 msgr2=0x7fdb2c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 1 -- 172.21.15.1:0/1867468415 shutdown_connections 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 1 -- 172.21.15.1:0/1867468415 wait complete. 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 10 monclient: build_initial_monmap 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.071+0000 7fdb335a3640 10 monclient: monmap: 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:05:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 librados: starting msgr at 2024-08-19T10:05:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 librados: starting objecter 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 Processor -- start 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 -- start start 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 librados: setting wanted keys 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 librados: calling monclient init 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient: init 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient: _reopen_session rank -1 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient: _add_conns ranks=[5,6,4,3,7,2,8,0,1] 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient(hunting): picked mon.noname-h con 0x7fdb2c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient(hunting): picked mon.noname-d con 0x7fdb2c0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient(hunting): picked mon.noname-c con 0x7fdb2c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c13ae20 con 0x7fdb2c13f5c0 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c139930 con 0x7fdb2c0a67c0 2024-08-19T10:05:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c13b3f0 con 0x7fdb2c0a6ba0 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb335a3640 10 monclient(hunting): authenticate will time out at 1608.148700s 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb30b3e640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fdb2c0a6ba0 0x7fdb2c13c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:42006/0 (socket says 172.21.15.1:42006) 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb30b3e640 1 -- 172.21.15.1:0/1823363525 learned_addr learned my addr 172.21.15.1:0/1823363525 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.072+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6791/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c13c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.074+0000 7fdb31b40640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6791/0 conn(0x7fdb2c13f5c0 legacy=0x7fdb2c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.074+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb2c13f5c0 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.074+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.074+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb2c0a6ba0 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.074+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.074+0000 7fdb30b3e640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6792/0 conn(0x7fdb2c0a6ba0 legacy=0x7fdb2c13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:52.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.273+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6791/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c13c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:05:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:52.674+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6791/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c13c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:05:53.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:53.475+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6791/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c13c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:05:55.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): tick 2024-08-19T10:05:55.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:05:55.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 monclient(hunting): continuing hunt 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6792/0 conn(0x7fdb2c0a6ba0 legacy=0x7fdb2c13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6791/0 conn(0x7fdb2c0a67c0 legacy=0x7fdb2c13c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6791/0 conn(0x7fdb2c13f5c0 legacy=0x7fdb2c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient: _add_conns ranks=[3,2,7,1,8,0] 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-g con 0x7fdb140038c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fdb140074b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-i con 0x7fdb1400b240 addr v1:172.21.15.82:6792/0 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c13b6d0 con 0x7fdb140074b0 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c139930 con 0x7fdb140038c0 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb14011100 con 0x7fdb1400b240 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb20ff9640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:55.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.073+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6790/0 conn(0x7fdb140038c0 legacy=0x7fdb14005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:05:55.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.075+0000 7fdb30b3e640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6790/0 conn(0x7fdb140074b0 legacy=0x7fdb14009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:55.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.075+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb140074b0 2024-08-19T10:05:55.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.075+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:55.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.075+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb1400b240 2024-08-19T10:05:55.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.075+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:55.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.075+0000 7fdb31b40640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6792/0 conn(0x7fdb1400b240 legacy=0x7fdb1400d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:55.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.274+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6790/0 conn(0x7fdb140038c0 legacy=0x7fdb14005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:05:55.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:55.675+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6790/0 conn(0x7fdb140038c0 legacy=0x7fdb14005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:05:56.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:56.476+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6790/0 conn(0x7fdb140038c0 legacy=0x7fdb14005d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): tick 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 monclient(hunting): continuing hunt 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6792/0 conn(0x7fdb1400b240 legacy=0x7fdb1400d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6790/0 conn(0x7fdb140038c0 legacy=0x7fdb14005d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6790/0 conn(0x7fdb140074b0 legacy=0x7fdb14009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient: _add_conns ranks=[0,1,8] 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fdb140114f0 addr v1:172.21.15.1:6789/0 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb31b40640 1 -- 172.21.15.1:0/1823363525 reap_dead start 2024-08-19T10:05:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-f con 0x7fdb140137a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:05:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-e con 0x7fdb1400a140 addr v1:172.21.15.1:6793/0 2024-08-19T10:05:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb14012460 con 0x7fdb140114f0 2024-08-19T10:05:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2c139930 con 0x7fdb140137a0 2024-08-19T10:05:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb1400a670 con 0x7fdb1400a140 2024-08-19T10:05:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.074+0000 7fdb20ff9640 10 monclient(hunting): _renew_subs 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6789/0 conn(0x7fdb140114f0 legacy=0x7fdb140118d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb140114f0 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb1400a140 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb31b40640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6793/0 conn(0x7fdb1400a140 legacy=0x7fdb14009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb140137a0 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.076+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:05:58.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:05:58.075+0000 7fdb30b3e640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6789/0 conn(0x7fdb140137a0 legacy=0x7fdb14015b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): tick 2024-08-19T10:06:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:06:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 monclient(hunting): continuing hunt 2024-08-19T10:06:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:06:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6793/0 conn(0x7fdb1400a140 legacy=0x7fdb14009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6789/0 conn(0x7fdb140137a0 legacy=0x7fdb14015b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6789/0 conn(0x7fdb140114f0 legacy=0x7fdb140118d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient: _add_conns ranks=[7,1,4,6,3,2,0,5,8] 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-i con 0x7fdb1400a840 addr v1:172.21.15.82:6792/0 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-f con 0x7fdb140137a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fdb1400ac80 addr v1:172.21.15.1:6791/0 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb14005dc0 con 0x7fdb140137a0 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb1400ecd0 con 0x7fdb1400ac80 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb1400d6f0 con 0x7fdb1400a840 2024-08-19T10:06:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.074+0000 7fdb20ff9640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb31b40640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6791/0 conn(0x7fdb1400ac80 legacy=0x7fdb14015b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb1400ac80 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb30b3e640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6789/0 conn(0x7fdb140137a0 legacy=0x7fdb1400e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb140137a0 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb21ffb640 10 client.?.objecter ms_handle_connect 0x7fdb1400a840 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb21ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:01.076+0000 7fdb3133f640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6792/0 conn(0x7fdb1400a840 legacy=0x7fdb1400deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:02.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.923+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155995122 0 0) 0x7fdb14005dc0 con 0x7fdb140137a0 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.923+0000 7fdb21ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.923+0000 7fdb21ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.923+0000 7fdb21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.923+0000 7fdb21ffb640 10 monclient(hunting): my global_id is 174997 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.923+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb10004eb0 con 0x7fdb140137a0 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 954133518 0 0) 0x7fdb10004eb0 con 0x7fdb140137a0 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb14005dc0 con 0x7fdb140137a0 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdb1c003610 con 0x7fdb140137a0 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): dump: 2024-08-19T10:06:02.929 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3820690409 0 0) 0x7fdb1400ecd0 con 0x7fdb1400ac80 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient(hunting): my global_id is 194476 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb10004eb0 con 0x7fdb1400ac80 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 126998377 0 0) 0x7fdb14005dc0 con 0x7fdb140137a0 2024-08-19T10:06:02.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6792/0 conn(0x7fdb1400a840 legacy=0x7fdb1400deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.1:6791/0 conn(0x7fdb1400ac80 legacy=0x7fdb14015b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 monclient: found mon.f 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1400d910 con 0x7fdb140137a0 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: _finish_auth 0 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 5 monclient: authenticate success, global_id 174997 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 10 monclient: _renew_subs 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c13a700 con 0x7fdb140137a0 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb1c003b10 con 0x7fdb140137a0 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fdb1c004080 con 0x7fdb140137a0 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb21ffb640 10 monclient: dump: 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb237fe640 4 set_mon_vals no callback set 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 10 client.174997.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 10 monclient: _renew_subs 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdb2c13e6e0 con 0x7fdb140137a0 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.924+0000 7fdb335a3640 1 librados: init done 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.925+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fdb1c004220 con 0x7fdb140137a0 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.925+0000 7fdb335a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb335a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb335a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2c13b910 con 0x7fdb140137a0 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7fdb1c04e120 con 0x7fdb140137a0 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb21ffb640 10 client.174997.objecter ms_dispatch 0x7fdb2c000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb21ffb640 3 client.174997.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb21ffb640 3 client.174997.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb21ffb640 20 client.174997.objecter dump_active .. 0 homeless 2024-08-19T10:06:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.926+0000 7fdb3133f640 10 monclient: get_auth_request con 0x7fdb10040f60 auth_method 0 2024-08-19T10:06:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.928+0000 7fdb21ffb640 10 client.174997.objecter ms_handle_connect 0x7fdb10040f60 2024-08-19T10:06:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.931+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fdb1c017420 con 0x7fdb140137a0 2024-08-19T10:06:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.931+0000 7fdb21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:02.931+0000 7fdb21ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:03.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.207+0000 7fdb335a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:06:03.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.207+0000 7fdb335a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:06:03.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.207+0000 7fdb335a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:03.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.207+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdb2c13b910 con 0x7fdb140137a0 2024-08-19T10:06:03.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.208+0000 7fdb21ffb640 1 -- 172.21.15.1:0/1823363525 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2691 (unknown 254746657 0 3046117463) 0x7fdb1c016d20 con 0x7fdb140137a0 2024-08-19T10:06:03.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.208+0000 7fdb21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:06:03.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.208+0000 7fdb21ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:03.210 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":134,"quorum":[0,1,2,4,6,7,8],"quorum_names":["a","f","b","c","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:06:03.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.209+0000 7fdb335a3640 10 librados: watch_flush enter 2024-08-19T10:06:03.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.209+0000 7fdb335a3640 10 librados: watch_flush exit 2024-08-19T10:06:03.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.209+0000 7fdb335a3640 20 client.174997.objecter shutdown clearing up homeless session... 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.209+0000 7fdb335a3640 10 client.174997.objecter successfully canceled tick 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6812/2549104415 conn(0x7fdb10040f60 legacy=0x7fdb10043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 10 monclient: shutdown 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 >> v1:172.21.15.82:6789/0 conn(0x7fdb140137a0 legacy=0x7fdb1400e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb31b40640 1 -- 172.21.15.1:0/1823363525 reap_dead start 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 shutdown_connections 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 >> 172.21.15.1:0/1823363525 conn(0x7fdb2c0ac4b0 msgr2=0x7fdb2c13ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 shutdown_connections 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 -- 172.21.15.1:0/1823363525 wait complete. 2024-08-19T10:06:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.210+0000 7fdb335a3640 1 librados: shutdown 2024-08-19T10:06:03.223 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-08-19T10:06:03.223 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-08-19T10:06:03.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:06:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.320+0000 7f2727d41640 10 monclient: get_monmap_and_config 2024-08-19T10:06:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.320+0000 7f2727d41640 10 monclient: build_initial_monmap 2024-08-19T10:06:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.320+0000 7f2727d41640 10 monclient: monmap: 2024-08-19T10:06:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:03.322672+0000 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:03.322672+0000 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 1 Processor -- start 2024-08-19T10:06:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 1 -- start start 2024-08-19T10:06:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 10 monclient: init 2024-08-19T10:06:03.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 10 monclient: _add_conns ranks=[0,3,5,8,1,6,4,7,2] 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 10 monclient(hunting): picked mon.noname-a con 0x7f27200a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.321+0000 7f2727d41640 10 monclient(hunting): picked mon.noname-g con 0x7f27200a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2727d41640 10 monclient(hunting): picked mon.noname-h con 0x7f272013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2727d41640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27200a3b40 con 0x7f27200a67c0 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2727d41640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27200a8130 con 0x7f27200a6ba0 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2727d41640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272006ede0 con 0x7f272013f5c0 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2727d41640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2727d41640 10 monclient(hunting): authenticate will time out at 1619.399050s 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2725add640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f27200a67c0 0x7f27200a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53520/0 (socket says 172.21.15.1:53520) 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f2725add640 1 -- 172.21.15.1:0/2063458764 learned_addr learned my addr 172.21.15.1:0/2063458764 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:03.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f27262de640 1 -- 172.21.15.1:0/2063458764 >> v1:172.21.15.82:6791/0 conn(0x7f272013f5c0 legacy=0x7f27201419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.322+0000 7f27252dc640 1 -- 172.21.15.1:0/2063458764 >> v1:172.21.15.82:6790/0 conn(0x7f27200a6ba0 legacy=0x7f27200a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2725add640 1 -- 172.21.15.1:0/2063458764 >> v1:172.21.15.1:6789/0 conn(0x7f27200a67c0 legacy=0x7f27200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725289287 0 0) 0x7f27200a3b40 con 0x7f27200a67c0 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient(hunting): my global_id is 204273 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270c003520 con 0x7f27200a67c0 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2009830111 0 0) 0x7f270c003520 con 0x7f27200a67c0 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 >> v1:172.21.15.82:6791/0 conn(0x7f272013f5c0 legacy=0x7f27201419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 >> v1:172.21.15.82:6790/0 conn(0x7f27200a6ba0 legacy=0x7f27200a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: _finish_hunting 0 2024-08-19T10:06:03.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 monclient: found mon.noname-a 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27200a92a0 con 0x7f27200a67c0 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: _finish_auth 0 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: _check_auth_tickets 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2727d41640 5 monclient: authenticate success, global_id 204273 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2727d41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f271c001ae0 con 0x7f27200a67c0 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: dump: 2024-08-19T10:06:03.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2727d41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f271c0048a0 con 0x7f27200a67c0 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 1 -- 172.21.15.1:0/2063458764 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f271c004e10 con 0x7f27200a67c0 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2727d41640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2727d41640 4 set_mon_vals no callback set 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.324+0000 7f2724adb640 10 monclient: dump: 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 10 monclient: shutdown 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 1 -- 172.21.15.1:0/2063458764 >> v1:172.21.15.1:6789/0 conn(0x7f27200a67c0 legacy=0x7f27200a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 1 -- 172.21.15.1:0/2063458764 shutdown_connections 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 1 -- 172.21.15.1:0/2063458764 >> 172.21.15.1:0/2063458764 conn(0x7f27200ac4b0 msgr2=0x7f27200a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 1 -- 172.21.15.1:0/2063458764 shutdown_connections 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 1 -- 172.21.15.1:0/2063458764 wait complete. 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 10 monclient: build_initial_monmap 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.325+0000 7f2727d41640 10 monclient: monmap: 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:03.336 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 librados: starting msgr at 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 librados: starting objecter 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 Processor -- start 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 -- start start 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 librados: setting wanted keys 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 librados: calling monclient init 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient: init 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient: _add_conns ranks=[7,6,2,4,8,1,3,0,5] 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient(hunting): picked mon.noname-i con 0x7f27200a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient(hunting): picked mon.noname-d con 0x7f27200a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient(hunting): picked mon.noname-b con 0x7f272013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2720142a60 con 0x7f272013f5c0 2024-08-19T10:06:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272013cc50 con 0x7f27200a6ba0 2024-08-19T10:06:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272013a7d0 con 0x7f27200a67c0 2024-08-19T10:06:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f2727d41640 10 monclient(hunting): authenticate will time out at 1619.403770s 2024-08-19T10:06:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f27252dc640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f27200a6ba0 0x7f272013a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:38476/0 (socket says 172.21.15.1:38476) 2024-08-19T10:06:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.326+0000 7f27252dc640 1 -- 172.21.15.1:0/4169487817 learned_addr learned my addr 172.21.15.1:0/4169487817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:03.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27252dc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6792/0 conn(0x7f27200a6ba0 legacy=0x7f272013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 client.?.objecter ms_handle_connect 0x7f27200a6ba0 2024-08-19T10:06:03.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:03.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 client.?.objecter ms_handle_connect 0x7f272013f5c0 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27262de640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6790/0 conn(0x7f272013f5c0 legacy=0x7f27201d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f2725add640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.82:6792/0 conn(0x7f27200a67c0 legacy=0x7f27201399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 client.?.objecter ms_handle_connect 0x7f27200a67c0 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316036952 0 0) 0x7f272013cc50 con 0x7f27200a6ba0 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): my global_id is 204252 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2704004eb0 con 0x7f27200a6ba0 2024-08-19T10:06:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3339908046 0 0) 0x7f2718004060 con 0x7f272013f5c0 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): my global_id is 184628 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f272013cc50 con 0x7f272013f5c0 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116355426 0 0) 0x7f272013a7d0 con 0x7f27200a67c0 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): my global_id is 204298 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2718004060 con 0x7f27200a67c0 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 921371059 0 0) 0x7f2704004eb0 con 0x7f27200a6ba0 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f272013a7d0 con 0x7f27200a6ba0 2024-08-19T10:06:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1589405212 0 0) 0x7f272013cc50 con 0x7f272013f5c0 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2704004eb0 con 0x7f272013f5c0 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2710004610 con 0x7f27200a6ba0 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): dump: 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2718004a50 con 0x7f272013f5c0 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): dump: 2024-08-19T10:06:03.350 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.351 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3737819130 0 0) 0x7f2718004060 con 0x7f27200a67c0 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f272013cc50 con 0x7f27200a67c0 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f271c004d90 con 0x7f27200a67c0 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient(hunting): dump: 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.352 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1023008104 0 0) 0x7f272013a7d0 con 0x7f27200a6ba0 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.82:6792/0 conn(0x7f27200a67c0 legacy=0x7f27201399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6790/0 conn(0x7f272013f5c0 legacy=0x7f27201d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient: _finish_hunting 0 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 monclient: found mon.d 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272013ab50 con 0x7f27200a6ba0 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient: _finish_auth 0 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 10 monclient: _check_auth_tickets 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f27167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.329+0000 7f2727d41640 5 monclient: authenticate success, global_id 204252 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 10 monclient: _renew_subs 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27200aa0f0 con 0x7f27200a6ba0 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 10 client.204252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:03.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 10 monclient: _renew_subs 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f272013b720 con 0x7f27200a6ba0 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 1 librados: init done 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2710004cf0 con 0x7f27200a6ba0 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27167fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2710005220 con 0x7f27200a6ba0 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27177fe640 4 set_mon_vals no callback set 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27167fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f27167fc640 10 monclient: dump: 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.355 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.356 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.330+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f27080065c0 con 0x7f2708002860 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f27100053c0 con 0x7f27200a6ba0 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(575..575 src has 299..575) ==== 5936+0+0 (unknown 3776193608 0 0) 0x7f271004e4e0 con 0x7f27200a6ba0 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27167fc640 10 client.204252.objecter ms_dispatch 0x7f2720000b90 osd_map(575..575 src has 299..575) 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27167fc640 3 client.204252.objecter handle_osd_map got epochs [575,575] > 0 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27167fc640 3 client.204252.objecter handle_osd_map decoding full epoch 575 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27167fc640 20 client.204252.objecter dump_active .. 0 homeless 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.331+0000 7f27262de640 10 monclient: get_auth_request con 0x7f2704041860 auth_method 0 2024-08-19T10:06:03.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 client.204252.objecter ms_handle_connect 0x7f2708002860 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f2725add640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6789/0 conn(0x7f2708002860 legacy=0x7f2708004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 client.204252.objecter resend_mon_ops 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936056896 0 0) 0x7f27080065c0 con 0x7f2708002860 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient(hunting): my global_id is 204282 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f272013a7d0 con 0x7f2708002860 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1011412701 0 0) 0x7f272013a7d0 con 0x7f2708002860 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27080065c0 con 0x7f2708002860 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f271c001580 con 0x7f2708002860 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 10 monclient: dump: 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4148510148 0 0) 0x7f27080065c0 con 0x7f2708002860 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.333+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f270800a160 con 0x7f2708002860 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.334+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f271c001320 con 0x7f2708002860 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.334+0000 7f27167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.334+0000 7f27167fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.334+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6789/0 conn(0x7f2708002860 legacy=0x7f2708004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.334+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6789/0 conn(0x7f2708002860 legacy=0x7f2708004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:03.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.334+0000 7f27167fc640 10 client.204252.objecter ms_handle_connect 0x7f2704041860 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.340+0000 7f2727d41640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.340+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2708009e70 con 0x7f2708009130 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 client.204252.objecter ms_handle_connect 0x7f2708009130 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 client.204252.objecter resend_mon_ops 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f2725add640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6789/0 conn(0x7f2708009130 legacy=0x7f27080095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2178507494 0 0) 0x7f2708009e70 con 0x7f2708009130 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient(hunting): my global_id is 204291 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27080065c0 con 0x7f2708009130 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3034480454 0 0) 0x7f27080065c0 con 0x7f2708009130 2024-08-19T10:06:03.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2708009e70 con 0x7f2708009130 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f271c005a50 con 0x7f2708009130 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 10 monclient: dump: 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.362 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1053450263 0 0) 0x7f2708009e70 con 0x7f2708009130 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.342+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f270800c3f0 con 0x7f2708009130 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7497 (unknown 0 0 536804822) 0x7f271c005f10 con 0x7f2708009130 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27167fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6789/0 conn(0x7f2708009130 legacy=0x7f27080095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27167fc640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6789/0 conn(0x7f2708009130 legacy=0x7f27080095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 29215, 2024-08-19T10:06:03.364 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:03.365 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:03.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:03.367 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:03.368 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:03.369 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:03.370 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:03.374 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:03.374 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.374 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.374 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.376 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:03.377 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:03.385 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:03.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:03.387 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:03.388 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:03.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:03.388 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:03.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:03.388 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:03.388 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 10 librados: watch_flush enter 2024-08-19T10:06:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 10 librados: watch_flush exit 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 20 client.204252.objecter shutdown clearing up homeless session... 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 10 client.204252.objecter successfully canceled tick 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2704041860 legacy=0x7f2704043d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 10 monclient: shutdown 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 >> v1:172.21.15.1:6792/0 conn(0x7f27200a6ba0 legacy=0x7f272013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27262de640 1 -- 172.21.15.1:0/4169487817 reap_dead start 2024-08-19T10:06:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f27262de640 1 -- 172.21.15.1:0/4169487817 reap_dead start 2024-08-19T10:06:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 shutdown_connections 2024-08-19T10:06:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 >> 172.21.15.1:0/4169487817 conn(0x7f27200ac4b0 msgr2=0x7f2720141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 shutdown_connections 2024-08-19T10:06:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 -- 172.21.15.1:0/4169487817 wait complete. 2024-08-19T10:06:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.343+0000 7f2727d41640 1 librados: shutdown 2024-08-19T10:06:03.394 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:03.394 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-08-19T10:06:03.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:06:03.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.492+0000 7fab4547e640 10 monclient: get_monmap_and_config 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.492+0000 7fab4547e640 10 monclient: build_initial_monmap 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.492+0000 7fab4547e640 10 monclient: monmap: 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:03.494750+0000 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:03.494750+0000 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:03.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 1 Processor -- start 2024-08-19T10:06:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 1 -- start start 2024-08-19T10:06:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient: init 2024-08-19T10:06:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient: _add_conns ranks=[6,5,1,7,4,8,0,2,3] 2024-08-19T10:06:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient(hunting): picked mon.noname-d con 0x7fab400a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient(hunting): picked mon.noname-h con 0x7fab400a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient(hunting): picked mon.noname-f con 0x7fab4013f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab400a3b40 con 0x7fab4013f5c0 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab400a8130 con 0x7fab400a6ba0 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4006ede0 con 0x7fab400a67c0 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.493+0000 7fab4547e640 10 monclient(hunting): authenticate will time out at 1619.570948s 2024-08-19T10:06:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.494+0000 7fab3e7fc640 1 -- >> v1:172.21.15.82:6791/0 conn(0x7fab400a6ba0 legacy=0x7fab400a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.494+0000 7fab3effd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fab400a67c0 0x7fab400a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:38484/0 (socket says 172.21.15.1:38484) 2024-08-19T10:06:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.494+0000 7fab3effd640 1 -- 172.21.15.1:0/3423048148 learned_addr learned my addr 172.21.15.1:0/3423048148 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3effd640 1 -- 172.21.15.1:0/3423048148 >> v1:172.21.15.1:6792/0 conn(0x7fab400a67c0 legacy=0x7fab400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3f7fe640 1 -- 172.21.15.1:0/3423048148 >> v1:172.21.15.82:6789/0 conn(0x7fab4013f5c0 legacy=0x7fab401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420787087 0 0) 0x7fab4006ede0 con 0x7fab400a67c0 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient(hunting): my global_id is 204261 2024-08-19T10:06:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab2c003520 con 0x7fab400a67c0 2024-08-19T10:06:03.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1050792638 0 0) 0x7fab2c003520 con 0x7fab400a67c0 2024-08-19T10:06:03.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 >> v1:172.21.15.82:6791/0 conn(0x7fab400a6ba0 legacy=0x7fab400a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:06:03.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 >> v1:172.21.15.82:6789/0 conn(0x7fab4013f5c0 legacy=0x7fab401419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:06:03.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 monclient: found mon.noname-d 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab400a92a0 con 0x7fab400a67c0 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: _finish_auth 0 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab30001e50 con 0x7fab400a67c0 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab4547e640 5 monclient: authenticate success, global_id 204261 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab4547e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: dump: 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab4547e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab30004c20 con 0x7fab400a67c0 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 1 -- 172.21.15.1:0/3423048148 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab30005190 con 0x7fab400a67c0 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab3dffb640 10 monclient: dump: 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab4547e640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.496+0000 7fab4547e640 4 set_mon_vals no callback set 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 10 monclient: shutdown 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 1 -- 172.21.15.1:0/3423048148 >> v1:172.21.15.1:6792/0 conn(0x7fab400a67c0 legacy=0x7fab400a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 1 -- 172.21.15.1:0/3423048148 shutdown_connections 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 1 -- 172.21.15.1:0/3423048148 >> 172.21.15.1:0/3423048148 conn(0x7fab400ac4b0 msgr2=0x7fab400a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 1 -- 172.21.15.1:0/3423048148 shutdown_connections 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 1 -- 172.21.15.1:0/3423048148 wait complete. 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 10 monclient: build_initial_monmap 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.497+0000 7fab4547e640 10 monclient: monmap: 2024-08-19T10:06:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 librados: starting msgr at 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 librados: starting objecter 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 Processor -- start 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 -- start start 2024-08-19T10:06:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 librados: setting wanted keys 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 librados: calling monclient init 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient: init 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient: _add_conns ranks=[1,3,5,0,8,6,2,4,7] 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient(hunting): picked mon.noname-f con 0x7fab400a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient(hunting): picked mon.noname-g con 0x7fab400a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient(hunting): picked mon.noname-h con 0x7fab4013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab401cf020 con 0x7fab400a67c0 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4013e400 con 0x7fab400a6ba0 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab40139bb0 con 0x7fab4013f5c0 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab4547e640 10 monclient(hunting): authenticate will time out at 1619.575870s 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab3e7fc640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7fab400a6ba0 legacy=0x7fab401ce910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.498+0000 7fab3f7fe640 1 -- >> v1:172.21.15.82:6791/0 conn(0x7fab4013f5c0 legacy=0x7fab401d6460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.499+0000 7fab3effd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fab400a67c0 0x7fab401ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:37338/0 (socket says 172.21.15.1:37338) 2024-08-19T10:06:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.499+0000 7fab3effd640 1 -- 172.21.15.1:0/3155543681 learned_addr learned my addr 172.21.15.1:0/3155543681 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab3effd640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6789/0 conn(0x7fab400a67c0 legacy=0x7fab401ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 10 client.?.objecter ms_handle_connect 0x7fab400a67c0 2024-08-19T10:06:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443726041 0 0) 0x7fab401cf020 con 0x7fab400a67c0 2024-08-19T10:06:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 10 monclient(hunting): _negotiate 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 10 monclient(hunting): my global_id is 175015 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab20004eb0 con 0x7fab400a67c0 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab3effd640 1 --1- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6789/0 conn(0x7fab400a67c0 0x7fab401ce200 :-1 s=OPENED pgs=3760 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fab20004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3762988818 0 0) 0x7fab20004eb0 con 0x7fab400a67c0 2024-08-19T10:06:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.501+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab401cf020 con 0x7fab400a67c0 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.502+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab30005110 con 0x7fab400a67c0 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.502+0000 7fab377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.502+0000 7fab377fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.502+0000 7fab377fe640 10 monclient(hunting): dump: 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:03.526 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.502+0000 7fab377fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:03.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.700+0000 7fab3e7fc640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6790/0 conn(0x7fab400a6ba0 legacy=0x7fab401ce910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:03.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:03.700+0000 7fab3f7fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6791/0 conn(0x7fab4013f5c0 legacy=0x7fab401d6460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.102+0000 7fab3f7fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6791/0 conn(0x7fab4013f5c0 legacy=0x7fab401d6460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.102+0000 7fab3e7fc640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6790/0 conn(0x7fab400a6ba0 legacy=0x7fab401ce910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2527345349 0 0) 0x7fab401cf020 con 0x7fab400a67c0 2024-08-19T10:06:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6791/0 conn(0x7fab4013f5c0 legacy=0x7fab401d6460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6790/0 conn(0x7fab400a6ba0 legacy=0x7fab401ce910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 1 monclient: found mon.f 2024-08-19T10:06:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.124+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4013bf10 con 0x7fab400a67c0 2024-08-19T10:06:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 5 monclient: authenticate success, global_id 175015 2024-08-19T10:06:04.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 10 monclient: _renew_subs 2024-08-19T10:06:04.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:04.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab4013b670 con 0x7fab400a67c0 2024-08-19T10:06:04.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 10 client.175015.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:04.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 10 monclient: _renew_subs 2024-08-19T10:06:04.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:06:04.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab400a2890 con 0x7fab400a67c0 2024-08-19T10:06:04.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 1 librados: init done 2024-08-19T10:06:04.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab30004c20 con 0x7fab400a67c0 2024-08-19T10:06:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab300020a0 con 0x7fab400a67c0 2024-08-19T10:06:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:06:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab3d7fa640 4 set_mon_vals no callback set 2024-08-19T10:06:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab377fe640 10 monclient: dump: 2024-08-19T10:06:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.137 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.125+0000 7fab4547e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fab240065c0 con 0x7fab24002860 2024-08-19T10:06:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fab30011d80 con 0x7fab400a67c0 2024-08-19T10:06:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(576..576 src has 299..576) ==== 5936+0+0 (unknown 2747268155 0 0) 0x7fab3004e850 con 0x7fab400a67c0 2024-08-19T10:06:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab377fe640 10 client.175015.objecter ms_dispatch 0x7fab40000b90 osd_map(576..576 src has 299..576) 2024-08-19T10:06:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab377fe640 3 client.175015.objecter handle_osd_map got epochs [576,576] > 0 2024-08-19T10:06:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab377fe640 3 client.175015.objecter handle_osd_map decoding full epoch 576 2024-08-19T10:06:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab377fe640 20 client.175015.objecter dump_active .. 0 homeless 2024-08-19T10:06:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.126+0000 7fab3f7fe640 10 monclient: get_auth_request con 0x7fab20040d70 auth_method 0 2024-08-19T10:06:04.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab3e7fc640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.1:6790/0 conn(0x7fab24002860 legacy=0x7fab24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 client.175015.objecter ms_handle_connect 0x7fab24002860 2024-08-19T10:06:04.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 client.175015.objecter resend_mon_ops 2024-08-19T10:06:04.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3291354009 0 0) 0x7fab240065c0 con 0x7fab24002860 2024-08-19T10:06:04.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient(hunting): my global_id is 184637 2024-08-19T10:06:04.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab401cf020 con 0x7fab24002860 2024-08-19T10:06:04.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3284766961 0 0) 0x7fab401cf020 con 0x7fab24002860 2024-08-19T10:06:04.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab240065c0 con 0x7fab24002860 2024-08-19T10:06:04.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab28004510 con 0x7fab24002860 2024-08-19T10:06:04.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:06:04.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.128+0000 7fab377fe640 10 monclient: dump: 2024-08-19T10:06:04.147 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.147 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.147 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.147 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.148 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.148 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.149 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.149 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.149 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.149 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.149 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.149 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2531874631 0 0) 0x7fab240065c0 con 0x7fab24002860 2024-08-19T10:06:04.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab2400a160 con 0x7fab24002860 2024-08-19T10:06:04.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 10 client.175015.objecter ms_handle_connect 0x7fab20040d70 2024-08-19T10:06:04.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fab280068c0 con 0x7fab24002860 2024-08-19T10:06:04.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.1:6790/0 conn(0x7fab24002860 legacy=0x7fab24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.129+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.1:6790/0 conn(0x7fab24002860 legacy=0x7fab24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.135+0000 7fab4547e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.135+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fab24009e70 con 0x7fab24009130 2024-08-19T10:06:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 client.175015.objecter ms_handle_connect 0x7fab24009130 2024-08-19T10:06:04.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 client.175015.objecter resend_mon_ops 2024-08-19T10:06:04.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab3e7fc640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.1:6790/0 conn(0x7fab24009130 legacy=0x7fab240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845506014 0 0) 0x7fab24009e70 con 0x7fab24009130 2024-08-19T10:06:04.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient(hunting): my global_id is 184646 2024-08-19T10:06:04.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab240065c0 con 0x7fab24009130 2024-08-19T10:06:04.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1790104630 0 0) 0x7fab240065c0 con 0x7fab24009130 2024-08-19T10:06:04.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab24009e70 con 0x7fab24009130 2024-08-19T10:06:04.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fab28004f40 con 0x7fab24009130 2024-08-19T10:06:04.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:06:04.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 10 monclient: dump: 2024-08-19T10:06:04.154 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.154 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.154 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.155 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.155 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.155 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.155 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.155 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.156 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.156 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.156 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.156 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.156 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.156 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.157 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 930620830 0 0) 0x7fab24009e70 con 0x7fab24009130 2024-08-19T10:06:04.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab2400c410 con 0x7fab24009130 2024-08-19T10:06:04.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.138+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7663 (unknown 0 0 418512962) 0x7fab28006490 con 0x7fab24009130 2024-08-19T10:06:04.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:04.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab377fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:04.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.1:6790/0 conn(0x7fab24009130 legacy=0x7fab240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab377fe640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.1:6790/0 conn(0x7fab24009130 legacy=0x7fab240095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:06:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 281387, 2024-08-19T10:06:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:06:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:04.198 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 10 librados: watch_flush enter 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 10 librados: watch_flush exit 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 20 client.175015.objecter shutdown clearing up homeless session... 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 10 client.175015.objecter successfully canceled tick 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6812/2549104415 conn(0x7fab20040d70 legacy=0x7fab20043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 10 monclient: shutdown 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 >> v1:172.21.15.82:6789/0 conn(0x7fab400a67c0 legacy=0x7fab401ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab3f7fe640 1 -- 172.21.15.1:0/3155543681 reap_dead start 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab3f7fe640 1 -- 172.21.15.1:0/3155543681 reap_dead start 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 shutdown_connections 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 >> 172.21.15.1:0/3155543681 conn(0x7fab400ac4b0 msgr2=0x7fab401419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 shutdown_connections 2024-08-19T10:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 -- 172.21.15.1:0/3155543681 wait complete. 2024-08-19T10:06:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.139+0000 7fab4547e640 1 librados: shutdown 2024-08-19T10:06:04.202 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:04.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-08-19T10:06:04.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:06:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.300+0000 7f626c2e5640 10 monclient: get_monmap_and_config 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.300+0000 7f626c2e5640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.300+0000 7f626c2e5640 10 monclient: monmap: 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:04.302674+0000 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:04.302674+0000 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.304 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.301+0000 7f626c2e5640 1 Processor -- start 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.301+0000 7f626c2e5640 1 -- start start 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.301+0000 7f626c2e5640 10 monclient: init 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient: _add_conns ranks=[1,8,3,5,2,6,0,4,7] 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient(hunting): picked mon.noname-f con 0x7f62640a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient(hunting): picked mon.noname-e con 0x7f62640a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient(hunting): picked mon.noname-g con 0x7f626413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62640a3b40 con 0x7f62640a67c0 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62640a8130 con 0x7f626413f5c0 2024-08-19T10:06:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f626406ede0 con 0x7f62640a6ba0 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626c2e5640 10 monclient(hunting): authenticate will time out at 1620.379198s 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f626a882640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f626413f5c0 legacy=0x7f62641419b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f6269880640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f62640a6ba0 0x7f62640a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:46876/0 (socket says 172.21.15.1:46876) 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.302+0000 7f6269880640 1 -- 172.21.15.1:0/789837742 learned_addr learned my addr 172.21.15.1:0/789837742 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f6269880640 1 -- 172.21.15.1:0/789837742 >> v1:172.21.15.1:6793/0 conn(0x7f62640a6ba0 legacy=0x7f62640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129363473 0 0) 0x7f626406ede0 con 0x7f62640a6ba0 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient(hunting): my global_id is 204308 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f624c003520 con 0x7f62640a6ba0 2024-08-19T10:06:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626a081640 1 -- 172.21.15.1:0/789837742 >> v1:172.21.15.82:6789/0 conn(0x7f62640a67c0 legacy=0x7f62640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 204943961 0 0) 0x7f624c003520 con 0x7f62640a6ba0 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 >> v1:172.21.15.82:6790/0 conn(0x7f626413f5c0 legacy=0x7f62641419b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 >> v1:172.21.15.82:6789/0 conn(0x7f62640a67c0 legacy=0x7f62640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 monclient: found mon.noname-e 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62640a92a0 con 0x7f62640a6ba0 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6258001ad0 con 0x7f62640a6ba0 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626c2e5640 5 monclient: authenticate success, global_id 204308 2024-08-19T10:06:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626c2e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: dump: 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626c2e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62580048a0 con 0x7f62640a6ba0 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626c2e5640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 1 -- 172.21.15.1:0/789837742 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6258004e10 con 0x7f62640a6ba0 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626c2e5640 4 set_mon_vals no callback set 2024-08-19T10:06:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.304+0000 7f626907f640 10 monclient: dump: 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 10 monclient: shutdown 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 1 -- 172.21.15.1:0/789837742 >> v1:172.21.15.1:6793/0 conn(0x7f62640a6ba0 legacy=0x7f62640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 1 -- 172.21.15.1:0/789837742 shutdown_connections 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 1 -- 172.21.15.1:0/789837742 >> 172.21.15.1:0/789837742 conn(0x7f62640ac4b0 msgr2=0x7f62640a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 1 -- 172.21.15.1:0/789837742 shutdown_connections 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.305+0000 7f626c2e5640 1 -- 172.21.15.1:0/789837742 wait complete. 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient: monmap: 2024-08-19T10:06:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 librados: starting msgr at 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 librados: starting objecter 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 Processor -- start 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 -- start start 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 librados: setting wanted keys 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 librados: calling monclient init 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient: init 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient: _add_conns ranks=[2,4,7,0,3,1,6,8,5] 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient(hunting): picked mon.noname-b con 0x7f62640a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient(hunting): picked mon.noname-c con 0x7f62640a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient(hunting): picked mon.noname-i con 0x7f626413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:06:04.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62640a2530 con 0x7f62640a67c0 2024-08-19T10:06:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62640a9d80 con 0x7f62640a6ba0 2024-08-19T10:06:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f626409f8a0 con 0x7f626413f5c0 2024-08-19T10:06:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626c2e5640 10 monclient(hunting): authenticate will time out at 1620.383898s 2024-08-19T10:06:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626a081640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f62640a67c0 0x7f626409ea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36794/0 (socket says 172.21.15.1:36794) 2024-08-19T10:06:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.306+0000 7f626a081640 1 -- 172.21.15.1:0/2607150346 learned_addr learned my addr 172.21.15.1:0/2607150346 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626a081640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6790/0 conn(0x7f62640a67c0 legacy=0x7f626409ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 client.?.objecter ms_handle_connect 0x7f62640a67c0 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6269880640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f62640a6ba0 legacy=0x7f626409f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 client.?.objecter ms_handle_connect 0x7f62640a6ba0 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824366424 0 0) 0x7f62640a9d80 con 0x7f62640a6ba0 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): my global_id is 194485 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6250004eb0 con 0x7f62640a6ba0 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121257376 0 0) 0x7f62640a2530 con 0x7f62640a67c0 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): my global_id is 184655 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62640a9d80 con 0x7f62640a67c0 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 client.?.objecter ms_handle_connect 0x7f626413f5c0 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626a882640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.82:6792/0 conn(0x7f626413f5c0 legacy=0x7f62641d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2987662466 0 0) 0x7f6250004eb0 con 0x7f62640a6ba0 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626a081640 1 --1- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6790/0 conn(0x7f62640a67c0 0x7f626409ea80 :-1 s=OPENED pgs=3560 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f62640a9d80 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:06:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62640a2530 con 0x7f62640a6ba0 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6258004d90 con 0x7f62640a6ba0 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): dump: 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 204958667 0 0) 0x7f62640a9d80 con 0x7f62640a67c0 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6250004eb0 con 0x7f62640a67c0 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957824767 0 0) 0x7f626409f8a0 con 0x7f626413f5c0 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient(hunting): my global_id is 204307 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62640a9d80 con 0x7f626413f5c0 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3155098276 0 0) 0x7f62640a2530 con 0x7f62640a6ba0 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.82:6792/0 conn(0x7f626413f5c0 legacy=0x7f62641d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6790/0 conn(0x7f62640a67c0 legacy=0x7f626409ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 monclient: found mon.c 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62640a19a0 con 0x7f62640a6ba0 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6254003990 con 0x7f62640a67c0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 5 monclient: authenticate success, global_id 194485 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 10 monclient: _renew_subs 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62640a04b0 con 0x7f62640a6ba0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 10 client.194485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 10 monclient: _renew_subs 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f626413f280 con 0x7f62640a6ba0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626c2e5640 1 librados: init done 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62580048a0 con 0x7f62640a6ba0 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6258001d20 con 0x7f62640a6ba0 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f6262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.309+0000 7f626887e640 4 set_mon_vals no callback set 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.310+0000 7f6262ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.310+0000 7f6262ffd640 10 monclient: dump: 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.310+0000 7f626c2e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.310+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f62440065c0 con 0x7f6244002860 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.310+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6258011a00 con 0x7f62640a6ba0 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.311+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(577..577 src has 299..577) ==== 5931+0+0 (unknown 601312446 0 0) 0x7f625804e3d0 con 0x7f62640a6ba0 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.311+0000 7f6262ffd640 10 client.194485.objecter ms_dispatch 0x7f6264000b90 osd_map(577..577 src has 299..577) 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.311+0000 7f6262ffd640 3 client.194485.objecter handle_osd_map got epochs [577,577] > 0 2024-08-19T10:06:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.311+0000 7f6262ffd640 3 client.194485.objecter handle_osd_map decoding full epoch 577 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.311+0000 7f6262ffd640 20 client.194485.objecter dump_active .. 0 homeless 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.311+0000 7f626a882640 10 monclient: get_auth_request con 0x7f62500411c0 auth_method 0 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.312+0000 7f6262ffd640 10 client.194485.objecter ms_handle_connect 0x7f6244002860 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.312+0000 7f626a081640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f6244002860 legacy=0x7f6244004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.312+0000 7f6262ffd640 10 client.194485.objecter resend_mon_ops 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92921566 0 0) 0x7f62440065c0 con 0x7f6244002860 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient(hunting): my global_id is 194494 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62640a2530 con 0x7f6244002860 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3059248018 0 0) 0x7f62640a2530 con 0x7f6244002860 2024-08-19T10:06:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62440065c0 con 0x7f6244002860 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6254004190 con 0x7f6244002860 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient: dump: 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.335 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2787555906 0 0) 0x7f62440065c0 con 0x7f6244002860 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f624400a160 con 0x7f6244002860 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 client.194485.objecter ms_handle_connect 0x7f62500411c0 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6254005b40 con 0x7f6244002860 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f6244002860 legacy=0x7f6244004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.313+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f6244002860 legacy=0x7f6244004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.319+0000 7f626c2e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.319+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6244009e30 con 0x7f6244009110 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 client.194485.objecter ms_handle_connect 0x7f6244009110 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f626a081640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f6244009110 legacy=0x7f6244009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 client.194485.objecter resend_mon_ops 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 528218500 0 0) 0x7f6244009e30 con 0x7f6244009110 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient(hunting): my global_id is 194503 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62440065c0 con 0x7f6244009110 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3318982750 0 0) 0x7f62440065c0 con 0x7f6244009110 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6244009e30 con 0x7f6244009110 2024-08-19T10:06:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f62540047e0 con 0x7f6244009110 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient: dump: 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3349605638 0 0) 0x7f6244009e30 con 0x7f6244009110 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f624400c3f0 con 0x7f6244009110 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7663 (unknown 0 0 4056783093) 0x7f6254008150 con 0x7f6244009110 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f6244009110 legacy=0x7f6244009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.322+0000 7f6262ffd640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f6244009110 legacy=0x7f6244009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 152010, 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:04.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.344 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.345 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:04.346 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.347 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:04.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.355 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.356 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.357 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:06:04.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.363 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:04.363 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 10 librados: watch_flush enter 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 10 librados: watch_flush exit 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 20 client.194485.objecter shutdown clearing up homeless session... 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 10 client.194485.objecter successfully canceled tick 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.82:6812/2549104415 conn(0x7f62500411c0 legacy=0x7f6250043680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 10 monclient: shutdown 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 >> v1:172.21.15.1:6791/0 conn(0x7f62640a6ba0 legacy=0x7f626409f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626a882640 1 -- 172.21.15.1:0/2607150346 reap_dead start 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626a882640 1 -- 172.21.15.1:0/2607150346 reap_dead start 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 shutdown_connections 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 >> 172.21.15.1:0/2607150346 conn(0x7f62640ac4b0 msgr2=0x7f62640a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 shutdown_connections 2024-08-19T10:06:04.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 -- 172.21.15.1:0/2607150346 wait complete. 2024-08-19T10:06:04.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.323+0000 7f626c2e5640 1 librados: shutdown 2024-08-19T10:06:04.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:04.366 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-08-19T10:06:04.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:06:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.464+0000 7fbc53728640 10 monclient: get_monmap_and_config 2024-08-19T10:06:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.464+0000 7fbc53728640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.464+0000 7fbc53728640 10 monclient: monmap: 2024-08-19T10:06:04.466 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:04.466309+0000 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:04.466309+0000 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.467 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.468 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.464+0000 7fbc53728640 1 Processor -- start 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 1 -- start start 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient: init 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient: _add_conns ranks=[8,5,2,0,3,7,1,6,4] 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient(hunting): picked mon.noname-e con 0x7fbc4c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:06:04.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient(hunting): picked mon.noname-h con 0x7fbc4c0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4c13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c0a3b40 con 0x7fbc4c13f5c0 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c0a8130 con 0x7fbc4c0a6ba0 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c06ede0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc53728640 10 monclient(hunting): authenticate will time out at 1620.542707s 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc514c4640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fbc4c0a67c0 0x7fbc4c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:46886/0 (socket says 172.21.15.1:46886) 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc51cc5640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fbc4c13f5c0 0x7fbc4c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36804/0 (socket says 172.21.15.1:36804) 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc514c4640 1 -- 172.21.15.1:0/4039225009 learned_addr learned my addr 172.21.15.1:0/4039225009 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.465+0000 7fbc50cc3640 1 -- 172.21.15.1:0/4039225009 >> v1:172.21.15.82:6791/0 conn(0x7fbc4c0a6ba0 legacy=0x7fbc4c0a6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.467+0000 7fbc514c4640 1 -- 172.21.15.1:0/4039225009 >> v1:172.21.15.1:6793/0 conn(0x7fbc4c0a67c0 legacy=0x7fbc4c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.467+0000 7fbc51cc5640 1 -- 172.21.15.1:0/4039225009 >> v1:172.21.15.1:6790/0 conn(0x7fbc4c13f5c0 legacy=0x7fbc4c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2811031473 0 0) 0x7fbc4c0a3b40 con 0x7fbc4c13f5c0 2024-08-19T10:06:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): my global_id is 184664 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc38003520 con 0x7fbc4c13f5c0 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167819426 0 0) 0x7fbc4c06ede0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient(hunting): my global_id is 204317 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc4c0a3b40 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2781149242 0 0) 0x7fbc4c0a3b40 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 >> v1:172.21.15.82:6791/0 conn(0x7fbc4c0a6ba0 legacy=0x7fbc4c0a6fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:06:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 >> v1:172.21.15.1:6790/0 conn(0x7fbc4c13f5c0 legacy=0x7fbc4c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 monclient: found mon.noname-e 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c0a92a0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc48001a50 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc53728640 5 monclient: authenticate success, global_id 204317 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc53728640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: dump: 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc53728640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc48001d20 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc53728640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 1 -- 172.21.15.1:0/4039225009 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc48004be0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc53728640 4 set_mon_vals no callback set 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.468+0000 7fbc43fff640 10 monclient: dump: 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 10 monclient: shutdown 2024-08-19T10:06:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 -- 172.21.15.1:0/4039225009 >> v1:172.21.15.1:6793/0 conn(0x7fbc4c0a67c0 legacy=0x7fbc4c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 -- 172.21.15.1:0/4039225009 shutdown_connections 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 -- 172.21.15.1:0/4039225009 >> 172.21.15.1:0/4039225009 conn(0x7fbc4c0ac4b0 msgr2=0x7fbc4c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 -- 172.21.15.1:0/4039225009 shutdown_connections 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 -- 172.21.15.1:0/4039225009 wait complete. 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 10 monclient: monmap: 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 librados: starting msgr at 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.469+0000 7fbc53728640 1 librados: starting objecter 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 Processor -- start 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 -- start start 2024-08-19T10:06:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 librados: setting wanted keys 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 librados: calling monclient init 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient: init 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient: _add_conns ranks=[0,5,3,6,8,4,7,1,2] 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient(hunting): picked mon.noname-a con 0x7fbc4c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient(hunting): picked mon.noname-h con 0x7fbc4c0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient(hunting): picked mon.noname-g con 0x7fbc4c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c0a1710 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c0aa030 con 0x7fbc4c13f5c0 2024-08-19T10:06:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c0a9cc0 con 0x7fbc4c0a6ba0 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc53728640 10 monclient(hunting): authenticate will time out at 1620.547680s 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc514c4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbc4c0a67c0 0x7fbc4c0a0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53542/0 (socket says 172.21.15.1:53542) 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc514c4640 1 -- 172.21.15.1:0/148039925 learned_addr learned my addr 172.21.15.1:0/148039925 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc51cc5640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.82:6790/0 conn(0x7fbc4c13f5c0 legacy=0x7fbc4c1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc50cc3640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.82:6791/0 conn(0x7fbc4c0a6ba0 legacy=0x7fbc4c0a0710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.470+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.472+0000 7fbc514c4640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6789/0 conn(0x7fbc4c0a67c0 legacy=0x7fbc4c0a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.472+0000 7fbc41ffb640 10 client.?.objecter ms_handle_connect 0x7fbc4c0a67c0 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.472+0000 7fbc41ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.472+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3829172227 0 0) 0x7fbc4c0a1710 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): my global_id is 204300 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc30004ee0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3106250641 0 0) 0x7fbc30004ee0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc4c0a1710 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc48004b20 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): dump: 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2255439457 0 0) 0x7fbc4c0a1710 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.82:6791/0 conn(0x7fbc4c0a6ba0 legacy=0x7fbc4c0a0710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.82:6790/0 conn(0x7fbc4c13f5c0 legacy=0x7fbc4c1d6250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 monclient: found mon.a 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c13a880 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 5 monclient: authenticate success, global_id 204300 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 10 monclient: _renew_subs 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:06:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c139fe0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc48005210 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc48005700 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc41ffb640 10 monclient: dump: 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 10 client.204300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 10 monclient: _renew_subs 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc437fe640 4 set_mon_vals no callback set 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:06:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc4c139ab0 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.473+0000 7fbc53728640 1 librados: init done 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbc48011b60 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc53728640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc4c0008d0 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(577..577 src has 299..577) ==== 5931+0+0 (unknown 601312446 0 0) 0x7fbc4804e550 con 0x7fbc4c0a67c0 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc41ffb640 10 client.204300.objecter ms_dispatch 0x7fbc4c000b90 osd_map(577..577 src has 299..577) 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc41ffb640 3 client.204300.objecter handle_osd_map got epochs [577,577] > 0 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc41ffb640 3 client.204300.objecter handle_osd_map decoding full epoch 577 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.474+0000 7fbc41ffb640 20 client.204300.objecter dump_active .. 0 homeless 2024-08-19T10:06:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.475+0000 7fbc51cc5640 10 monclient: get_auth_request con 0x7fbc30040d10 auth_method 0 2024-08-19T10:06:04.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc50cc3640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6792/0 conn(0x7fbc4c0a0e20 legacy=0x7fbc4c13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 10 client.204300.objecter ms_handle_connect 0x7fbc4c0a0e20 2024-08-19T10:06:04.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 10 client.204300.objecter resend_mon_ops 2024-08-19T10:06:04.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232940410 0 0) 0x7fbc4c0008d0 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 10 monclient(hunting): my global_id is 204270 2024-08-19T10:06:04.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.476+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc4c0a1710 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 369228520 0 0) 0x7fbc4c0a1710 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc4c0008d0 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc3c0043b0 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 10 monclient: dump: 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.500 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.501 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.501 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.501 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.501 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.501 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.501 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 102114465 0 0) 0x7fbc4c0008d0 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc4c13a450 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 10 client.204300.objecter ms_handle_connect 0x7fbc30040d10 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbc3c006750 con 0x7fbc4c0a0e20 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6792/0 conn(0x7fbc4c0a0e20 legacy=0x7fbc4c13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.477+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6792/0 conn(0x7fbc4c0a0e20 legacy=0x7fbc4c13f9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.483+0000 7fbc53728640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.483+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc4c02e5b0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc50cc3640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6792/0 conn(0x7fbc4c02e1d0 legacy=0x7fbc4c1400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 client.204300.objecter ms_handle_connect 0x7fbc4c02e1d0 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 client.204300.objecter resend_mon_ops 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942365021 0 0) 0x7fbc4c02e5b0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient(hunting): my global_id is 204279 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc4c0008d0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2890501649 0 0) 0x7fbc4c0008d0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc4c02e5b0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbc3c004d80 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient: dump: 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.504 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 867773535 0 0) 0x7fbc4c02e5b0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc4c1407c0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 1551583494) 0x7fbc3c0061c0 con 0x7fbc4c02e1d0 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6792/0 conn(0x7fbc4c02e1d0 legacy=0x7fbc4c1400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.486+0000 7fbc41ffb640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6792/0 conn(0x7fbc4c02e1d0 legacy=0x7fbc4c1400b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.506 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30357, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:04.529 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:04.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 10 librados: watch_flush enter 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 10 librados: watch_flush exit 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 20 client.204300.objecter shutdown clearing up homeless session... 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 10 client.204300.objecter successfully canceled tick 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.82:6812/2549104415 conn(0x7fbc30040d10 legacy=0x7fbc300431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 10 monclient: shutdown 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 >> v1:172.21.15.1:6789/0 conn(0x7fbc4c0a67c0 legacy=0x7fbc4c0a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc51cc5640 1 -- 172.21.15.1:0/148039925 reap_dead start 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc51cc5640 1 -- 172.21.15.1:0/148039925 reap_dead start 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 shutdown_connections 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 >> 172.21.15.1:0/148039925 conn(0x7fbc4c0ac4b0 msgr2=0x7fbc4c0aa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 shutdown_connections 2024-08-19T10:06:04.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 -- 172.21.15.1:0/148039925 wait complete. 2024-08-19T10:06:04.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.487+0000 7fbc53728640 1 librados: shutdown 2024-08-19T10:06:04.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:04.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-08-19T10:06:04.533 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:06:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.630+0000 7f4ba21f2640 10 monclient: get_monmap_and_config 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.630+0000 7f4ba21f2640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.630+0000 7f4ba21f2640 10 monclient: monmap: 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:04.632658+0000 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:04.632658+0000 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.634 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 1 Processor -- start 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 1 -- start start 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 10 monclient: init 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 10 monclient: _add_conns ranks=[4,7,1,0,8,6,3,2,5] 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 10 monclient(hunting): picked mon.noname-c con 0x7f4b9c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 10 monclient(hunting): picked mon.noname-i con 0x7f4b9c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 10 monclient(hunting): picked mon.noname-f con 0x7f4b9c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.631+0000 7f4ba21f2640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c0a3b40 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.632+0000 7f4ba21f2640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c0a8130 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.632+0000 7f4ba21f2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c06ede0 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.632+0000 7f4ba21f2640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.632+0000 7f4ba21f2640 10 monclient(hunting): authenticate will time out at 1620.709038s 2024-08-19T10:06:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.632+0000 7f4b9b7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4b9c0a67c0 0x7f4b9c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57398/0 (socket says 172.21.15.1:57398) 2024-08-19T10:06:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.632+0000 7f4b9b7fe640 1 -- 172.21.15.1:0/728117995 learned_addr learned my addr 172.21.15.1:0/728117995 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9b7fe640 1 -- 172.21.15.1:0/728117995 >> v1:172.21.15.1:6791/0 conn(0x7f4b9c0a67c0 legacy=0x7f4b9c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952512067 0 0) 0x7f4b9c0a8130 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): my global_id is 194512 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b84003520 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9bfff640 1 -- 172.21.15.1:0/728117995 >> v1:172.21.15.82:6789/0 conn(0x7f4b9c13f5c0 legacy=0x7f4b9c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9affd640 1 -- 172.21.15.1:0/728117995 >> v1:172.21.15.82:6792/0 conn(0x7f4b9c0a6ba0 legacy=0x7f4b9c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1940734290 0 0) 0x7f4b9c0a3b40 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient(hunting): my global_id is 175033 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9c0a8130 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 427979067 0 0) 0x7f4b84003520 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 >> v1:172.21.15.82:6792/0 conn(0x7f4b9c0a6ba0 legacy=0x7f4b9c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 >> v1:172.21.15.82:6789/0 conn(0x7f4b9c13f5c0 legacy=0x7f4b9c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 monclient: found mon.noname-c 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9c0a92a0 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b8c001ae0 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4ba21f2640 5 monclient: authenticate success, global_id 194512 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: dump: 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4ba21f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b8c0048a0 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 1 -- 172.21.15.1:0/728117995 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b8c004e10 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4ba21f2640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4ba21f2640 4 set_mon_vals no callback set 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.634+0000 7f4b9a7fc640 10 monclient: dump: 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.651 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 10 monclient: shutdown 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 1 -- 172.21.15.1:0/728117995 >> v1:172.21.15.1:6791/0 conn(0x7f4b9c0a67c0 legacy=0x7f4b9c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 1 -- 172.21.15.1:0/728117995 shutdown_connections 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 1 -- 172.21.15.1:0/728117995 >> 172.21.15.1:0/728117995 conn(0x7f4b9c0ac4b0 msgr2=0x7f4b9c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 1 -- 172.21.15.1:0/728117995 shutdown_connections 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 1 -- 172.21.15.1:0/728117995 wait complete. 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.635+0000 7f4ba21f2640 10 monclient: monmap: 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 librados: starting msgr at 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 librados: starting objecter 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 Processor -- start 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 -- start start 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 librados: setting wanted keys 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 librados: calling monclient init 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient: init 2024-08-19T10:06:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient: _add_conns ranks=[5,8,7,6,0,3,2,1,4] 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient(hunting): picked mon.noname-h con 0x7f4b9c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient(hunting): picked mon.noname-e con 0x7f4b9c0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient(hunting): picked mon.noname-i con 0x7f4b9c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c1cf070 con 0x7f4b9c0a67c0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c142e80 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9c13afa0 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4ba21f2640 10 monclient(hunting): authenticate will time out at 1620.713918s 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.636+0000 7f4b9b7fe640 1 -- >> v1:172.21.15.82:6791/0 conn(0x7f4b9c0a67c0 legacy=0x7f4b9c1ce250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.637+0000 7f4b9affd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f4b9c0a6ba0 0x7f4b9c1ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:46902/0 (socket says 172.21.15.1:46902) 2024-08-19T10:06:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.637+0000 7f4b9affd640 1 -- 172.21.15.1:0/1816337448 learned_addr learned my addr 172.21.15.1:0/1816337448 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b9affd640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b9c0a6ba0 legacy=0x7f4b9c1ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 client.?.objecter ms_handle_connect 0x7f4b9c0a6ba0 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 client.?.objecter ms_handle_connect 0x7f4b9c13f5c0 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b9bfff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.82:6792/0 conn(0x7f4b9c13f5c0 legacy=0x7f4b9c1d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655715842 0 0) 0x7f4b9c13afa0 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): my global_id is 204326 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b88004eb0 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429868397 0 0) 0x7f4b9c142e80 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): my global_id is 204325 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9c13afa0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b9affd640 1 --1- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b9c0a6ba0 0x7f4b9c1ce960 :-1 s=OPENED pgs=3651 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f4b90003610 mon_map magic: 0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1825947185 0 0) 0x7f4b88004eb0 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b9c142e80 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2411403361 0 0) 0x7f4b9c13afa0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b88004eb0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b94002070 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:06:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): dump: 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b90003610 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient(hunting): dump: 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3451211428 0 0) 0x7f4b88004eb0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b9c0a6ba0 legacy=0x7f4b9c1ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.82:6791/0 conn(0x7f4b9c0a67c0 legacy=0x7f4b9c1ce250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 monclient: found mon.i 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9c139ab0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 5 monclient: authenticate success, global_id 204325 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2600534542 0 0) 0x7f4b9c142e80 con 0x7f4b9c0a6ba0 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 10 monclient: _renew_subs 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9c13be10 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 10 client.204325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 10 monclient: _renew_subs 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b9c13e680 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4ba21f2640 1 librados: init done 2024-08-19T10:06:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b94003c40 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b940041b0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b99ffb640 4 set_mon_vals no callback set 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.639+0000 7f4b83fff640 10 monclient: dump: 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.640+0000 7f4ba21f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.640+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4b7c0065c0 con 0x7f4b7c002860 2024-08-19T10:06:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.640+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4b940043d0 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.641+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(577..577 src has 299..577) ==== 5931+0+0 (unknown 601312446 0 0) 0x7f4b9404e740 con 0x7f4b9c13f5c0 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.641+0000 7f4b83fff640 10 client.204325.objecter ms_dispatch 0x7f4b9c000b90 osd_map(577..577 src has 299..577) 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.641+0000 7f4b83fff640 3 client.204325.objecter handle_osd_map got epochs [577,577] > 0 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.641+0000 7f4b83fff640 3 client.204325.objecter handle_osd_map decoding full epoch 577 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.641+0000 7f4b83fff640 20 client.204325.objecter dump_active .. 0 homeless 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.641+0000 7f4b9affd640 10 monclient: get_auth_request con 0x7f4b880413a0 auth_method 0 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 10 client.204325.objecter ms_handle_connect 0x7f4b7c002860 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 10 client.204325.objecter resend_mon_ops 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b9b7fe640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b7c002860 legacy=0x7f4b7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773901449 0 0) 0x7f4b7c0065c0 con 0x7f4b7c002860 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.642+0000 7f4b83fff640 10 monclient(hunting): my global_id is 204335 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9c142e80 con 0x7f4b7c002860 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2033540368 0 0) 0x7f4b9c142e80 con 0x7f4b7c002860 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b7c0065c0 con 0x7f4b7c002860 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b8c004dd0 con 0x7f4b7c002860 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 10 monclient: dump: 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.666 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3666772787 0 0) 0x7f4b7c0065c0 con 0x7f4b7c002860 2024-08-19T10:06:04.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b7c00a160 con 0x7f4b7c002860 2024-08-19T10:06:04.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4b8c001d30 con 0x7f4b7c002860 2024-08-19T10:06:04.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:04.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b7c002860 legacy=0x7f4b7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b7c002860 legacy=0x7f4b7c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.643+0000 7f4b83fff640 10 client.204325.objecter ms_handle_connect 0x7f4b880413a0 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.649+0000 7f4ba21f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.649+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4b7c009e50 con 0x7f4b7c009110 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b9b7fe640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b7c009110 legacy=0x7f4b7c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 client.204325.objecter ms_handle_connect 0x7f4b7c009110 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 client.204325.objecter resend_mon_ops 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451660350 0 0) 0x7f4b7c009e50 con 0x7f4b7c009110 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient(hunting): my global_id is 204344 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b7c0065c0 con 0x7f4b7c009110 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4230937831 0 0) 0x7f4b7c0065c0 con 0x7f4b7c009110 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b7c009e50 con 0x7f4b7c009110 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4b8c005490 con 0x7f4b7c009110 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient: dump: 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.671 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 831785193 0 0) 0x7f4b7c009e50 con 0x7f4b7c009110 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4b7c00c3f0 con 0x7f4b7c009110 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 705834051) 0x7f4b8c001ac0 con 0x7f4b7c009110 2024-08-19T10:06:04.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b7c009110 legacy=0x7f4b7c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4b83fff640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.1:6793/0 conn(0x7f4b7c009110 legacy=0x7f4b7c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30525, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.675 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.676 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:04.677 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.678 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:04.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.680 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:04.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.686 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:04.689 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.690 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.692 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:04.693 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:04.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4ba21f2640 10 librados: watch_flush enter 2024-08-19T10:06:04.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4ba21f2640 10 librados: watch_flush exit 2024-08-19T10:06:04.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4ba21f2640 20 client.204325.objecter shutdown clearing up homeless session... 2024-08-19T10:06:04.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.652+0000 7f4ba21f2640 10 client.204325.objecter successfully canceled tick 2024-08-19T10:06:04.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4b880413a0 legacy=0x7f4b88043860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 10 monclient: shutdown 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 >> v1:172.21.15.82:6792/0 conn(0x7f4b9c13f5c0 legacy=0x7f4b9c1d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4b9bfff640 1 -- 172.21.15.1:0/1816337448 reap_dead start 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4b9bfff640 1 -- 172.21.15.1:0/1816337448 reap_dead start 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 shutdown_connections 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 >> 172.21.15.1:0/1816337448 conn(0x7f4b9c0ac4b0 msgr2=0x7f4b9c13ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 shutdown_connections 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 -- 172.21.15.1:0/1816337448 wait complete. 2024-08-19T10:06:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.653+0000 7f4ba21f2640 1 librados: shutdown 2024-08-19T10:06:04.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:04.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-08-19T10:06:04.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:06:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.794+0000 7f568eef6640 10 monclient: get_monmap_and_config 2024-08-19T10:06:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.794+0000 7f568eef6640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.794+0000 7f568eef6640 10 monclient: monmap: 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:04.796386+0000 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:04.796386+0000 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.798 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 1 Processor -- start 2024-08-19T10:06:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 1 -- start start 2024-08-19T10:06:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient: init 2024-08-19T10:06:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient: _add_conns ranks=[1,0,2,5,8,6,7,3,4] 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient(hunting): picked mon.noname-f con 0x7f56880a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient(hunting): picked mon.noname-a con 0x7f56880a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient(hunting): picked mon.noname-b con 0x7f568813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56880a3b40 con 0x7f56880a6ba0 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56880a8130 con 0x7f56880a67c0 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568806ede0 con 0x7f568813f5c0 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f568eef6640 10 monclient(hunting): authenticate will time out at 1620.872767s 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f5687fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56880a6ba0 0x7f56880a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53546/0 (socket says 172.21.15.1:53546) 2024-08-19T10:06:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.795+0000 7f5687fff640 1 -- 172.21.15.1:0/1994506927 learned_addr learned my addr 172.21.15.1:0/1994506927 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568d493640 1 -- 172.21.15.1:0/1994506927 >> v1:172.21.15.1:6790/0 conn(0x7f568813f5c0 legacy=0x7f56881419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f5687fff640 1 -- 172.21.15.1:0/1994506927 >> v1:172.21.15.1:6789/0 conn(0x7f56880a6ba0 legacy=0x7f56880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624846997 0 0) 0x7f568806ede0 con 0x7f568813f5c0 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): my global_id is 184673 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5674003520 con 0x7f568813f5c0 2024-08-19T10:06:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378561545 0 0) 0x7f56880a3b40 con 0x7f56880a6ba0 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient(hunting): my global_id is 204309 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568806ede0 con 0x7f56880a6ba0 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568cc92640 1 -- 172.21.15.1:0/1994506927 >> v1:172.21.15.82:6789/0 conn(0x7f56880a67c0 legacy=0x7f56880a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2178589442 0 0) 0x7f5674003520 con 0x7f568813f5c0 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 >> v1:172.21.15.82:6789/0 conn(0x7f56880a67c0 legacy=0x7f56880a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 >> v1:172.21.15.1:6789/0 conn(0x7f56880a6ba0 legacy=0x7f56880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 monclient: found mon.noname-b 2024-08-19T10:06:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56880a92a0 con 0x7f568813f5c0 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5680001be0 con 0x7f568813f5c0 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568eef6640 5 monclient: authenticate success, global_id 184673 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568eef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: dump: 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568eef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5680003610 con 0x7f568813f5c0 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 1 -- 172.21.15.1:0/1994506927 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5680003b00 con 0x7f568813f5c0 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f56877fe640 10 monclient: dump: 2024-08-19T10:06:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568eef6640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.798+0000 7f568eef6640 4 set_mon_vals no callback set 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 10 monclient: shutdown 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 -- 172.21.15.1:0/1994506927 >> v1:172.21.15.1:6790/0 conn(0x7f568813f5c0 legacy=0x7f56881419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 -- 172.21.15.1:0/1994506927 shutdown_connections 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 -- 172.21.15.1:0/1994506927 >> 172.21.15.1:0/1994506927 conn(0x7f56880ac4b0 msgr2=0x7f56880a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 -- 172.21.15.1:0/1994506927 shutdown_connections 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 -- 172.21.15.1:0/1994506927 wait complete. 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 10 monclient: monmap: 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 librados: starting msgr at 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.799+0000 7f568eef6640 1 librados: starting objecter 2024-08-19T10:06:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 Processor -- start 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 -- start start 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 librados: setting wanted keys 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 librados: calling monclient init 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient: init 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient: _add_conns ranks=[3,1,2,5,6,0,8,7,4] 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient(hunting): picked mon.noname-g con 0x7f56880a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient(hunting): picked mon.noname-f con 0x7f56880a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient(hunting): picked mon.noname-b con 0x7f568813f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56880a9d80 con 0x7f56880a6ba0 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5688142c60 con 0x7f568813f5c0 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56880a1720 con 0x7f56880a67c0 2024-08-19T10:06:04.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568eef6640 10 monclient(hunting): authenticate will time out at 1620.877775s 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568cc92640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f56880a67c0 legacy=0x7f568809fd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568d493640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f568813f5c0 0x7f56880a0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36812/0 (socket says 172.21.15.1:36812) 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.800+0000 7f568d493640 1 -- 172.21.15.1:0/2793063718 learned_addr learned my addr 172.21.15.1:0/2793063718 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.802+0000 7f568d493640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.1:6790/0 conn(0x7f568813f5c0 legacy=0x7f56880a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.802+0000 7f56857fa640 10 client.?.objecter ms_handle_connect 0x7f568813f5c0 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226043503 0 0) 0x7f5688142c60 con 0x7f568813f5c0 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): my global_id is 184682 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5668004eb0 con 0x7f568813f5c0 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 client.?.objecter ms_handle_connect 0x7f56880a6ba0 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f5687fff640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f56880a6ba0 legacy=0x7f56880a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 274629313 0 0) 0x7f5668004eb0 con 0x7f568813f5c0 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5688142c60 con 0x7f568813f5c0 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5680001ba0 con 0x7f568813f5c0 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:06:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): dump: 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788957309 0 0) 0x7f56880a9d80 con 0x7f56880a6ba0 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient(hunting): my global_id is 175051 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5668004eb0 con 0x7f56880a6ba0 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 131517007 0 0) 0x7f5688142c60 con 0x7f568813f5c0 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6790/0 conn(0x7f56880a67c0 legacy=0x7f568809fd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f56880a6ba0 legacy=0x7f56880a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 monclient: found mon.b 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568813a080 con 0x7f568813f5c0 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 5 monclient: authenticate success, global_id 184682 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 10 monclient: _renew_subs 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f568813a920 con 0x7f568813f5c0 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 10 client.184682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 10 monclient: _renew_subs 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f56881d7440 con 0x7f568813f5c0 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f568eef6640 1 librados: init done 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5680003b20 con 0x7f568813f5c0 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5680004340 con 0x7f568813f5c0 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f5686ffd640 4 set_mon_vals no callback set 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.803+0000 7f56857fa640 10 monclient: dump: 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f568eef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f566c0065c0 con 0x7f566c002860 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f5680011a80 con 0x7f568813f5c0 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(577..577 src has 299..577) ==== 5931+0+0 (unknown 601312446 0 0) 0x7f568004e3f0 con 0x7f568813f5c0 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f56857fa640 10 client.184682.objecter ms_dispatch 0x7f5688000b90 osd_map(577..577 src has 299..577) 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f56857fa640 3 client.184682.objecter handle_osd_map got epochs [577,577] > 0 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f56857fa640 3 client.184682.objecter handle_osd_map decoding full epoch 577 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f56857fa640 20 client.184682.objecter dump_active .. 0 homeless 2024-08-19T10:06:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.804+0000 7f5687fff640 10 monclient: get_auth_request con 0x7f5668040ed0 auth_method 0 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.806+0000 7f568cc92640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f566c002860 legacy=0x7f566c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.806+0000 7f56857fa640 10 client.184682.objecter ms_handle_connect 0x7f566c002860 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.806+0000 7f56857fa640 10 client.184682.objecter resend_mon_ops 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3621468997 0 0) 0x7f566c0065c0 con 0x7f566c002860 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient(hunting): my global_id is 175060 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5688142c60 con 0x7f566c002860 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 client.184682.objecter ms_handle_connect 0x7f5668040ed0 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4017779644 0 0) 0x7f5688142c60 con 0x7f566c002860 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f566c0065c0 con 0x7f566c002860 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5678001fa0 con 0x7f566c002860 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient: dump: 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.830 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3837372145 0 0) 0x7f566c0065c0 con 0x7f566c002860 2024-08-19T10:06:04.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f566c00a160 con 0x7f566c002860 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5678006b90 con 0x7f566c002860 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.807+0000 7f56857fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.808+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f566c002860 legacy=0x7f566c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.808+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f566c002860 legacy=0x7f566c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.814+0000 7f568eef6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.814+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f566c009e70 con 0x7f566c009130 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f568cc92640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f566c009130 legacy=0x7f566c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 10 client.184682.objecter ms_handle_connect 0x7f566c009130 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 10 client.184682.objecter resend_mon_ops 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879593402 0 0) 0x7f566c009e70 con 0x7f566c009130 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 10 monclient(hunting): my global_id is 175069 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f566c0065c0 con 0x7f566c009130 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.816+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3847152743 0 0) 0x7f566c0065c0 con 0x7f566c009130 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f566c009e70 con 0x7f566c009130 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f5678004100 con 0x7f566c009130 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 10 monclient: dump: 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.833 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.834 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2258699279 0 0) 0x7f566c009e70 con 0x7f566c009130 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f566c00c410 con 0x7f566c009130 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7830 (unknown 0 0 171303538) 0x7f56780043b0 con 0x7f566c009130 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f566c009130 legacy=0x7f566c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f56857fa640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6789/0 conn(0x7f566c009130 legacy=0x7f566c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 414877, 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:04.853 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:04.854 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.855 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:04.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:04.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:04.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:04.857 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:04.857 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 10 librados: watch_flush enter 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 10 librados: watch_flush exit 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 20 client.184682.objecter shutdown clearing up homeless session... 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 10 client.184682.objecter successfully canceled tick 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.82:6812/2549104415 conn(0x7f5668040ed0 legacy=0x7f5668043390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 10 monclient: shutdown 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 >> v1:172.21.15.1:6790/0 conn(0x7f568813f5c0 legacy=0x7f56880a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568d493640 1 -- 172.21.15.1:0/2793063718 reap_dead start 2024-08-19T10:06:04.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.817+0000 7f568d493640 1 -- 172.21.15.1:0/2793063718 reap_dead start 2024-08-19T10:06:04.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.818+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 shutdown_connections 2024-08-19T10:06:04.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.818+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 >> 172.21.15.1:0/2793063718 conn(0x7f56880ac4b0 msgr2=0x7f56880ab2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.818+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 shutdown_connections 2024-08-19T10:06:04.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.818+0000 7f568eef6640 1 -- 172.21.15.1:0/2793063718 wait complete. 2024-08-19T10:06:04.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.818+0000 7f568eef6640 1 librados: shutdown 2024-08-19T10:06:04.860 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:04.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-08-19T10:06:04.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:06:04.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.961+0000 7f6c90fa3640 10 monclient: get_monmap_and_config 2024-08-19T10:06:04.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.961+0000 7f6c90fa3640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.961+0000 7f6c90fa3640 10 monclient: monmap: 2024-08-19T10:06:04.963 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.963 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:06:04.963096+0000 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:06:04.963096+0000 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.965 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.961+0000 7f6c90fa3640 1 Processor -- start 2024-08-19T10:06:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.961+0000 7f6c90fa3640 1 -- start start 2024-08-19T10:06:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.961+0000 7f6c90fa3640 10 monclient: init 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient: _add_conns ranks=[3,2,6,5,7,8,0,4,1] 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient(hunting): picked mon.noname-g con 0x7f6c8c139580 addr v1:172.21.15.82:6790/0 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f6c8c0a4500 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient(hunting): picked mon.noname-d con 0x7f6c8c06f250 addr v1:172.21.15.1:6792/0 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c0a77d0 con 0x7f6c8c0a4500 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c0a3330 con 0x7f6c8c139580 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c0a3bb0 con 0x7f6c8c06f250 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c90fa3640 10 monclient(hunting): authenticate will time out at 1621.039451s 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c89d9b640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f6c8c0a4500 0x7f6c8c13e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36828/0 (socket says 172.21.15.1:36828) 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c8a59c640 1 -- >> v1:172.21.15.82:6790/0 conn(0x7f6c8c139580 legacy=0x7f6c8c139960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6790/0 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c8ad9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f6c8c06f250 0x7f6c8c141ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:38522/0 (socket says 172.21.15.1:38522) 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.962+0000 7f6c89d9b640 1 -- 172.21.15.1:0/3246171354 learned_addr learned my addr 172.21.15.1:0/3246171354 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8ad9d640 1 -- 172.21.15.1:0/3246171354 >> v1:172.21.15.1:6792/0 conn(0x7f6c8c06f250 legacy=0x7f6c8c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c89d9b640 1 -- 172.21.15.1:0/3246171354 >> v1:172.21.15.1:6790/0 conn(0x7f6c8c0a4500 legacy=0x7f6c8c13e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1077044170 0 0) 0x7f6c8c0a3bb0 con 0x7f6c8c06f250 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): my global_id is 204288 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c74003520 con 0x7f6c8c06f250 2024-08-19T10:06:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306246013 0 0) 0x7f6c8c0a77d0 con 0x7f6c8c0a4500 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 10 monclient(hunting): my global_id is 184691 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.964+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c8c0a3bb0 con 0x7f6c8c0a4500 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3984081667 0 0) 0x7f6c74003520 con 0x7f6c8c06f250 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 >> v1:172.21.15.82:6790/0 conn(0x7f6c8c139580 legacy=0x7f6c8c139960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 >> v1:172.21.15.1:6790/0 conn(0x7f6c8c0a4500 legacy=0x7f6c8c13e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 monclient: found mon.noname-d 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:06:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c0a95b0 con 0x7f6c8c06f250 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c84001e60 con 0x7f6c8c06f250 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 5 monclient: authenticate success, global_id 204288 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: dump: 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c84004b60 con 0x7f6c8c06f250 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 1 -- 172.21.15.1:0/3246171354 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c84005110 con 0x7f6c8c06f250 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c8959a640 10 monclient: dump: 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 10 monclient: get_monmap_and_config success 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 4 set_mon_vals no callback set 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 10 monclient: shutdown 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 1 -- 172.21.15.1:0/3246171354 >> v1:172.21.15.1:6792/0 conn(0x7f6c8c06f250 legacy=0x7f6c8c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 1 -- 172.21.15.1:0/3246171354 shutdown_connections 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 1 -- 172.21.15.1:0/3246171354 >> 172.21.15.1:0/3246171354 conn(0x7f6c8c0ac5d0 msgr2=0x7f6c8c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 1 -- 172.21.15.1:0/3246171354 shutdown_connections 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.965+0000 7f6c90fa3640 1 -- 172.21.15.1:0/3246171354 wait complete. 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.966+0000 7f6c90fa3640 10 monclient: build_initial_monmap 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.966+0000 7f6c90fa3640 10 monclient: monmap: 2024-08-19T10:06:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:06:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:06:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:06:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:06:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.966+0000 7f6c90fa3640 1 librados: starting msgr at 2024-08-19T10:06:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.966+0000 7f6c90fa3640 1 librados: starting objecter 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.966+0000 7f6c90fa3640 1 Processor -- start 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 1 -- start start 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 1 librados: setting wanted keys 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 1 librados: calling monclient init 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient: init 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient: _reopen_session rank -1 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient: _add_conns ranks=[5,4,2,6,1,7,8,3,0] 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient(hunting): picked mon.noname-h con 0x7f6c8c06f250 addr v1:172.21.15.82:6791/0 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f6c8c139580 addr v1:172.21.15.1:6791/0 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f6c8c137dd0 addr v1:172.21.15.1:6790/0 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c13a7e0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c0a6fe0 con 0x7f6c8c139580 2024-08-19T10:06:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c136b60 con 0x7f6c8c06f250 2024-08-19T10:06:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient(hunting): _renew_subs 2024-08-19T10:06:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c90fa3640 10 monclient(hunting): authenticate will time out at 1621.044458s 2024-08-19T10:06:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c8ad9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f6c8c137dd0 0x7f6c8c1d65a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36832/0 (socket says 172.21.15.1:36832) 2024-08-19T10:06:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c89d9b640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f6c8c139580 0x7f6c8c136450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:57412/0 (socket says 172.21.15.1:57412) 2024-08-19T10:06:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c8ad9d640 1 -- 172.21.15.1:0/1658272694 learned_addr learned my addr 172.21.15.1:0/1658272694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:06:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.967+0000 7f6c8a59c640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6791/0 conn(0x7f6c8c06f250 legacy=0x7f6c8c135d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6791/0 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c8ad9d640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.1:6790/0 conn(0x7f6c8c137dd0 legacy=0x7f6c8c1d65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 client.?.objecter ms_handle_connect 0x7f6c8c137dd0 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c89d9b640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.1:6791/0 conn(0x7f6c8c139580 legacy=0x7f6c8c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 client.?.objecter ms_handle_connect 0x7f6c8c139580 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459552874 0 0) 0x7f6c8c13a7e0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): my global_id is 184700 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c6c004ee0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 162541551 0 0) 0x7f6c8c0a6fe0 con 0x7f6c8c139580 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 10 monclient(hunting): my global_id is 194521 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.969+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c8c13a7e0 con 0x7f6c8c139580 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1535517439 0 0) 0x7f6c6c004ee0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c8c0a6fe0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c84001de0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): dump: 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1688228927 0 0) 0x7f6c8c13a7e0 con 0x7f6c8c139580 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c6c004ee0 con 0x7f6c8c139580 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c800047e0 con 0x7f6c8c139580 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient(hunting): dump: 2024-08-19T10:06:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2394445110 0 0) 0x7f6c8c0a6fe0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6791/0 conn(0x7f6c8c06f250 legacy=0x7f6c8c135d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.1:6791/0 conn(0x7f6c8c139580 legacy=0x7f6c8c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: _finish_hunting 0 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 monclient: found mon.b 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c1391b0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: _finish_auth 0 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: _check_auth_tickets 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:06:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 5 monclient: authenticate success, global_id 184700 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 10 monclient: _renew_subs 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c1422d0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 10 client.184700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 10 monclient: _renew_subs 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c8c1d7cc0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c90fa3640 1 librados: init done 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c84005070 con 0x7f6c8c137dd0 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c84005940 con 0x7f6c8c137dd0 2024-08-19T10:06:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c88d99640 4 set_mon_vals no callback set 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.970+0000 7f6c7affd640 10 monclient: dump: 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c90fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6c700065c0 con 0x7f6c70002860 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6c84011da0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(577..577 src has 299..577) ==== 5931+0+0 (unknown 601312446 0 0) 0x7f6c8404e6d0 con 0x7f6c8c137dd0 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c7affd640 10 client.184700.objecter ms_dispatch 0x7f6c8c000b90 osd_map(577..577 src has 299..577) 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c7affd640 3 client.184700.objecter handle_osd_map got epochs [577,577] > 0 2024-08-19T10:06:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c7affd640 3 client.184700.objecter handle_osd_map decoding full epoch 577 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c7affd640 20 client.184700.objecter dump_active .. 0 homeless 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.971+0000 7f6c89d9b640 10 monclient: get_auth_request con 0x7f6c6c041470 auth_method 0 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c8a59c640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6792/0 conn(0x7f6c70002860 legacy=0x7f6c70004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 10 client.184700.objecter ms_handle_connect 0x7f6c70002860 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 10 client.184700.objecter resend_mon_ops 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997016597 0 0) 0x7f6c700065c0 con 0x7f6c70002860 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 10 monclient(hunting): my global_id is 204334 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.973+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c8c0a6fe0 con 0x7f6c70002860 2024-08-19T10:06:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 10 client.184700.objecter ms_handle_connect 0x7f6c6c041470 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 328036509 0 0) 0x7f6c8c0a6fe0 con 0x7f6c70002860 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c700065c0 con 0x7f6c70002860 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c7c004490 con 0x7f6c70002860 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 10 monclient: dump: 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3618121808 0 0) 0x7f6c700065c0 con 0x7f6c70002860 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c7000a160 con 0x7f6c70002860 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6c7c006840 con 0x7f6c70002860 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6792/0 conn(0x7f6c70002860 legacy=0x7f6c70004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.974+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6792/0 conn(0x7f6c70002860 legacy=0x7f6c70004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.980+0000 7f6c90fa3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.980+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6c70009e70 con 0x7f6c70009130 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c8a59c640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6792/0 conn(0x7f6c70009130 legacy=0x7f6c700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 client.184700.objecter ms_handle_connect 0x7f6c70009130 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 client.184700.objecter resend_mon_ops 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616058929 0 0) 0x7f6c70009e70 con 0x7f6c70009130 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient(hunting): _negotiate 2024-08-19T10:06:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient(hunting): my global_id is 204343 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c700065c0 con 0x7f6c70009130 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 28769836 0 0) 0x7f6c700065c0 con 0x7f6c70009130 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c70009e70 con 0x7f6c70009130 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6c7c004f80 con 0x7f6c70009130 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.983+0000 7f6c7affd640 10 monclient: dump: 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:06:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3892003412 0 0) 0x7f6c70009e70 con 0x7f6c70009130 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6c7000c410 con 0x7f6c70009130 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7495 (unknown 0 0 3516710416) 0x7f6c7c006320 con 0x7f6c70009130 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6792/0 conn(0x7f6c70009130 legacy=0x7f6c700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c7affd640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6792/0 conn(0x7f6c70009130 legacy=0x7f6c700095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 134, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30856, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:06:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:06:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:06:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:06:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:06:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:06:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:06:05.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:06:05.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:06:05.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:06:05.023 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:06:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:06:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 10 librados: watch_flush enter 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 10 librados: watch_flush exit 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 20 client.184700.objecter shutdown clearing up homeless session... 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 10 client.184700.objecter successfully canceled tick 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6c6c041470 legacy=0x7f6c6c043930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 10 monclient: shutdown 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 >> v1:172.21.15.1:6790/0 conn(0x7f6c8c137dd0 legacy=0x7f6c8c1d65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c8ad9d640 1 -- 172.21.15.1:0/1658272694 reap_dead start 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c8ad9d640 1 -- 172.21.15.1:0/1658272694 reap_dead start 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 shutdown_connections 2024-08-19T10:06:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 >> 172.21.15.1:0/1658272694 conn(0x7f6c8c0ac5d0 msgr2=0x7f6c8c0aa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:06:05.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 shutdown_connections 2024-08-19T10:06:05.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.984+0000 7f6c90fa3640 1 -- 172.21.15.1:0/1658272694 wait complete. 2024-08-19T10:06:05.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:06:04.985+0000 7f6c90fa3640 1 librados: shutdown 2024-08-19T10:06:05.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-08-19T10:06:05.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T10:06:20.190 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8502 ms) 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6061 ms) 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5426 ms) 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25511 ms) 2024-08-19T10:06:20.191 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13371 ms) 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12468 ms) 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 00000010 69 73 |is| 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 00000012 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12773 ms) 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (13423 ms) 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (722958 ms total) 2024-08-19T10:06:20.192 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1012 ms) 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (25374 ms) 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17981 ms) 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (44370 ms total) 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-08-19T10:06:20.193 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7875 ms) 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7173 ms) 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: waiting for scrub... 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: done waiting 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (23958 ms) 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10003 ms) 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8050 ms) 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (15126 ms) 2024-08-19T10:06:20.194 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-08-19T10:06:20.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (25388 ms) 2024-08-19T10:06:20.195 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-08-19T10:07:35.027 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-08-19T10:07:35.028 INFO:tasks.ceph.mon.h:Restarting daemon 2024-08-19T10:07:35.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-08-19T10:07:35.033 INFO:tasks.ceph.mon.h:Started 2024-08-19T10:07:35.033 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-08-19T10:07:35.033 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T10:07:35.033 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T10:07:35.035 INFO:tasks.ceph.mon.g:Started 2024-08-19T10:07:35.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:07:35.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:07:35.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.146+0000 7f8746638640 10 monclient: get_monmap_and_config 2024-08-19T10:07:35.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.146+0000 7f8746638640 10 monclient: build_initial_monmap 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.146+0000 7f8746638640 10 monclient: monmap: 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:35.147375+0000 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:35.147375+0000 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:35.148 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:35.149 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:35.149 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:35.149 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:35.149 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:35.149 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:35.149 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:35.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 1 Processor -- start 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 1 -- start start 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient: init 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient: _add_conns ranks=[4,6,8,5,2,7,1,0,3] 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient(hunting): picked mon.noname-c con 0x7f87400a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient(hunting): picked mon.noname-d con 0x7f87400694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient(hunting): picked mon.noname-e con 0x7f87400a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87400a3c70 con 0x7f87400a6830 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87400a33d0 con 0x7f87400694b0 2024-08-19T10:07:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87400a36b0 con 0x7f87400a46a0 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f8746638640 10 monclient(hunting): authenticate will time out at 1711.223923s 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.147+0000 7f873f7fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f87400694b0 0x7f87400a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:40146/0 (socket says 172.21.15.1:40146) 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.148+0000 7f873f7fe640 1 -- 172.21.15.1:0/704928396 learned_addr learned my addr 172.21.15.1:0/704928396 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.148+0000 7f873ffff640 1 --1- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6791/0 conn(0x7f87400a6830 0x7f87400a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:56756/0 (socket says 172.21.15.1:56756) 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873f7fe640 1 -- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6792/0 conn(0x7f87400694b0 legacy=0x7f87400a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873ffff640 1 -- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6791/0 conn(0x7f87400a6830 legacy=0x7f87400a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f8744bd5640 1 -- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6793/0 conn(0x7f87400a46a0 legacy=0x7f8740141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815750713 0 0) 0x7f87400a33d0 con 0x7f87400694b0 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): my global_id is 204297 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8728003520 con 0x7f87400694b0 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315118565 0 0) 0x7f87400a3c70 con 0x7f87400a6830 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): my global_id is 194530 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87400a33d0 con 0x7f87400a6830 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308954202 0 0) 0x7f87400a36b0 con 0x7f87400a46a0 2024-08-19T10:07:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient(hunting): my global_id is 204353 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87400a3c70 con 0x7f87400a46a0 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1257669378 0 0) 0x7f8728003520 con 0x7f87400694b0 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6793/0 conn(0x7f87400a46a0 legacy=0x7f8740141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6791/0 conn(0x7f87400a6830 legacy=0x7f87400a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 monclient: found mon.noname-d 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:07:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874006dbb0 con 0x7f87400694b0 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: _finish_auth 0 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8734001ae0 con 0x7f87400694b0 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f8746638640 5 monclient: authenticate success, global_id 204297 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: dump: 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.159 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f8746638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87340048a0 con 0x7f87400694b0 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f8746638640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f8746638640 4 set_mon_vals no callback set 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 1 -- 172.21.15.1:0/704928396 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8734004dd0 con 0x7f87400694b0 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.150+0000 7f873effd640 10 monclient: dump: 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 10 monclient: shutdown 2024-08-19T10:07:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 1 -- 172.21.15.1:0/704928396 >> v1:172.21.15.1:6792/0 conn(0x7f87400694b0 legacy=0x7f87400a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 1 -- 172.21.15.1:0/704928396 shutdown_connections 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 1 -- 172.21.15.1:0/704928396 >> 172.21.15.1:0/704928396 conn(0x7f87400a9020 msgr2=0x7f8740068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 1 -- 172.21.15.1:0/704928396 shutdown_connections 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 1 -- 172.21.15.1:0/704928396 wait complete. 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.151+0000 7f8746638640 10 monclient: build_initial_monmap 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 10 monclient: monmap: 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:35.163 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 1 librados: starting msgr at 2024-08-19T10:07:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 1 librados: starting objecter 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 1 Processor -- start 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 1 -- start start 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 1 librados: setting wanted keys 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 1 librados: calling monclient init 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 10 monclient: init 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.152+0000 7f8746638640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 10 monclient: _add_conns ranks=[5,7,0,1,2,8,4,6,3] 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 10 monclient(hunting): picked mon.noname-h con 0x7f87400694b0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 10 monclient(hunting): picked mon.noname-i con 0x7f87400a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 10 monclient(hunting): picked mon.noname-a con 0x7f87400a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87401397d0 con 0x7f87400a6830 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87400aba90 con 0x7f87400694b0 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f874013b290 con 0x7f87400a46a0 2024-08-19T10:07:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:35.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8746638640 10 monclient(hunting): authenticate will time out at 1711.229115s 2024-08-19T10:07:35.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8744bd5640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f87400a6830 0x7f87401d5e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45202/0 (socket says 172.21.15.1:45202) 2024-08-19T10:07:35.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.153+0000 7f8744bd5640 1 -- 172.21.15.1:0/1062512032 learned_addr learned my addr 172.21.15.1:0/1062512032 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f8744bd5640 1 -- 172.21.15.1:0/1062512032 >> v1:172.21.15.1:6789/0 conn(0x7f87400a6830 legacy=0x7f87401d5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 client.?.objecter ms_handle_connect 0x7f87400a6830 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 client.?.objecter ms_handle_connect 0x7f87400694b0 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873ffff640 1 -- 172.21.15.1:0/1062512032 >> v1:172.21.15.82:6791/0 conn(0x7f87400694b0 legacy=0x7f87401cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4283347760 0 0) 0x7f87401397d0 con 0x7f87400a6830 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): my global_id is 204318 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f872c004eb0 con 0x7f87400a6830 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3398662986 0 0) 0x7f872c004eb0 con 0x7f87400a6830 2024-08-19T10:07:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87401397d0 con 0x7f87400a6830 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8738002070 con 0x7f87400a6830 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): dump: 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1721058067 0 0) 0x7f87401397d0 con 0x7f87400a6830 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 >> v1:172.21.15.82:6792/0 conn(0x7f87400a46a0 legacy=0x7f87401420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 >> v1:172.21.15.82:6791/0 conn(0x7f87400694b0 legacy=0x7f87401cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient: _finish_hunting 0 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 monclient: found mon.a 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874013b570 con 0x7f87400a6830 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient: _finish_auth 0 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 10 monclient: _check_auth_tickets 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f873cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f8746638640 5 monclient: authenticate success, global_id 204318 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f8746638640 10 monclient: _renew_subs 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f8746638640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.155+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f874013a5d0 con 0x7f87400a6830 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f8746638640 10 client.204318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f8746638640 10 monclient: _renew_subs 2024-08-19T10:07:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f8746638640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87401429a0 con 0x7f87400a6830 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f8746638640 1 librados: init done 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8738003e60 con 0x7f87400a6830 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8738004390 con 0x7f87400a6830 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873e7fc640 4 set_mon_vals no callback set 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 10 monclient: dump: 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.156+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8738004530 con 0x7f87400a6830 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f8746638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f8746638640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f8746638640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8720005080 con 0x7f87400a6830 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7f873804e820 con 0x7f87400a6830 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f873cff9640 10 client.204318.objecter ms_dispatch 0x7f8740000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f873cff9640 3 client.204318.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f873cff9640 3 client.204318.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f873ffff640 10 monclient: get_auth_request con 0x7f872c040ed0 auth_method 0 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.157+0000 7f873cff9640 20 client.204318.objecter dump_active .. 0 homeless 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.159+0000 7f873cff9640 10 client.204318.objecter ms_handle_connect 0x7f872c040ed0 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.161+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f8738017930 con 0x7f87400a6830 2024-08-19T10:07:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.161+0000 7f873cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:35.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.161+0000 7f873cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:35.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.420+0000 7f8746638640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:07:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.420+0000 7f8746638640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:07:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.420+0000 7f8746638640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:35.420+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8720005370 con 0x7f87400a6830 2024-08-19T10:07:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:38.153+0000 7f87277fe640 10 monclient: tick 2024-08-19T10:07:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:38.153+0000 7f87277fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:38.153+0000 7f87277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:38.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:38.153+0000 7f87277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:39.153+0000 7f87277fe640 10 monclient: tick 2024-08-19T10:07:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:39.153+0000 7f87277fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:39.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:39.153+0000 7f87277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.153+0000 7f87277fe640 10 monclient: tick 2024-08-19T10:07:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.153+0000 7f87277fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.153+0000 7f87277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.157+0000 7f873d7fa640 10 client.204318.objecter tick 2024-08-19T10:07:40.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.320+0000 7f873cff9640 1 -- 172.21.15.1:0/1062512032 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2697 (unknown 254746657 0 2193218186) 0x7f8738017230 con 0x7f87400a6830 2024-08-19T10:07:40.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.320+0000 7f873cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:07:40.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.320+0000 7f873cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:40.321 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":136,"quorum":[0,1,2,3,4,6,7,8],"quorum_names":["a","f","b","g","c","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:07:40.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.323+0000 7f8746638640 10 librados: watch_flush enter 2024-08-19T10:07:40.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 10 librados: watch_flush exit 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 20 client.204318.objecter shutdown clearing up homeless session... 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 10 client.204318.objecter successfully canceled tick 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 >> v1:172.21.15.82:6812/2549104415 conn(0x7f872c040ed0 legacy=0x7f872c043390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 10 monclient: shutdown 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 >> v1:172.21.15.1:6789/0 conn(0x7f87400a6830 legacy=0x7f87401d5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 shutdown_connections 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 >> 172.21.15.1:0/1062512032 conn(0x7f87400a9020 msgr2=0x7f87400a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 shutdown_connections 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 -- 172.21.15.1:0/1062512032 wait complete. 2024-08-19T10:07:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:40.324+0000 7f8746638640 1 librados: shutdown 2024-08-19T10:07:43.347 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:07:43.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.457+0000 7f8fab5e9640 10 monclient: get_monmap_and_config 2024-08-19T10:07:43.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.457+0000 7f8fab5e9640 10 monclient: build_initial_monmap 2024-08-19T10:07:43.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.457+0000 7f8fab5e9640 10 monclient: monmap: 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:43.458277+0000 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:43.458277+0000 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:43.460 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:43.460 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:43.460 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.457+0000 7f8fab5e9640 1 Processor -- start 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 1 -- start start 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient: init 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient: _add_conns ranks=[0,3,4,7,2,8,5,1,6] 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient(hunting): picked mon.noname-g con 0x7f8fa40a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa40a3b40 con 0x7f8fa40a67c0 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa40a8130 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa406ede0 con 0x7f8fa413f5c0 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fab5e9640 10 monclient(hunting): authenticate will time out at 1719.534746s 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fa9385640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8fa40a67c0 0x7f8fa40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54690/0 (socket says 172.21.15.1:54690) 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fa9b86640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f8fa413f5c0 0x7f8fa41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40830/0 (socket says 172.21.15.1:40830) 2024-08-19T10:07:43.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.458+0000 7f8fa9385640 1 -- 172.21.15.1:0/70679017 learned_addr learned my addr 172.21.15.1:0/70679017 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fa9385640 1 -- 172.21.15.1:0/70679017 >> v1:172.21.15.1:6789/0 conn(0x7f8fa40a67c0 legacy=0x7f8fa40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fa9b86640 1 -- 172.21.15.1:0/70679017 >> v1:172.21.15.1:6791/0 conn(0x7f8fa413f5c0 legacy=0x7f8fa41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895297812 0 0) 0x7f8fa40a3b40 con 0x7f8fa40a67c0 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): my global_id is 204327 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f90003520 con 0x7f8fa40a67c0 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fa8b84640 1 -- 172.21.15.1:0/70679017 >> v1:172.21.15.82:6790/0 conn(0x7f8fa40a6ba0 legacy=0x7f8fa40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105878461 0 0) 0x7f8fa406ede0 con 0x7f8fa413f5c0 2024-08-19T10:07:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient(hunting): my global_id is 194539 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fa40a3b40 con 0x7f8fa413f5c0 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3202604874 0 0) 0x7f8f90003520 con 0x7f8fa40a67c0 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 >> v1:172.21.15.1:6791/0 conn(0x7f8fa413f5c0 legacy=0x7f8fa41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 >> v1:172.21.15.82:6790/0 conn(0x7f8fa40a6ba0 legacy=0x7f8fa40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: _finish_hunting 0 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 monclient: found mon.noname-a 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa40a92a0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: _finish_auth 0 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: _check_auth_tickets 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fa0001ae0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fab5e9640 5 monclient: authenticate success, global_id 204327 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: dump: 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fab5e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8fa00048a0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 1 -- 172.21.15.1:0/70679017 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fa0004e10 con 0x7f8fa40a67c0 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8f9bfff640 10 monclient: dump: 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fab5e9640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.461+0000 7f8fab5e9640 4 set_mon_vals no callback set 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 10 monclient: shutdown 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 1 -- 172.21.15.1:0/70679017 >> v1:172.21.15.1:6789/0 conn(0x7f8fa40a67c0 legacy=0x7f8fa40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 1 -- 172.21.15.1:0/70679017 shutdown_connections 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 1 -- 172.21.15.1:0/70679017 >> 172.21.15.1:0/70679017 conn(0x7f8fa40ac4b0 msgr2=0x7f8fa40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 1 -- 172.21.15.1:0/70679017 shutdown_connections 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 1 -- 172.21.15.1:0/70679017 wait complete. 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 10 monclient: build_initial_monmap 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.462+0000 7f8fab5e9640 10 monclient: monmap: 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 librados: starting msgr at 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 librados: starting objecter 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 Processor -- start 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 -- start start 2024-08-19T10:07:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 librados: setting wanted keys 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 librados: calling monclient init 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient: init 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient: _add_conns ranks=[4,8,3,2,1,6,0,7,5] 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient(hunting): picked mon.noname-e con 0x7f8fa40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient(hunting): picked mon.noname-g con 0x7f8fa413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa413a640 con 0x7f8fa413f5c0 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa413a920 con 0x7f8fa40a67c0 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa40a9dc0 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fab5e9640 10 monclient(hunting): authenticate will time out at 1719.539777s 2024-08-19T10:07:43.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fa9385640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f8fa40a67c0 0x7f8fa409fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40838/0 (socket says 172.21.15.1:40838) 2024-08-19T10:07:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fa9385640 1 -- 172.21.15.1:0/4035296373 learned_addr learned my addr 172.21.15.1:0/4035296373 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.463+0000 7f8fa8b84640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f8fa40a6ba0 0x7f8fa40a0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50612/0 (socket says 172.21.15.1:50612) 2024-08-19T10:07:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fa9385640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.1:6791/0 conn(0x7f8fa40a67c0 legacy=0x7f8fa409fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fa8b84640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.1:6793/0 conn(0x7f8fa40a6ba0 legacy=0x7f8fa40a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 client.?.objecter ms_handle_connect 0x7f8fa40a67c0 2024-08-19T10:07:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 client.?.objecter ms_handle_connect 0x7f8fa40a6ba0 2024-08-19T10:07:43.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423484333 0 0) 0x7f8fa413a920 con 0x7f8fa40a67c0 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): my global_id is 194548 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fa9b86640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.82:6790/0 conn(0x7f8fa413f5c0 legacy=0x7f8fa41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f88004eb0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 client.?.objecter ms_handle_connect 0x7f8fa413f5c0 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229093436 0 0) 0x7f8fa40a9dc0 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): my global_id is 204362 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fa413a920 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3581385505 0 0) 0x7f8f88004eb0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fa40a9dc0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fa0004d90 con 0x7f8fa40a67c0 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3518695033 0 0) 0x7f8fa413a920 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f88004eb0 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8f94003610 con 0x7f8fa40a6ba0 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 288981174 0 0) 0x7f8fa40a9dc0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.1:6793/0 conn(0x7f8fa40a6ba0 legacy=0x7f8fa40a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.82:6790/0 conn(0x7f8fa413f5c0 legacy=0x7f8fa41d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 monclient: found mon.c 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa4142c80 con 0x7f8fa40a67c0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: _finish_auth 0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fab5e9640 5 monclient: authenticate success, global_id 194548 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fab5e9640 10 monclient: _renew_subs 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fab5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa413a360 con 0x7f8fa40a67c0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8fa0001a80 con 0x7f8fa40a67c0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8fa0005740 con 0x7f8fa40a67c0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f99ffb640 10 monclient: dump: 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.466+0000 7f8f9b7fe640 4 set_mon_vals no callback set 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.467+0000 7f8fab5e9640 10 client.194548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.467+0000 7f8fab5e9640 10 monclient: _renew_subs 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.467+0000 7f8fab5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.467+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f8fa40a0d40 con 0x7f8fa40a67c0 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.467+0000 7f8fab5e9640 1 librados: init done 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.467+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8fa0011ba0 con 0x7f8fa40a67c0 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7f8fa004e500 con 0x7f8fa40a67c0 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8f99ffb640 10 client.194548.objecter ms_dispatch 0x7f8fa4000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8f99ffb640 3 client.194548.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8f99ffb640 3 client.194548.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8fa8b84640 10 monclient: get_auth_request con 0x7f8f88041580 auth_method 0 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8f99ffb640 20 client.194548.objecter dump_active .. 0 homeless 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8fab5e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8fab5e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8fab5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.468+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa40aa310 con 0x7f8fa40a67c0 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.470+0000 7f8f99ffb640 10 client.194548.objecter ms_handle_connect 0x7f8f88041580 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.473+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f8fa0017730 con 0x7f8fa40a67c0 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.473+0000 7f8f99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.473+0000 7f8f99ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8fab5e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8fab5e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8fab5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fa4068f60 con 0x7f8fa40a67c0 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8f99ffb640 1 -- 172.21.15.1:0/4035296373 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 885997323) 0x7f8fa0017030 con 0x7f8fa40a67c0 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8f99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.730+0000 7f8f99ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:43.732 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":138,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 10 librados: watch_flush enter 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 10 librados: watch_flush exit 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 20 client.194548.objecter shutdown clearing up homeless session... 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 10 client.194548.objecter successfully canceled tick 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8f88041580 legacy=0x7f8f88043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 10 monclient: shutdown 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:43.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 >> v1:172.21.15.1:6791/0 conn(0x7f8fa40a67c0 legacy=0x7f8fa409fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 shutdown_connections 2024-08-19T10:07:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 >> 172.21.15.1:0/4035296373 conn(0x7f8fa40ac4b0 msgr2=0x7f8fa40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.732+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 shutdown_connections 2024-08-19T10:07:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.733+0000 7f8fab5e9640 1 -- 172.21.15.1:0/4035296373 wait complete. 2024-08-19T10:07:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.733+0000 7f8fab5e9640 1 librados: shutdown 2024-08-19T10:07:43.744 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:07:43.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:07:43.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.843+0000 7f80855c9640 10 monclient: get_monmap_and_config 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.844+0000 7f80855c9640 10 monclient: build_initial_monmap 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.844+0000 7f80855c9640 10 monclient: monmap: 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:43.844943+0000 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:43.844943+0000 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:43.846 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.844+0000 7f80855c9640 1 Processor -- start 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.844+0000 7f80855c9640 1 -- start start 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.844+0000 7f80855c9640 10 monclient: init 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient: _add_conns ranks=[5,6,0,7,2,4,8,1,3] 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient(hunting): picked mon.noname-h con 0x7f80800a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient(hunting): picked mon.noname-d con 0x7f80800694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient(hunting): picked mon.noname-a con 0x7f80800a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80800a3c70 con 0x7f80800a46a0 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80800a33d0 con 0x7f80800a6830 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80800a36b0 con 0x7f80800694b0 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f80855c9640 10 monclient(hunting): authenticate will time out at 1719.921139s 2024-08-19T10:07:43.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f807e7fc640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f80800694b0 0x7f80800a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:48124/0 (socket says 172.21.15.1:48124) 2024-08-19T10:07:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f807e7fc640 1 -- 172.21.15.1:0/2259409998 learned_addr learned my addr 172.21.15.1:0/2259409998 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.845+0000 7f807f7fe640 1 --1- 172.21.15.1:0/2259409998 >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 0x7f8080141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54692/0 (socket says 172.21.15.1:54692) 2024-08-19T10:07:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807e7fc640 1 -- 172.21.15.1:0/2259409998 >> v1:172.21.15.1:6792/0 conn(0x7f80800694b0 legacy=0x7f80800a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807f7fe640 1 -- 172.21.15.1:0/2259409998 >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 legacy=0x7f8080141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807effd640 1 -- 172.21.15.1:0/2259409998 >> v1:172.21.15.82:6791/0 conn(0x7f80800a6830 legacy=0x7f80800a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007590728 0 0) 0x7f80800a3c70 con 0x7f80800a46a0 2024-08-19T10:07:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient(hunting): my global_id is 204336 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f806c003520 con 0x7f80800a46a0 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2644666303 0 0) 0x7f806c003520 con 0x7f80800a46a0 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 >> v1:172.21.15.1:6792/0 conn(0x7f80800694b0 legacy=0x7f80800a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 >> v1:172.21.15.82:6791/0 conn(0x7f80800a6830 legacy=0x7f80800a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 monclient: found mon.noname-a 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808006dbb0 con 0x7f80800a46a0 2024-08-19T10:07:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient: _finish_auth 0 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8070001e60 con 0x7f80800a46a0 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f80855c9640 5 monclient: authenticate success, global_id 204336 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f80855c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.847+0000 7f807dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 10 monclient: dump: 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.854 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:43.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8070004b60 con 0x7f80800a46a0 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 1 -- 172.21.15.1:0/2259409998 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f80700050d0 con 0x7f80800a46a0 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 4 set_mon_vals no callback set 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f807dffb640 10 monclient: dump: 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.856 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 10 monclient: shutdown 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 1 -- 172.21.15.1:0/2259409998 >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 legacy=0x7f8080141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 1 -- 172.21.15.1:0/2259409998 shutdown_connections 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 1 -- 172.21.15.1:0/2259409998 >> 172.21.15.1:0/2259409998 conn(0x7f80800a9020 msgr2=0x7f8080068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 1 -- 172.21.15.1:0/2259409998 shutdown_connections 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.848+0000 7f80855c9640 1 -- 172.21.15.1:0/2259409998 wait complete. 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 10 monclient: build_initial_monmap 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 10 monclient: monmap: 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:43.858 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 1 librados: starting msgr at 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 1 librados: starting objecter 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 1 Processor -- start 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 1 -- start start 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 1 librados: setting wanted keys 2024-08-19T10:07:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 1 librados: calling monclient init 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.849+0000 7f80855c9640 10 monclient: init 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient: _add_conns ranks=[2,0,7,6,8,4,3,5,1] 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient(hunting): picked mon.noname-b con 0x7f80800694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient(hunting): picked mon.noname-a con 0x7f80800a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient(hunting): picked mon.noname-i con 0x7f80800a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8080123810 con 0x7f80800a46a0 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8080123af0 con 0x7f80800694b0 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80800a9f50 con 0x7f80800a6830 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f80855c9640 10 monclient(hunting): authenticate will time out at 1719.926173s 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f807e7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 0x7f80801d3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54706/0 (socket says 172.21.15.1:54706) 2024-08-19T10:07:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.850+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 learned_addr learned my addr 172.21.15.1:0/521094238 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 legacy=0x7f80801d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f807effd640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6790/0 conn(0x7f80800694b0 legacy=0x7f80801cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 client.?.objecter ms_handle_connect 0x7f80800a46a0 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 client.?.objecter ms_handle_connect 0x7f80800694b0 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 client.?.objecter ms_handle_connect 0x7f80800a6830 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f807f7fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.82:6792/0 conn(0x7f80800a6830 legacy=0x7f80801d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 390107316 0 0) 0x7f8080123810 con 0x7f80800a46a0 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): my global_id is 204345 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8060004eb0 con 0x7f80800a46a0 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f807e7fc640 1 --1- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 0x7f80801d3c90 :-1 s=OPENED pgs=810 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f8060004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1374498790 0 0) 0x7f8060004eb0 con 0x7f80800a46a0 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8080123810 con 0x7f80800a46a0 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8068001c60 con 0x7f80800a46a0 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): dump: 2024-08-19T10:07:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.864 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.852+0000 7f80777fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1556402732 0 0) 0x7f8080123af0 con 0x7f80800694b0 2024-08-19T10:07:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): my global_id is 184709 2024-08-19T10:07:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8060004eb0 con 0x7f80800694b0 2024-08-19T10:07:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415306773 0 0) 0x7f80800a9f50 con 0x7f80800a6830 2024-08-19T10:07:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 10 monclient(hunting): my global_id is 204352 2024-08-19T10:07:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.943+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8080123af0 con 0x7f80800a6830 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1566500895 0 0) 0x7f8060004eb0 con 0x7f80800694b0 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80800a9f50 con 0x7f80800694b0 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8078001cd0 con 0x7f80800694b0 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient(hunting): dump: 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.959 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.960 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 385927400 0 0) 0x7f80800a9f50 con 0x7f80800694b0 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.82:6792/0 conn(0x7f80800a6830 legacy=0x7f80801d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f80800a46a0 legacy=0x7f80801d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient: _finish_hunting 0 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 monclient: found mon.b 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8080142d40 con 0x7f80800694b0 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient: _finish_auth 0 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2755845207 0 0) 0x7f8080123810 con 0x7f80800a46a0 2024-08-19T10:07:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.955+0000 7f80855c9640 5 monclient: authenticate success, global_id 184709 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 10 monclient: _renew_subs 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8080123530 con 0x7f80800694b0 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 10 client.184709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 10 monclient: _renew_subs 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f80800a15e0 con 0x7f80800694b0 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 1 librados: init done 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f80780038c0 con 0x7f80800694b0 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8078003db0 con 0x7f80800694b0 2024-08-19T10:07:43.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f807d7fa640 4 set_mon_vals no callback set 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80777fe640 10 monclient: dump: 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.956+0000 7f80855c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f80640065c0 con 0x7f8064002860 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8078003f50 con 0x7f80800694b0 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7f807804e270 con 0x7f80800694b0 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80777fe640 10 client.184709.objecter ms_dispatch 0x7f8080000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80777fe640 3 client.184709.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80777fe640 3 client.184709.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:43.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f80777fe640 20 client.184709.objecter dump_active .. 0 homeless 2024-08-19T10:07:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.957+0000 7f807f7fe640 10 monclient: get_auth_request con 0x7f8060041620 auth_method 0 2024-08-19T10:07:43.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 client.184709.objecter ms_handle_connect 0x7f8064002860 2024-08-19T10:07:43.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 legacy=0x7f8064004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 client.184709.objecter resend_mon_ops 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612762792 0 0) 0x7f80640065c0 con 0x7f8064002860 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient(hunting): my global_id is 204354 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8080123810 con 0x7f8064002860 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 134173637 0 0) 0x7f8080123810 con 0x7f8064002860 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80640065c0 con 0x7f8064002860 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8068005240 con 0x7f8064002860 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 monclient: dump: 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2521294038 0 0) 0x7f80640065c0 con 0x7f8064002860 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f806400a160 con 0x7f8064002860 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 legacy=0x7f8064004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 legacy=0x7f8064004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f807e7fc640 1 --1- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 0x7f8064004d00 :-1 s=OPENED pgs=813 cs=1 l=1).handle_message read tag failed 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f807e7fc640 1 --1- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 0x7f8064004d00 :-1 s=OPENED pgs=813 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 legacy=0x7f8064004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f8064002860 legacy=0x7f8064004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:43.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f80640065c0 con 0x7f806004c9f0 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.959+0000 7f80777fe640 10 client.184709.objecter ms_handle_connect 0x7f8060041620 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 client.184709.objecter ms_handle_connect 0x7f806004c9f0 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f806004c9f0 legacy=0x7f806004b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 client.184709.objecter resend_mon_ops 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2169551858 0 0) 0x7f80640065c0 con 0x7f806004c9f0 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient(hunting): my global_id is 204363 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8080123810 con 0x7f806004c9f0 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1120357019 0 0) 0x7f8080123810 con 0x7f806004c9f0 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80640065c0 con 0x7f806004c9f0 2024-08-19T10:07:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8068005d80 con 0x7f806004c9f0 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient: dump: 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2883515354 0 0) 0x7f80640065c0 con 0x7f806004c9f0 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f806004c450 con 0x7f806004c9f0 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8068008480 con 0x7f806004c9f0 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f806004c9f0 legacy=0x7f806004b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.962+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f806004c9f0 legacy=0x7f806004b090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.968+0000 7f80855c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.968+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8064009e00 con 0x7f80640090e0 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f807e7fc640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f80640090e0 legacy=0x7f8064009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 client.184709.objecter ms_handle_connect 0x7f80640090e0 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 client.184709.objecter resend_mon_ops 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599943241 0 0) 0x7f8064009e00 con 0x7f80640090e0 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient(hunting): my global_id is 204372 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80640065c0 con 0x7f80640090e0 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2506618456 0 0) 0x7f80640065c0 con 0x7f80640090e0 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8064009e00 con 0x7f80640090e0 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8068006570 con 0x7f80640090e0 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient: dump: 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1872956057 0 0) 0x7f8064009e00 con 0x7f80640090e0 2024-08-19T10:07:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f806400c3a0 con 0x7f80640090e0 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7687 (unknown 0 0 1595977253) 0x7f8068008630 con 0x7f80640090e0 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f80640090e0 legacy=0x7f8064009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80777fe640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6789/0 conn(0x7f80640090e0 legacy=0x7f8064009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f807f7fe640 1 -- 172.21.15.1:0/521094238 reap_dead start 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:43.978 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 129843, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:43.994 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.996 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:43.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:43.998 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:43.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:43.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:43.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:43.999 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.971+0000 7f80855c9640 10 librados: watch_flush enter 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 10 librados: watch_flush exit 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 20 client.184709.objecter shutdown clearing up homeless session... 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 10 client.184709.objecter successfully canceled tick 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8060041620 legacy=0x7f8060043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 10 monclient: shutdown 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 >> v1:172.21.15.1:6790/0 conn(0x7f80800694b0 legacy=0x7f80801cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 shutdown_connections 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 >> 172.21.15.1:0/521094238 conn(0x7f80800a9020 msgr2=0x7f80800a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 shutdown_connections 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 -- 172.21.15.1:0/521094238 wait complete. 2024-08-19T10:07:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:43.972+0000 7f80855c9640 1 librados: shutdown 2024-08-19T10:07:44.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:44.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:07:44.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.100+0000 7f37da333640 10 monclient: get_monmap_and_config 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.100+0000 7f37da333640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.100+0000 7f37da333640 10 monclient: monmap: 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:44.101838+0000 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:44.101838+0000 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.102 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.103 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.101+0000 7f37da333640 1 Processor -- start 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.101+0000 7f37da333640 1 -- start start 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.101+0000 7f37da333640 10 monclient: init 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient: _add_conns ranks=[0,1,7,8,2,4,3,6,5] 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient(hunting): picked mon.noname-a con 0x7f37d40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient(hunting): picked mon.noname-f con 0x7f37d40a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient(hunting): picked mon.noname-i con 0x7f37d413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d40a3b40 con 0x7f37d40a67c0 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d40a8130 con 0x7f37d40a6ba0 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d406ede0 con 0x7f37d413f5c0 2024-08-19T10:07:44.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37da333640 10 monclient(hunting): authenticate will time out at 1720.178209s 2024-08-19T10:07:44.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37d3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f37d40a67c0 0x7f37d40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54740/0 (socket says 172.21.15.1:54740) 2024-08-19T10:07:44.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.102+0000 7f37d3fff640 1 -- 172.21.15.1:0/3400142735 learned_addr learned my addr 172.21.15.1:0/3400142735 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d3fff640 1 -- 172.21.15.1:0/3400142735 >> v1:172.21.15.1:6789/0 conn(0x7f37d40a67c0 legacy=0x7f37d40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203184107 0 0) 0x7f37d40a3b40 con 0x7f37d40a67c0 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient(hunting): my global_id is 204381 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37bc003520 con 0x7f37d40a67c0 2024-08-19T10:07:44.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d37fe640 1 -- 172.21.15.1:0/3400142735 >> v1:172.21.15.82:6789/0 conn(0x7f37d40a6ba0 legacy=0x7f37d40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d88d0640 1 -- 172.21.15.1:0/3400142735 >> v1:172.21.15.82:6792/0 conn(0x7f37d413f5c0 legacy=0x7f37d41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2662726417 0 0) 0x7f37bc003520 con 0x7f37d40a67c0 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 >> v1:172.21.15.82:6792/0 conn(0x7f37d413f5c0 legacy=0x7f37d41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 >> v1:172.21.15.82:6789/0 conn(0x7f37d40a6ba0 legacy=0x7f37d40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 monclient: found mon.noname-a 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d40a92a0 con 0x7f37d40a67c0 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37da333640 5 monclient: authenticate success, global_id 204381 2024-08-19T10:07:44.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37da333640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c4001ae0 con 0x7f37d40a67c0 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37d2ffd640 10 monclient: dump: 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.104+0000 7f37da333640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37c40048a0 con 0x7f37d40a67c0 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37d2ffd640 1 -- 172.21.15.1:0/3400142735 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c4004dd0 con 0x7f37d40a67c0 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:44.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 4 set_mon_vals no callback set 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37d2ffd640 10 monclient: dump: 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 10 monclient: shutdown 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 1 -- 172.21.15.1:0/3400142735 >> v1:172.21.15.1:6789/0 conn(0x7f37d40a67c0 legacy=0x7f37d40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 1 -- 172.21.15.1:0/3400142735 shutdown_connections 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 1 -- 172.21.15.1:0/3400142735 >> 172.21.15.1:0/3400142735 conn(0x7f37d40ac4b0 msgr2=0x7f37d40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 1 -- 172.21.15.1:0/3400142735 shutdown_connections 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.105+0000 7f37da333640 1 -- 172.21.15.1:0/3400142735 wait complete. 2024-08-19T10:07:44.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 10 monclient: monmap: 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 1 librados: starting msgr at 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 1 librados: starting objecter 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 1 Processor -- start 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 1 -- start start 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 1 librados: setting wanted keys 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 1 librados: calling monclient init 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.106+0000 7f37da333640 10 monclient: init 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient: _add_conns ranks=[5,1,3,0,2,8,4,6,7] 2024-08-19T10:07:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient(hunting): picked mon.noname-h con 0x7f37d40a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient(hunting): picked mon.noname-f con 0x7f37d40a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient(hunting): picked mon.noname-g con 0x7f37d413f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d413bef0 con 0x7f37d40a6ba0 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d41cf070 con 0x7f37d413f5c0 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d40a9cc0 con 0x7f37d40a67c0 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37da333640 10 monclient(hunting): authenticate will time out at 1720.183185s 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37d3fff640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f37d40a67c0 0x7f37d41ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:42138/0 (socket says 172.21.15.1:42138) 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37d3fff640 1 -- 172.21.15.1:0/2743551744 learned_addr learned my addr 172.21.15.1:0/2743551744 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.107+0000 7f37d37fe640 1 --1- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6789/0 conn(0x7f37d40a6ba0 0x7f37d41ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.1:46748/0 (socket says 172.21.15.1:46748) 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d3fff640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6791/0 conn(0x7f37d40a67c0 legacy=0x7f37d41ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d37fe640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6789/0 conn(0x7f37d40a6ba0 legacy=0x7f37d41ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d88d0640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6790/0 conn(0x7f37d413f5c0 legacy=0x7f37d41d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 client.?.objecter ms_handle_connect 0x7f37d40a67c0 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 client.?.objecter ms_handle_connect 0x7f37d40a6ba0 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 client.?.objecter ms_handle_connect 0x7f37d413f5c0 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324798870 0 0) 0x7f37d40a9cc0 con 0x7f37d40a67c0 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): my global_id is 214106 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c0004eb0 con 0x7f37d40a67c0 2024-08-19T10:07:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372089532 0 0) 0x7f37d413bef0 con 0x7f37d40a6ba0 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): my global_id is 175087 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d40a9cc0 con 0x7f37d40a6ba0 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126052228 0 0) 0x7f37d41cf070 con 0x7f37d413f5c0 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 10 monclient(hunting): my global_id is 214104 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.109+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d413bef0 con 0x7f37d413f5c0 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3970771867 0 0) 0x7f37c0004eb0 con 0x7f37d40a67c0 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37d41cf070 con 0x7f37d40a67c0 2024-08-19T10:07:44.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c4004d90 con 0x7f37d40a67c0 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): dump: 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.130 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.131 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.131 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.131 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3614331633 0 0) 0x7f37d40a9cc0 con 0x7f37d40a6ba0 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c0004eb0 con 0x7f37d40a6ba0 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c8003610 con 0x7f37d40a6ba0 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): dump: 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3722771776 0 0) 0x7f37d413bef0 con 0x7f37d413f5c0 2024-08-19T10:07:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37d40a9cc0 con 0x7f37d413f5c0 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37cc001f40 con 0x7f37d413f5c0 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient(hunting): dump: 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2297230956 0 0) 0x7f37d41cf070 con 0x7f37d40a67c0 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6790/0 conn(0x7f37d413f5c0 legacy=0x7f37d41d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6789/0 conn(0x7f37d40a6ba0 legacy=0x7f37d41ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 monclient: found mon.h 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d413bc10 con 0x7f37d40a67c0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 5 monclient: authenticate success, global_id 214106 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 10 monclient: _renew_subs 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37d413aac0 con 0x7f37d40a67c0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 10 client.214106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 10 monclient: _renew_subs 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f37d40a27e0 con 0x7f37d40a67c0 2024-08-19T10:07:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37da333640 1 librados: init done 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37c4001d30 con 0x7f37d40a67c0 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c40048a0 con 0x7f37d40a67c0 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d27fc640 4 set_mon_vals no callback set 2024-08-19T10:07:44.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.110+0000 7f37d0ff9640 10 monclient: dump: 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.139 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f37c40119c0 con 0x7f37d40a67c0 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37da333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37b40065c0 con 0x7f37b4002860 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7f37c404e5e0 con 0x7f37d40a67c0 2024-08-19T10:07:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d0ff9640 10 client.214106.objecter ms_dispatch 0x7f37d4000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d0ff9640 3 client.214106.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d0ff9640 3 client.214106.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d0ff9640 20 client.214106.objecter dump_active .. 0 homeless 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.111+0000 7f37d88d0640 10 monclient: get_auth_request con 0x7f37c0041bd0 auth_method 0 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d37fe640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.1:6790/0 conn(0x7f37b4002860 legacy=0x7f37b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 10 client.214106.objecter ms_handle_connect 0x7f37b4002860 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 10 client.214106.objecter resend_mon_ops 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659721249 0 0) 0x7f37b40065c0 con 0x7f37b4002860 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 10 monclient(hunting): my global_id is 184718 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.113+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d41cf070 con 0x7f37b4002860 2024-08-19T10:07:44.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3857573919 0 0) 0x7f37d41cf070 con 0x7f37b4002860 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37b40065c0 con 0x7f37b4002860 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c8003dd0 con 0x7f37b4002860 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 10 monclient: dump: 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 10 client.214106.objecter ms_handle_connect 0x7f37c0041bd0 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2393277945 0 0) 0x7f37b40065c0 con 0x7f37b4002860 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37b400a160 con 0x7f37b4002860 2024-08-19T10:07:44.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f37c8004040 con 0x7f37b4002860 2024-08-19T10:07:44.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:44.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.1:6790/0 conn(0x7f37b4002860 legacy=0x7f37b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.114+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.1:6790/0 conn(0x7f37b4002860 legacy=0x7f37b4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.120+0000 7f37da333640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.120+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37b4009e50 con 0x7f37b4009110 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d37fe640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.1:6790/0 conn(0x7f37b4009110 legacy=0x7f37b40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 client.214106.objecter ms_handle_connect 0x7f37b4009110 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 client.214106.objecter resend_mon_ops 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2937454357 0 0) 0x7f37b4009e50 con 0x7f37b4009110 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient(hunting): my global_id is 184727 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37b40065c0 con 0x7f37b4009110 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4076811504 0 0) 0x7f37b40065c0 con 0x7f37b4009110 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37b4009e50 con 0x7f37b4009110 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f37c8005b00 con 0x7f37b4009110 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient: dump: 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.147 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3292727811 0 0) 0x7f37b4009e50 con 0x7f37b4009110 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37b400c3a0 con 0x7f37b4009110 2024-08-19T10:07:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 976904130) 0x7f37c8007e10 con 0x7f37b4009110 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.1:6790/0 conn(0x7f37b4009110 legacy=0x7f37b40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37d0ff9640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.1:6790/0 conn(0x7f37b4009110 legacy=0x7f37b40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:44.149 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 381372, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.150 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.151 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.152 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:44.153 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.154 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.156 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.157 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.158 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.160 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:44.162 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.163 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.166 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.167 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:44.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.170 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:44.171 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37da333640 10 librados: watch_flush enter 2024-08-19T10:07:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37da333640 10 librados: watch_flush exit 2024-08-19T10:07:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37da333640 20 client.214106.objecter shutdown clearing up homeless session... 2024-08-19T10:07:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.123+0000 7f37da333640 10 client.214106.objecter successfully canceled tick 2024-08-19T10:07:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6812/2549104415 conn(0x7f37c0041bd0 legacy=0x7f37c0044090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 10 monclient: shutdown 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 >> v1:172.21.15.82:6791/0 conn(0x7f37d40a67c0 legacy=0x7f37d41ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37d88d0640 1 -- 172.21.15.1:0/2743551744 reap_dead start 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37d88d0640 1 -- 172.21.15.1:0/2743551744 reap_dead start 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 shutdown_connections 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 >> 172.21.15.1:0/2743551744 conn(0x7f37d40ac4b0 msgr2=0x7f37d4141bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 shutdown_connections 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 -- 172.21.15.1:0/2743551744 wait complete. 2024-08-19T10:07:44.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.124+0000 7f37da333640 1 librados: shutdown 2024-08-19T10:07:44.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:44.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:07:44.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:07:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.296+0000 7f4c51060640 10 monclient: get_monmap_and_config 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.296+0000 7f4c51060640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.296+0000 7f4c51060640 10 monclient: monmap: 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:44.297474+0000 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:44.297474+0000 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.299 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 1 Processor -- start 2024-08-19T10:07:44.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 1 -- start start 2024-08-19T10:07:44.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 10 monclient: init 2024-08-19T10:07:44.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 10 monclient: _add_conns ranks=[0,7,3,6,4,8,5,1,2] 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 10 monclient(hunting): picked mon.noname-a con 0x7f4c4c0a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 10 monclient(hunting): picked mon.noname-i con 0x7f4c4c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.297+0000 7f4c51060640 10 monclient(hunting): picked mon.noname-g con 0x7f4c4c0a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c51060640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c0a3c70 con 0x7f4c4c0a6830 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c51060640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c0a33d0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c51060640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c0a36b0 con 0x7f4c4c0694b0 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c51060640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c51060640 10 monclient(hunting): authenticate will time out at 1720.374039s 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c4ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4c4c0a6830 0x7f4c4c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54754/0 (socket says 172.21.15.1:54754) 2024-08-19T10:07:44.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.298+0000 7f4c4ad9d640 1 -- 172.21.15.1:0/3324674021 learned_addr learned my addr 172.21.15.1:0/3324674021 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c4ad9d640 1 -- 172.21.15.1:0/3324674021 >> v1:172.21.15.1:6789/0 conn(0x7f4c4c0a6830 legacy=0x7f4c4c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c4a59c640 1 -- 172.21.15.1:0/3324674021 >> v1:172.21.15.82:6792/0 conn(0x7f4c4c0694b0 legacy=0x7f4c4c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 492133022 0 0) 0x7f4c4c0a3c70 con 0x7f4c4c0a6830 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c4b59e640 1 -- 172.21.15.1:0/3324674021 >> v1:172.21.15.82:6790/0 conn(0x7f4c4c0a46a0 legacy=0x7f4c4c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): my global_id is 204390 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c34003520 con 0x7f4c4c0a6830 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159107490 0 0) 0x7f4c4c0a33d0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 10 monclient(hunting): my global_id is 214113 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c0a3c70 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.300+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2705023765 0 0) 0x7f4c34003520 con 0x7f4c4c0a6830 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 >> v1:172.21.15.82:6792/0 conn(0x7f4c4c0694b0 legacy=0x7f4c4c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 >> v1:172.21.15.82:6790/0 conn(0x7f4c4c0a46a0 legacy=0x7f4c4c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 monclient: found mon.noname-a 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c06dbb0 con 0x7f4c4c0a6830 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c3c001ae0 con 0x7f4c4c0a6830 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 5 monclient: authenticate success, global_id 204390 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: dump: 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c3c0048a0 con 0x7f4c4c0a6830 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 1 -- 172.21.15.1:0/3324674021 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c3c004e10 con 0x7f4c4c0a6830 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 4 set_mon_vals no callback set 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c49d9b640 10 monclient: dump: 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 10 monclient: shutdown 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 1 -- 172.21.15.1:0/3324674021 >> v1:172.21.15.1:6789/0 conn(0x7f4c4c0a6830 legacy=0x7f4c4c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.301+0000 7f4c51060640 1 -- 172.21.15.1:0/3324674021 shutdown_connections 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.302+0000 7f4c51060640 1 -- 172.21.15.1:0/3324674021 >> 172.21.15.1:0/3324674021 conn(0x7f4c4c0a9020 msgr2=0x7f4c4c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.302+0000 7f4c51060640 1 -- 172.21.15.1:0/3324674021 shutdown_connections 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.302+0000 7f4c51060640 1 -- 172.21.15.1:0/3324674021 wait complete. 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.302+0000 7f4c51060640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.302+0000 7f4c51060640 10 monclient: monmap: 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.313 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 librados: starting msgr at 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 librados: starting objecter 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 Processor -- start 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 -- start start 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 librados: setting wanted keys 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 librados: calling monclient init 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient: init 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient: _add_conns ranks=[1,6,5,8,3,4,0,2,7] 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient(hunting): picked mon.noname-f con 0x7f4c4c0694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient(hunting): picked mon.noname-d con 0x7f4c4c0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:44.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient(hunting): picked mon.noname-h con 0x7f4c4c0a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:44.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c0a9f30 con 0x7f4c4c0694b0 2024-08-19T10:07:44.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c142c10 con 0x7f4c4c0a6830 2024-08-19T10:07:44.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c138b30 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.303+0000 7f4c51060640 10 monclient(hunting): authenticate will time out at 1720.379959s 2024-08-19T10:07:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.304+0000 7f4c4a59c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f4c4c0a46a0 0x7f4c4c135b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:48142/0 (socket says 172.21.15.1:48142) 2024-08-19T10:07:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.304+0000 7f4c4a59c640 1 -- 172.21.15.1:0/3311257471 learned_addr learned my addr 172.21.15.1:0/3311257471 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c4a59c640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6792/0 conn(0x7f4c4c0a46a0 legacy=0x7f4c4c135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 client.?.objecter ms_handle_connect 0x7f4c4c0a46a0 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 client.?.objecter ms_handle_connect 0x7f4c4c0a6830 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c4b59e640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.82:6791/0 conn(0x7f4c4c0a6830 legacy=0x7f4c4c1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 client.?.objecter ms_handle_connect 0x7f4c4c0694b0 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c4ad9d640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.82:6789/0 conn(0x7f4c4c0694b0 legacy=0x7f4c4c135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065584298 0 0) 0x7f4c4c138b30 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): my global_id is 204306 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c38004ee0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2567714052 0 0) 0x7f4c4c0a3c70 con 0x7f4c4c0a6830 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): my global_id is 214115 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c138b30 con 0x7f4c4c0a6830 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980707523 0 0) 0x7f4c4c06dbb0 con 0x7f4c4c0694b0 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): my global_id is 175096 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c0a3c70 con 0x7f4c4c0694b0 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1282959805 0 0) 0x7f4c38004ee0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4c06dbb0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c40003610 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): dump: 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 349345183 0 0) 0x7f4c4c138b30 con 0x7f4c4c0a6830 2024-08-19T10:07:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c38004ee0 con 0x7f4c4c0a6830 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c44003990 con 0x7f4c4c0a6830 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): dump: 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 349985650 0 0) 0x7f4c4c0a3c70 con 0x7f4c4c0694b0 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4c138b30 con 0x7f4c4c0694b0 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c3c001d30 con 0x7f4c4c0694b0 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 10 monclient(hunting): dump: 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2738726149 0 0) 0x7f4c4c06dbb0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.82:6791/0 conn(0x7f4c4c0a6830 legacy=0x7f4c4c1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.306+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.82:6789/0 conn(0x7f4c4c0694b0 legacy=0x7f4c4c135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 1 monclient: found mon.d 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c138e10 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 5 monclient: authenticate success, global_id 204306 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 10 monclient: _renew_subs 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4c138850 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 10 client.204306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 10 monclient: _renew_subs 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c4c136240 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 1 librados: init done 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c40003b90 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c400040c0 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c4959a640 4 set_mon_vals no callback set 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c337fe640 10 monclient: dump: 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.307+0000 7f4c51060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4c40004260 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c2c0065c0 con 0x7f4c2c002860 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7f4c4004e480 con 0x7f4c4c0a46a0 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c337fe640 10 client.204306.objecter ms_dispatch 0x7f4c4c000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c337fe640 3 client.204306.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c337fe640 3 client.204306.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c337fe640 20 client.204306.objecter dump_active .. 0 homeless 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.308+0000 7f4c4b59e640 10 monclient: get_auth_request con 0x7f4c38041ae0 auth_method 0 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 client.204306.objecter ms_handle_connect 0x7f4c2c002860 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c4ad9d640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6791/0 conn(0x7f4c2c002860 legacy=0x7f4c2c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 client.204306.objecter resend_mon_ops 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749459121 0 0) 0x7f4c2c0065c0 con 0x7f4c2c002860 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient(hunting): my global_id is 194557 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c06dbb0 con 0x7f4c2c002860 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1432850911 0 0) 0x7f4c4c06dbb0 con 0x7f4c2c002860 2024-08-19T10:07:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c2c0065c0 con 0x7f4c2c002860 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c3c001580 con 0x7f4c2c002860 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 monclient: dump: 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2759673757 0 0) 0x7f4c2c0065c0 con 0x7f4c2c002860 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c00a160 con 0x7f4c2c002860 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.310+0000 7f4c337fe640 10 client.204306.objecter ms_handle_connect 0x7f4c38041ae0 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.311+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4c3c0050d0 con 0x7f4c2c002860 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.311+0000 7f4c337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.311+0000 7f4c337fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.311+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6791/0 conn(0x7f4c2c002860 legacy=0x7f4c2c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.311+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6791/0 conn(0x7f4c2c002860 legacy=0x7f4c2c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.317+0000 7f4c51060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.317+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c2c009e70 con 0x7f4c2c009130 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c4ad9d640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6791/0 conn(0x7f4c2c009130 legacy=0x7f4c2c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 client.204306.objecter ms_handle_connect 0x7f4c2c009130 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 client.204306.objecter resend_mon_ops 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108071679 0 0) 0x7f4c2c009e70 con 0x7f4c2c009130 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient(hunting): my global_id is 194566 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c2c0065c0 con 0x7f4c2c009130 2024-08-19T10:07:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4293967093 0 0) 0x7f4c2c0065c0 con 0x7f4c2c009130 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c2c009e70 con 0x7f4c2c009130 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4c3c0057f0 con 0x7f4c2c009130 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 10 monclient: dump: 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.341 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.342 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3999412054 0 0) 0x7f4c2c009e70 con 0x7f4c2c009130 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.319+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c2c00c3f0 con 0x7f4c2c009130 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3144992292) 0x7f4c3c007d60 con 0x7f4c2c009130 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c337fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6791/0 conn(0x7f4c2c009130 legacy=0x7f4c2c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c337fe640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6791/0 conn(0x7f4c2c009130 legacy=0x7f4c2c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.343 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 252007, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:44.344 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.345 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:44.347 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.348 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.349 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:44.364 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 10 librados: watch_flush enter 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 10 librados: watch_flush exit 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 20 client.204306.objecter shutdown clearing up homeless session... 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 10 client.204306.objecter successfully canceled tick 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4c38041ae0 legacy=0x7f4c38043fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 10 monclient: shutdown 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 >> v1:172.21.15.1:6792/0 conn(0x7f4c4c0a46a0 legacy=0x7f4c4c135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c4b59e640 1 -- 172.21.15.1:0/3311257471 reap_dead start 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c4b59e640 1 -- 172.21.15.1:0/3311257471 reap_dead start 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 shutdown_connections 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 >> 172.21.15.1:0/3311257471 conn(0x7f4c4c0a9020 msgr2=0x7f4c4c0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 shutdown_connections 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 -- 172.21.15.1:0/3311257471 wait complete. 2024-08-19T10:07:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.320+0000 7f4c51060640 1 librados: shutdown 2024-08-19T10:07:44.367 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:44.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:07:44.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:07:44.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.466+0000 7fd3847d9640 10 monclient: get_monmap_and_config 2024-08-19T10:07:44.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.466+0000 7fd3847d9640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.466+0000 7fd3847d9640 10 monclient: monmap: 2024-08-19T10:07:44.467 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.467 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:44.467169+0000 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:44.467169+0000 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.468 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.466+0000 7fd3847d9640 1 Processor -- start 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.466+0000 7fd3847d9640 1 -- start start 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.466+0000 7fd3847d9640 10 monclient: init 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient: _add_conns ranks=[1,8,7,3,4,2,0,6,5] 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient(hunting): picked mon.noname-f con 0x7fd37c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient(hunting): picked mon.noname-e con 0x7fd37c0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient(hunting): picked mon.noname-i con 0x7fd37c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:44.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c0a3b40 con 0x7fd37c0a67c0 2024-08-19T10:07:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c0a8130 con 0x7fd37c13f5c0 2024-08-19T10:07:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c06ede0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd3847d9640 10 monclient(hunting): authenticate will time out at 1720.543509s 2024-08-19T10:07:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd381d74640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd37c0a6ba0 0x7fd37c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50618/0 (socket says 172.21.15.1:50618) 2024-08-19T10:07:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.467+0000 7fd381d74640 1 -- 172.21.15.1:0/1878922719 learned_addr learned my addr 172.21.15.1:0/1878922719 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.469+0000 7fd381d74640 1 -- 172.21.15.1:0/1878922719 >> v1:172.21.15.1:6793/0 conn(0x7fd37c0a6ba0 legacy=0x7fd37c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.469+0000 7fd382d76640 1 -- 172.21.15.1:0/1878922719 >> v1:172.21.15.82:6792/0 conn(0x7fd37c13f5c0 legacy=0x7fd37c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.469+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509093116 0 0) 0x7fd37c06ede0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.469+0000 7fd381573640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.469+0000 7fd381573640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.469+0000 7fd381573640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): my global_id is 204371 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd382575640 1 -- 172.21.15.1:0/1878922719 >> v1:172.21.15.82:6789/0 conn(0x7fd37c0a67c0 legacy=0x7fd37c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd368003520 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506864727 0 0) 0x7fd37c0a3b40 con 0x7fd37c0a67c0 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): my global_id is 175105 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c06ede0 con 0x7fd37c0a67c0 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485232773 0 0) 0x7fd37c0a8130 con 0x7fd37c13f5c0 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient(hunting): my global_id is 204370 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c0a3b40 con 0x7fd37c13f5c0 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4275938489 0 0) 0x7fd368003520 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 >> v1:172.21.15.82:6792/0 conn(0x7fd37c13f5c0 legacy=0x7fd37c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 >> v1:172.21.15.82:6789/0 conn(0x7fd37c0a67c0 legacy=0x7fd37c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 monclient: found mon.noname-e 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37c0a92a0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd36c001ad0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 5 monclient: authenticate success, global_id 204371 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: dump: 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd36c0048a0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 1 -- 172.21.15.1:0/1878922719 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd36c004e10 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 4 set_mon_vals no callback set 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd381573640 10 monclient: dump: 2024-08-19T10:07:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.480 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 10 monclient: shutdown 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 1 -- 172.21.15.1:0/1878922719 >> v1:172.21.15.1:6793/0 conn(0x7fd37c0a6ba0 legacy=0x7fd37c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 1 -- 172.21.15.1:0/1878922719 shutdown_connections 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.470+0000 7fd3847d9640 1 -- 172.21.15.1:0/1878922719 >> 172.21.15.1:0/1878922719 conn(0x7fd37c0ac4b0 msgr2=0x7fd37c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 -- 172.21.15.1:0/1878922719 shutdown_connections 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 -- 172.21.15.1:0/1878922719 wait complete. 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 10 monclient: monmap: 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:44.481 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 librados: starting msgr at 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 librados: starting objecter 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 Processor -- start 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 -- start start 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 librados: setting wanted keys 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 1 librados: calling monclient init 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.471+0000 7fd3847d9640 10 monclient: init 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient: _add_conns ranks=[4,0,3,1,2,5,7,8,6] 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient(hunting): picked mon.noname-c con 0x7fd37c0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient(hunting): picked mon.noname-a con 0x7fd37c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient(hunting): picked mon.noname-g con 0x7fd37c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c139a70 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c0a9cc0 con 0x7fd37c13f5c0 2024-08-19T10:07:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c13bdb0 con 0x7fd37c0a67c0 2024-08-19T10:07:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd3847d9640 10 monclient(hunting): authenticate will time out at 1720.548449s 2024-08-19T10:07:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd382575640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd37c0a67c0 0x7fd37c1ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40864/0 (socket says 172.21.15.1:40864) 2024-08-19T10:07:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.472+0000 7fd382575640 1 -- 172.21.15.1:0/1961747053 learned_addr learned my addr 172.21.15.1:0/1961747053 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd382575640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6791/0 conn(0x7fd37c0a67c0 legacy=0x7fd37c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd381d74640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6789/0 conn(0x7fd37c0a6ba0 legacy=0x7fd37c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 client.?.objecter ms_handle_connect 0x7fd37c0a67c0 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 client.?.objecter ms_handle_connect 0x7fd37c0a6ba0 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd382d76640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.82:6790/0 conn(0x7fd37c13f5c0 legacy=0x7fd37c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 client.?.objecter ms_handle_connect 0x7fd37c13f5c0 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 201351489 0 0) 0x7fd37c139a70 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): my global_id is 204399 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd360004eb0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779041283 0 0) 0x7fd37c13bdb0 con 0x7fd37c0a67c0 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): my global_id is 194575 2024-08-19T10:07:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c139a70 con 0x7fd37c0a67c0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2571160241 0 0) 0x7fd37c0a3b40 con 0x7fd37c13f5c0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 10 monclient(hunting): my global_id is 214122 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.474+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c13bdb0 con 0x7fd37c13f5c0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3367011708 0 0) 0x7fd360004eb0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd37c0a3b40 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd36c004d90 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3441602428 0 0) 0x7fd37c13bdb0 con 0x7fd37c13f5c0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd360004eb0 con 0x7fd37c13f5c0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2313472879 0 0) 0x7fd37c139a70 con 0x7fd37c0a67c0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd37c13bdb0 con 0x7fd37c0a67c0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd374002070 con 0x7fd37c13f5c0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd378003610 con 0x7fd37c0a67c0 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.497 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 53793839 0 0) 0x7fd37c0a3b40 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6791/0 conn(0x7fd37c0a67c0 legacy=0x7fd37c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.82:6790/0 conn(0x7fd37c13f5c0 legacy=0x7fd37c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 monclient: found mon.a 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37c1cf020 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 5 monclient: authenticate success, global_id 204399 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 10 monclient: _renew_subs 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd37c13af40 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 10 client.204399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 10 monclient: _renew_subs 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd37c13a880 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd3847d9640 1 librados: init done 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd36c0048a0 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd36c001d20 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd380d72640 4 set_mon_vals no callback set 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.475+0000 7fd372ffd640 10 monclient: dump: 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd3847d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd3640065c0 con 0x7fd364002860 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd36c011a00 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7fd36c04e390 con 0x7fd37c0a6ba0 2024-08-19T10:07:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd372ffd640 10 client.204399.objecter ms_dispatch 0x7fd37c000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd372ffd640 3 client.204399.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd372ffd640 3 client.204399.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd372ffd640 20 client.204399.objecter dump_active .. 0 homeless 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.476+0000 7fd382d76640 10 monclient: get_auth_request con 0x7fd360041930 auth_method 0 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 client.204399.objecter ms_handle_connect 0x7fd364002860 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 client.204399.objecter resend_mon_ops 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd382575640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6792/0 conn(0x7fd364002860 legacy=0x7fd364004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4142183574 0 0) 0x7fd3640065c0 con 0x7fd364002860 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient(hunting): my global_id is 204315 2024-08-19T10:07:44.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c0a3b40 con 0x7fd364002860 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 624160700 0 0) 0x7fd37c0a3b40 con 0x7fd364002860 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3640065c0 con 0x7fd364002860 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd378004000 con 0x7fd364002860 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.478+0000 7fd372ffd640 10 monclient: dump: 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.503 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3624006467 0 0) 0x7fd3640065c0 con 0x7fd364002860 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd36400a160 con 0x7fd364002860 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 10 client.204399.objecter ms_handle_connect 0x7fd360041930 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd3780042b0 con 0x7fd364002860 2024-08-19T10:07:44.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6792/0 conn(0x7fd364002860 legacy=0x7fd364004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.479+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6792/0 conn(0x7fd364002860 legacy=0x7fd364004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.485+0000 7fd3847d9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.485+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd36400c4f0 con 0x7fd364009110 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.487+0000 7fd382575640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6792/0 conn(0x7fd364009110 legacy=0x7fd364009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.487+0000 7fd372ffd640 10 client.204399.objecter ms_handle_connect 0x7fd364009110 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.487+0000 7fd372ffd640 10 client.204399.objecter resend_mon_ops 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1677790583 0 0) 0x7fd36400c4f0 con 0x7fd364009110 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient(hunting): my global_id is 204324 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3640065c0 con 0x7fd364009110 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1532331697 0 0) 0x7fd3640065c0 con 0x7fd364009110 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd36400c4f0 con 0x7fd364009110 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd378005cf0 con 0x7fd364009110 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient: dump: 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.506 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2436492907 0 0) 0x7fd36400c4f0 con 0x7fd364009110 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd364009ce0 con 0x7fd364009110 2024-08-19T10:07:44.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1735223178) 0x7fd378004460 con 0x7fd364009110 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6792/0 conn(0x7fd364009110 legacy=0x7fd364009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd372ffd640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6792/0 conn(0x7fd364009110 legacy=0x7fd364009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:44.508 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 130359, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:44.509 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.510 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.511 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.512 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.513 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.514 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.515 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.518 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:44.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:44.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:44.526 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.527 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:44.530 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.531 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.532 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:44.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.533 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.533 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:44.533 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 10 librados: watch_flush enter 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 10 librados: watch_flush exit 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 20 client.204399.objecter shutdown clearing up homeless session... 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 10 client.204399.objecter successfully canceled tick 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd360041930 legacy=0x7fd360043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 10 monclient: shutdown 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.488+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 >> v1:172.21.15.1:6789/0 conn(0x7fd37c0a6ba0 legacy=0x7fd37c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd382d76640 1 -- 172.21.15.1:0/1961747053 reap_dead start 2024-08-19T10:07:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd382d76640 1 -- 172.21.15.1:0/1961747053 reap_dead start 2024-08-19T10:07:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 shutdown_connections 2024-08-19T10:07:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 >> 172.21.15.1:0/1961747053 conn(0x7fd37c0ac4b0 msgr2=0x7fd37c1421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 shutdown_connections 2024-08-19T10:07:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd3847d9640 1 -- 172.21.15.1:0/1961747053 wait complete. 2024-08-19T10:07:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.489+0000 7fd3847d9640 1 librados: shutdown 2024-08-19T10:07:44.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:44.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:07:44.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:07:44.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.635+0000 7fbab1de2640 10 monclient: get_monmap_and_config 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.635+0000 7fbab1de2640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.635+0000 7fbab1de2640 10 monclient: monmap: 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:44.636724+0000 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:44.636724+0000 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.638 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.636+0000 7fbab1de2640 1 Processor -- start 2024-08-19T10:07:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.636+0000 7fbab1de2640 1 -- start start 2024-08-19T10:07:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.636+0000 7fbab1de2640 10 monclient: init 2024-08-19T10:07:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.636+0000 7fbab1de2640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.636+0000 7fbab1de2640 10 monclient: _add_conns ranks=[2,6,8,1,4,5,0,7,3] 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 10 monclient(hunting): picked mon.noname-b con 0x7fbaac0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 10 monclient(hunting): picked mon.noname-d con 0x7fbaac0a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 10 monclient(hunting): picked mon.noname-e con 0x7fbaac13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac0a3b40 con 0x7fbaac0a67c0 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac0a8130 con 0x7fbaac0a6ba0 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac06ede0 con 0x7fbaac13f5c0 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbab1de2640 10 monclient(hunting): authenticate will time out at 1720.713068s 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbaab7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fbaac0a67c0 0x7fbaac0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:57330/0 (socket says 172.21.15.1:57330) 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbaaaffd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fbaac0a6ba0 0x7fbaac0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:48168/0 (socket says 172.21.15.1:48168) 2024-08-19T10:07:44.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.637+0000 7fbaab7fe640 1 -- 172.21.15.1:0/3963605308 learned_addr learned my addr 172.21.15.1:0/3963605308 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaabfff640 1 -- 172.21.15.1:0/3963605308 >> v1:172.21.15.1:6793/0 conn(0x7fbaac13f5c0 legacy=0x7fbaac1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaaffd640 1 -- 172.21.15.1:0/3963605308 >> v1:172.21.15.1:6792/0 conn(0x7fbaac0a6ba0 legacy=0x7fbaac0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaab7fe640 1 -- 172.21.15.1:0/3963605308 >> v1:172.21.15.1:6790/0 conn(0x7fbaac0a67c0 legacy=0x7fbaac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609714674 0 0) 0x7fbaac0a8130 con 0x7fbaac0a6ba0 2024-08-19T10:07:44.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): my global_id is 204333 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba94003520 con 0x7fbaac0a6ba0 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2214556819 0 0) 0x7fbaac06ede0 con 0x7fbaac13f5c0 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): my global_id is 204380 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaac0a8130 con 0x7fbaac13f5c0 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128064757 0 0) 0x7fbaac0a3b40 con 0x7fbaac0a67c0 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient(hunting): my global_id is 184736 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaac06ede0 con 0x7fbaac0a67c0 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645641249 0 0) 0x7fbaac0a8130 con 0x7fbaac13f5c0 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 >> v1:172.21.15.1:6792/0 conn(0x7fbaac0a6ba0 legacy=0x7fbaac0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 >> v1:172.21.15.1:6790/0 conn(0x7fbaac0a67c0 legacy=0x7fbaac0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 monclient: found mon.noname-e 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac0a92a0 con 0x7fbaac13f5c0 2024-08-19T10:07:44.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbaa4001a20 con 0x7fbaac13f5c0 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: dump: 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.647 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbab1de2640 5 monclient: authenticate success, global_id 204380 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbab1de2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbaa4001d30 con 0x7fbaac13f5c0 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 1 -- 172.21.15.1:0/3963605308 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbaa4003950 con 0x7fbaac13f5c0 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbab1de2640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbab1de2640 4 set_mon_vals no callback set 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.639+0000 7fbaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbaaa7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbaaa7fc640 10 monclient: dump: 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 10 monclient: shutdown 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 1 -- 172.21.15.1:0/3963605308 >> v1:172.21.15.1:6793/0 conn(0x7fbaac13f5c0 legacy=0x7fbaac1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 1 -- 172.21.15.1:0/3963605308 shutdown_connections 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 1 -- 172.21.15.1:0/3963605308 >> 172.21.15.1:0/3963605308 conn(0x7fbaac0ac4b0 msgr2=0x7fbaac0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 1 -- 172.21.15.1:0/3963605308 shutdown_connections 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.640+0000 7fbab1de2640 1 -- 172.21.15.1:0/3963605308 wait complete. 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 10 monclient: monmap: 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.651 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 1 librados: starting msgr at 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 1 librados: starting objecter 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 1 Processor -- start 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 1 -- start start 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 1 librados: setting wanted keys 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 1 librados: calling monclient init 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.641+0000 7fbab1de2640 10 monclient: init 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient: _add_conns ranks=[4,3,2,6,8,1,7,5,0] 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient(hunting): picked mon.noname-c con 0x7fbaac0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient(hunting): picked mon.noname-g con 0x7fbaac0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient(hunting): picked mon.noname-b con 0x7fbaac13f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac13a800 con 0x7fbaac13f5c0 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac0a2860 con 0x7fbaac0a6ba0 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac0a2200 con 0x7fbaac0a67c0 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbab1de2640 10 monclient(hunting): authenticate will time out at 1720.718225s 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbaab7fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fbaac0a67c0 0x7fbaac09fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40878/0 (socket says 172.21.15.1:40878) 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbaabfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fbaac13f5c0 0x7fbaac0a09b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:57332/0 (socket says 172.21.15.1:57332) 2024-08-19T10:07:44.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.642+0000 7fbaab7fe640 1 -- 172.21.15.1:0/3780518985 learned_addr learned my addr 172.21.15.1:0/3780518985 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaab7fe640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6791/0 conn(0x7fbaac0a67c0 legacy=0x7fbaac09fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 client.?.objecter ms_handle_connect 0x7fbaac0a67c0 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaabfff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6790/0 conn(0x7fbaac13f5c0 legacy=0x7fbaac0a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 client.?.objecter ms_handle_connect 0x7fbaac13f5c0 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815357208 0 0) 0x7fbaac13a800 con 0x7fbaac13f5c0 2024-08-19T10:07:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): my global_id is 184745 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba98004eb0 con 0x7fbaac13f5c0 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3326260170 0 0) 0x7fbaac0a2200 con 0x7fbaac0a67c0 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): my global_id is 194584 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaac13a800 con 0x7fbaac0a67c0 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaaaffd640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6790/0 conn(0x7fbaac0a6ba0 legacy=0x7fbaac0a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 client.?.objecter ms_handle_connect 0x7fbaac0a6ba0 2024-08-19T10:07:44.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaaaffd640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6790/0 conn(0x7fbaac0a6ba0 legacy=0x7fbaac0a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaaaffd640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6790/0 conn(0x7fbaac0a6ba0 legacy=0x7fbaac0a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaaaffd640 1 --1- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6790/0 conn(0x7fbaac0a6ba0 0x7fbaac0a02a0 :-1 s=OPENED pgs=741 cs=1 l=1).handle_message read tag failed 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fbaaaffd640 1 --1- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6790/0 conn(0x7fbaac0a6ba0 0x7fbaac0a02a0 :-1 s=OPENED pgs=741 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6790/0 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 236249109 0 0) 0x7fba98004eb0 con 0x7fbaac13f5c0 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbaac0a2200 con 0x7fbaac13f5c0 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbaa4003950 con 0x7fbaac13f5c0 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): dump: 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.662 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.663 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 884443868 0 0) 0x7fbaac13a800 con 0x7fbaac0a67c0 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba98004eb0 con 0x7fbaac0a67c0 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fba9c001ca0 con 0x7fbaac0a67c0 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.644+0000 7fba93fff640 10 monclient(hunting): dump: 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1414704808 0 0) 0x7fbaac0a2200 con 0x7fbaac13f5c0 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6791/0 conn(0x7fbaac0a67c0 legacy=0x7fbaac09fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6790/0 conn(0x7fbaac0a6ba0 legacy=0x7fbaac0a02a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 monclient: found mon.b 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac139b50 con 0x7fbaac13f5c0 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 5 monclient: authenticate success, global_id 184745 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 10 monclient: _renew_subs 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbaac0a1600 con 0x7fbaac13f5c0 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 10 client.184745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 10 monclient: _renew_subs 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbaac139f60 con 0x7fbaac13f5c0 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbab1de2640 1 librados: init done 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbaa4001d30 con 0x7fbaac13f5c0 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fbaa40040e0 con 0x7fbaac13f5c0 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fbaa9ffb640 4 set_mon_vals no callback set 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 10 monclient: dump: 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.645+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fbaa4011a80 con 0x7fbaac13f5c0 2024-08-19T10:07:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fbab1de2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fba8c0065c0 con 0x7fba8c002860 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7fbaa404e280 con 0x7fbaac13f5c0 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fba93fff640 10 client.184745.objecter ms_dispatch 0x7fbaac000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fba93fff640 3 client.184745.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fba93fff640 3 client.184745.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fba93fff640 20 client.184745.objecter dump_active .. 0 homeless 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.646+0000 7fbaaaffd640 10 monclient: get_auth_request con 0x7fba980414d0 auth_method 0 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fbaab7fe640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6793/0 conn(0x7fba8c002860 legacy=0x7fba8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 client.184745.objecter ms_handle_connect 0x7fba8c002860 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 client.184745.objecter resend_mon_ops 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311137263 0 0) 0x7fba8c0065c0 con 0x7fba8c002860 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient(hunting): my global_id is 204389 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaac0a2200 con 0x7fba8c002860 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 client.184745.objecter ms_handle_connect 0x7fba980414d0 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2871888389 0 0) 0x7fbaac0a2200 con 0x7fba8c002860 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba8c0065c0 con 0x7fba8c002860 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fba9c0053d0 con 0x7fba8c002860 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 10 monclient: dump: 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 837360316 0 0) 0x7fba8c0065c0 con 0x7fba8c002860 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.648+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba8c00a160 con 0x7fba8c002860 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.649+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fba9c007670 con 0x7fba8c002860 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.649+0000 7fba93fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.649+0000 7fba93fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.649+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6793/0 conn(0x7fba8c002860 legacy=0x7fba8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.649+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6793/0 conn(0x7fba8c002860 legacy=0x7fba8c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.655+0000 7fbab1de2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.655+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fba8c009e70 con 0x7fba8c009130 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fbaab7fe640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6793/0 conn(0x7fba8c009130 legacy=0x7fba8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 client.184745.objecter ms_handle_connect 0x7fba8c009130 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 client.184745.objecter resend_mon_ops 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412846233 0 0) 0x7fba8c009e70 con 0x7fba8c009130 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient(hunting): my global_id is 204398 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba8c0065c0 con 0x7fba8c009130 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4092643510 0 0) 0x7fba8c0065c0 con 0x7fba8c009130 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba8c009e70 con 0x7fba8c009130 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fba9c006e60 con 0x7fba8c009130 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.657+0000 7fba93fff640 10 monclient: dump: 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1835052721 0 0) 0x7fba8c009e70 con 0x7fba8c009130 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fba8c00c410 con 0x7fba8c009130 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 38416466) 0x7fba9c0071f0 con 0x7fba8c009130 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6793/0 conn(0x7fba8c009130 legacy=0x7fba8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fba93fff640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6793/0 conn(0x7fba8c009130 legacy=0x7fba8c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:44.675 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 130531, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.678 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.679 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.681 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:44.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.685 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:44.696 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:44.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 10 librados: watch_flush enter 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 10 librados: watch_flush exit 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 20 client.184745.objecter shutdown clearing up homeless session... 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 10 client.184745.objecter successfully canceled tick 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.82:6812/2549104415 conn(0x7fba980414d0 legacy=0x7fba98043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 10 monclient: shutdown 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 >> v1:172.21.15.1:6790/0 conn(0x7fbaac13f5c0 legacy=0x7fbaac0a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbaabfff640 1 -- 172.21.15.1:0/3780518985 reap_dead start 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbaabfff640 1 -- 172.21.15.1:0/3780518985 reap_dead start 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 shutdown_connections 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 >> 172.21.15.1:0/3780518985 conn(0x7fbaac0ac4b0 msgr2=0x7fbaac0a4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 shutdown_connections 2024-08-19T10:07:44.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 -- 172.21.15.1:0/3780518985 wait complete. 2024-08-19T10:07:44.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.658+0000 7fbab1de2640 1 librados: shutdown 2024-08-19T10:07:44.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:44.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:07:44.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.797+0000 7f3aac667640 10 monclient: get_monmap_and_config 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.797+0000 7f3aac667640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.797+0000 7f3aac667640 10 monclient: monmap: 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:44.798827+0000 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:44.798827+0000 2024-08-19T10:07:44.799 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.800 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.798+0000 7f3aac667640 1 Processor -- start 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.798+0000 7f3aac667640 1 -- start start 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.798+0000 7f3aac667640 10 monclient: init 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient: _add_conns ranks=[3,2,1,5,8,7,6,0,4] 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient(hunting): picked mon.noname-g con 0x7f3aa40a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa40a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient(hunting): picked mon.noname-f con 0x7f3aa413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa40a3b40 con 0x7f3aa413f5c0 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa40a8130 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa406ede0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aac667640 10 monclient(hunting): authenticate will time out at 1720.875238s 2024-08-19T10:07:44.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aa9c02640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f3aa40a6ba0 0x7f3aa40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:57346/0 (socket says 172.21.15.1:57346) 2024-08-19T10:07:44.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.799+0000 7f3aa9c02640 1 -- 172.21.15.1:0/1752455159 learned_addr learned my addr 172.21.15.1:0/1752455159 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aa9c02640 1 -- 172.21.15.1:0/1752455159 >> v1:172.21.15.1:6790/0 conn(0x7f3aa40a6ba0 legacy=0x7f3aa40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aaa403640 1 -- 172.21.15.1:0/1752455159 >> v1:172.21.15.82:6790/0 conn(0x7f3aa40a67c0 legacy=0x7f3aa40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aaac04640 1 -- 172.21.15.1:0/1752455159 >> v1:172.21.15.82:6789/0 conn(0x7f3aa413f5c0 legacy=0x7f3aa41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710104923 0 0) 0x7f3aa40a8130 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aa9401640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aa9401640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aa9401640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.801+0000 7f3aa9401640 10 monclient(hunting): my global_id is 184754 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a8c003520 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1769991547 0 0) 0x7f3aa40a3b40 con 0x7f3aa413f5c0 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient(hunting): my global_id is 175114 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa40a8130 con 0x7f3aa413f5c0 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1852129048 0 0) 0x7f3a8c003520 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 >> v1:172.21.15.82:6790/0 conn(0x7f3aa40a67c0 legacy=0x7f3aa40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 >> v1:172.21.15.82:6789/0 conn(0x7f3aa413f5c0 legacy=0x7f3aa41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 monclient: found mon.noname-b 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa40a92a0 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a98001ae0 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: dump: 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aac667640 5 monclient: authenticate success, global_id 184754 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aac667640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3a980048a0 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 1 -- 172.21.15.1:0/1752455159 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a98004e10 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aac667640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aac667640 4 set_mon_vals no callback set 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.802+0000 7f3aa9401640 10 monclient: dump: 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 10 monclient: shutdown 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 1 -- 172.21.15.1:0/1752455159 >> v1:172.21.15.1:6790/0 conn(0x7f3aa40a6ba0 legacy=0x7f3aa40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 1 -- 172.21.15.1:0/1752455159 shutdown_connections 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 1 -- 172.21.15.1:0/1752455159 >> 172.21.15.1:0/1752455159 conn(0x7f3aa40ac4b0 msgr2=0x7f3aa40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 1 -- 172.21.15.1:0/1752455159 shutdown_connections 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 1 -- 172.21.15.1:0/1752455159 wait complete. 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.803+0000 7f3aac667640 10 monclient: monmap: 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.813 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 librados: starting msgr at 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 librados: starting objecter 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 Processor -- start 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 -- start start 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 librados: setting wanted keys 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 librados: calling monclient init 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient: init 2024-08-19T10:07:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient: _add_conns ranks=[6,8,4,1,0,2,3,5,7] 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient(hunting): picked mon.noname-d con 0x7f3aa40a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient(hunting): picked mon.noname-e con 0x7f3aa40a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa413ca10 con 0x7f3aa413f5c0 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa413a7d0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa4142b60 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aac667640 10 monclient(hunting): authenticate will time out at 1720.880700s 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aaa403640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f3aa40a67c0 0x7f3aa41399b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:48174/0 (socket says 172.21.15.1:48174) 2024-08-19T10:07:44.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.804+0000 7f3aaa403640 1 -- 172.21.15.1:0/3205097795 learned_addr learned my addr 172.21.15.1:0/3205097795 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aaac04640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.1:6791/0 conn(0x7f3aa413f5c0 legacy=0x7f3aa41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa9c02640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.1:6793/0 conn(0x7f3aa40a6ba0 legacy=0x7f3aa413a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aaa403640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.1:6792/0 conn(0x7f3aa40a67c0 legacy=0x7f3aa41399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 client.?.objecter ms_handle_connect 0x7f3aa413f5c0 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 client.?.objecter ms_handle_connect 0x7f3aa40a6ba0 2024-08-19T10:07:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 client.?.objecter ms_handle_connect 0x7f3aa40a67c0 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4145857434 0 0) 0x7f3aa413ca10 con 0x7f3aa413f5c0 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 194593 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a90004eb0 con 0x7f3aa413f5c0 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2620486554 0 0) 0x7f3aa4142b60 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 204407 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa413ca10 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1319813055 0 0) 0x7f3aa413a7d0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 204342 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa4142b60 con 0x7f3aa40a67c0 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 119571911 0 0) 0x7f3aa4142b60 con 0x7f3aa40a67c0 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3aa413a7d0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a94003990 con 0x7f3aa40a67c0 2024-08-19T10:07:44.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2767504723 0 0) 0x7f3aa413ca10 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3aa4142b60 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 61525529 0 0) 0x7f3a90004eb0 con 0x7f3aa413f5c0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3aa413ca10 con 0x7f3aa413f5c0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a98004d90 con 0x7f3aa40a6ba0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a9c001c20 con 0x7f3aa413f5c0 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.828 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2012977864 0 0) 0x7f3aa413a7d0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.1:6793/0 conn(0x7f3aa40a6ba0 legacy=0x7f3aa413a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.1:6791/0 conn(0x7f3aa413f5c0 legacy=0x7f3aa41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 monclient: found mon.d 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa40a2950 con 0x7f3aa40a67c0 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 5 monclient: authenticate success, global_id 204342 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 10 monclient: _renew_subs 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa413c730 con 0x7f3aa40a67c0 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 10 client.204342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 10 monclient: _renew_subs 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f3aa413b210 con 0x7f3aa40a67c0 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aac667640 1 librados: init done 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3a94003e90 con 0x7f3aa40a67c0 2024-08-19T10:07:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a940043c0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa8c00640 4 set_mon_vals no callback set 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.807+0000 7f3aa2ffd640 10 monclient: dump: 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3a94004560 con 0x7f3aa40a67c0 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aac667640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3a840065c0 con 0x7f3a84002860 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(653..653 src has 299..653) ==== 6013+0+0 (unknown 2501110211 0 0) 0x7f3a9404e5b0 con 0x7f3aa40a67c0 2024-08-19T10:07:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aa2ffd640 10 client.204342.objecter ms_dispatch 0x7f3aa4000b90 osd_map(653..653 src has 299..653) 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aa2ffd640 3 client.204342.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.808+0000 7f3aa2ffd640 3 client.204342.objecter handle_osd_map decoding full epoch 653 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.809+0000 7f3aa2ffd640 20 client.204342.objecter dump_active .. 0 homeless 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.809+0000 7f3aaac04640 10 monclient: get_auth_request con 0x7f3a90041b50 auth_method 0 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa9c02640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6789/0 conn(0x7f3a84002860 legacy=0x7f3a84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 client.204342.objecter ms_handle_connect 0x7f3a84002860 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 client.204342.objecter resend_mon_ops 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 client.204342.objecter ms_handle_connect 0x7f3a90041b50 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878904793 0 0) 0x7f3a840065c0 con 0x7f3a84002860 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 175123 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa413a7d0 con 0x7f3a84002860 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 522513093 0 0) 0x7f3aa413a7d0 con 0x7f3a84002860 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a840065c0 con 0x7f3a84002860 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a98001580 con 0x7f3a84002860 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 10 monclient: dump: 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1166721834 0 0) 0x7f3a840065c0 con 0x7f3a84002860 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.811+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a8400a160 con 0x7f3a84002860 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.812+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3a980048a0 con 0x7f3a84002860 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.812+0000 7f3aa2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.812+0000 7f3aa2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.812+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6789/0 conn(0x7f3a84002860 legacy=0x7f3a84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.812+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6789/0 conn(0x7f3a84002860 legacy=0x7f3a84004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.818+0000 7f3aac667640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.818+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3a84009e50 con 0x7f3a84009110 2024-08-19T10:07:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa9c02640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6789/0 conn(0x7f3a84009110 legacy=0x7f3a840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 client.204342.objecter ms_handle_connect 0x7f3a84009110 2024-08-19T10:07:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 client.204342.objecter resend_mon_ops 2024-08-19T10:07:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246412441 0 0) 0x7f3a84009e50 con 0x7f3a84009110 2024-08-19T10:07:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 175132 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a840065c0 con 0x7f3a84009110 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4251772868 0 0) 0x7f3a840065c0 con 0x7f3a84009110 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a84009e50 con 0x7f3a84009110 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3a98005a00 con 0x7f3a84009110 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 10 monclient: dump: 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.840 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 519874195 0 0) 0x7f3a84009e50 con 0x7f3a84009110 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.821+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a8400c3a0 con 0x7f3a84009110 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7852 (unknown 0 0 963899668) 0x7f3a98005d90 con 0x7f3a84009110 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aa2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aa2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6789/0 conn(0x7f3a84009110 legacy=0x7f3a840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aa2ffd640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6789/0 conn(0x7f3a84009110 legacy=0x7f3a840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:44.841 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 514881, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:44.842 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.843 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.844 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.845 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:44.846 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:44.859 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:44.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.861 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:07:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:44.863 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 10 librados: watch_flush enter 2024-08-19T10:07:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 10 librados: watch_flush exit 2024-08-19T10:07:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 20 client.204342.objecter shutdown clearing up homeless session... 2024-08-19T10:07:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 10 client.204342.objecter successfully canceled tick 2024-08-19T10:07:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3a90041b50 legacy=0x7f3a90044010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 10 monclient: shutdown 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 >> v1:172.21.15.1:6792/0 conn(0x7f3aa40a67c0 legacy=0x7f3aa41399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aaac04640 1 -- 172.21.15.1:0/3205097795 reap_dead start 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aaac04640 1 -- 172.21.15.1:0/3205097795 reap_dead start 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 shutdown_connections 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.822+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 >> 172.21.15.1:0/3205097795 conn(0x7f3aa40ac4b0 msgr2=0x7f3aa4141c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.823+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 shutdown_connections 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.823+0000 7f3aac667640 1 -- 172.21.15.1:0/3205097795 wait complete. 2024-08-19T10:07:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.823+0000 7f3aac667640 1 librados: shutdown 2024-08-19T10:07:44.867 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:44.867 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:07:44.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.981+0000 7f7753d12640 10 monclient: get_monmap_and_config 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.981+0000 7f7753d12640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.981+0000 7f7753d12640 10 monclient: monmap: 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:44.982859+0000 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:44.982859+0000 2024-08-19T10:07:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:44.985 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.982+0000 7f7753d12640 1 Processor -- start 2024-08-19T10:07:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.982+0000 7f7753d12640 1 -- start start 2024-08-19T10:07:44.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.982+0000 7f7753d12640 10 monclient: init 2024-08-19T10:07:44.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:44.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient: _add_conns ranks=[2,7,8,6,0,1,5,4,3] 2024-08-19T10:07:44.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient(hunting): picked mon.noname-b con 0x7f774c0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:44.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient(hunting): picked mon.noname-i con 0x7f774c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:44.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient(hunting): picked mon.noname-e con 0x7f774c0a46a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c0a3c70 con 0x7f774c0a6830 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c0a33d0 con 0x7f774c0694b0 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c0a36b0 con 0x7f774c0a46a0 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7753d12640 10 monclient(hunting): authenticate will time out at 1721.059250s 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7751aae640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f774c0a6830 0x7f774c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:57352/0 (socket says 172.21.15.1:57352) 2024-08-19T10:07:44.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.983+0000 7f7751aae640 1 -- 172.21.15.1:0/735236629 learned_addr learned my addr 172.21.15.1:0/735236629 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7751aae640 1 -- 172.21.15.1:0/735236629 >> v1:172.21.15.1:6790/0 conn(0x7f774c0a6830 legacy=0x7f774c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f77522af640 1 -- 172.21.15.1:0/735236629 >> v1:172.21.15.1:6793/0 conn(0x7f774c0a46a0 legacy=0x7f774c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240654720 0 0) 0x7f774c0a36b0 con 0x7f774c0a46a0 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f77512ad640 1 -- 172.21.15.1:0/735236629 >> v1:172.21.15.82:6792/0 conn(0x7f774c0694b0 legacy=0x7f774c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): my global_id is 204416 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7738003520 con 0x7f774c0a46a0 2024-08-19T10:07:44.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899029545 0 0) 0x7f774c0a3c70 con 0x7f774c0a6830 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 10 monclient(hunting): my global_id is 184763 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f774c0a36b0 con 0x7f774c0a6830 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.985+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 354304262 0 0) 0x7f774c0a33d0 con 0x7f774c0694b0 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient(hunting): _negotiate 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient(hunting): my global_id is 204379 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f774c0a3c70 con 0x7f774c0694b0 2024-08-19T10:07:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3939415086 0 0) 0x7f7738003520 con 0x7f774c0a46a0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 >> v1:172.21.15.82:6792/0 conn(0x7f774c0694b0 legacy=0x7f774c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 >> v1:172.21.15.1:6790/0 conn(0x7f774c0a6830 legacy=0x7f774c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: _finish_hunting 0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 monclient: found mon.noname-e 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c06dbb0 con 0x7f774c0a46a0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: _finish_auth 0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: _check_auth_tickets 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7744001de0 con 0x7f774c0a46a0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: dump: 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 5 monclient: authenticate success, global_id 204416 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7744002030 con 0x7f774c0a46a0 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 4 set_mon_vals no callback set 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 1 -- 172.21.15.1:0/735236629 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7744003d10 con 0x7f774c0a46a0 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7750aac640 10 monclient: dump: 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 10 monclient: shutdown 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 1 -- 172.21.15.1:0/735236629 >> v1:172.21.15.1:6793/0 conn(0x7f774c0a46a0 legacy=0x7f774c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 1 -- 172.21.15.1:0/735236629 shutdown_connections 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.986+0000 7f7753d12640 1 -- 172.21.15.1:0/735236629 >> 172.21.15.1:0/735236629 conn(0x7f774c0a9020 msgr2=0x7f774c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 -- 172.21.15.1:0/735236629 shutdown_connections 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 -- 172.21.15.1:0/735236629 wait complete. 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 10 monclient: build_initial_monmap 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 10 monclient: monmap: 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:44.999 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 librados: starting msgr at 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 librados: starting objecter 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 Processor -- start 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 -- start start 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 librados: setting wanted keys 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.987+0000 7f7753d12640 1 librados: calling monclient init 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient: init 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient: _add_conns ranks=[8,6,2,4,5,3,1,7,0] 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient(hunting): picked mon.noname-e con 0x7f774c0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient(hunting): picked mon.noname-d con 0x7f774c0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient(hunting): picked mon.noname-b con 0x7f774c0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c138810 con 0x7f774c0a6830 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c138af0 con 0x7f774c0a46a0 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c142600 con 0x7f774c0694b0 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7753d12640 10 monclient(hunting): authenticate will time out at 1721.064543s 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7751aae640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f774c0694b0 0x7f774c134f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50672/0 (socket says 172.21.15.1:50672) 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.988+0000 7f7751aae640 1 -- 172.21.15.1:0/3079329349 learned_addr learned my addr 172.21.15.1:0/3079329349 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.990+0000 7f7751aae640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.1:6793/0 conn(0x7f774c0694b0 legacy=0x7f774c134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.990+0000 7f77522af640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.1:6790/0 conn(0x7f774c0a6830 legacy=0x7f774c1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.990+0000 7f77427fc640 10 client.?.objecter ms_handle_connect 0x7f774c0694b0 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.990+0000 7f77427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.990+0000 7f77512ad640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.1:6792/0 conn(0x7f774c0a46a0 legacy=0x7f774c135620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.990+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 client.?.objecter ms_handle_connect 0x7f774c0a6830 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 client.?.objecter ms_handle_connect 0x7f774c0a46a0 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134545726 0 0) 0x7f774c142600 con 0x7f774c0694b0 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): my global_id is 204425 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7730004ee0 con 0x7f774c0694b0 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562421959 0 0) 0x7f774c138af0 con 0x7f774c0a46a0 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): my global_id is 204351 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f774c142600 con 0x7f774c0a46a0 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007287830 0 0) 0x7f774c138810 con 0x7f774c0a6830 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): my global_id is 184772 2024-08-19T10:07:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f774c138af0 con 0x7f774c0a6830 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2746199609 0 0) 0x7f774c142600 con 0x7f774c0a46a0 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f774c138810 con 0x7f774c0a46a0 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f773c003610 con 0x7f774c0a46a0 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): dump: 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2509977566 0 0) 0x7f774c138af0 con 0x7f774c0a6830 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f774c142600 con 0x7f774c0a6830 2024-08-19T10:07:45.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2813194354 0 0) 0x7f7730004ee0 con 0x7f774c0694b0 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f774c138af0 con 0x7f774c0694b0 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7744001d60 con 0x7f774c0a6830 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): dump: 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.014 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f77480047e0 con 0x7f774c0694b0 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient(hunting): dump: 2024-08-19T10:07:45.015 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.016 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3999580796 0 0) 0x7f774c138810 con 0x7f774c0a46a0 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.1:6793/0 conn(0x7f774c0694b0 legacy=0x7f774c134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.1:6790/0 conn(0x7f774c0a6830 legacy=0x7f774c1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 monclient: found mon.d 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c137c80 con 0x7f774c0a46a0 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 5 monclient: authenticate success, global_id 204351 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 10 monclient: _renew_subs 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f774c136790 con 0x7f774c0a46a0 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 10 client.204351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 10 monclient: _renew_subs 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f774c1372f0 con 0x7f774c0a46a0 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7753d12640 1 librados: init done 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f773c003b10 con 0x7f774c0a46a0 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f773c004040 con 0x7f774c0a46a0 2024-08-19T10:07:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f7743fff640 4 set_mon_vals no callback set 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.993+0000 7f77427fc640 10 monclient: dump: 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.994+0000 7f7753d12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.994+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f77340065c0 con 0x7f7734002860 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.994+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f773c0041e0 con 0x7f774c0a46a0 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.995+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(654..654 src has 299..654) ==== 5979+0+0 (unknown 3667047492 0 0) 0x7f773c04e2e0 con 0x7f774c0a46a0 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.995+0000 7f77427fc640 10 client.204351.objecter ms_dispatch 0x7f774c000b90 osd_map(654..654 src has 299..654) 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.995+0000 7f77427fc640 3 client.204351.objecter handle_osd_map got epochs [654,654] > 0 2024-08-19T10:07:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.995+0000 7f77427fc640 3 client.204351.objecter handle_osd_map decoding full epoch 654 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.995+0000 7f77427fc640 20 client.204351.objecter dump_active .. 0 homeless 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.995+0000 7f77522af640 10 monclient: get_auth_request con 0x7f7730041ac0 auth_method 0 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 client.204351.objecter ms_handle_connect 0x7f7734002860 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 client.204351.objecter resend_mon_ops 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f7751aae640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6790/0 conn(0x7f7734002860 legacy=0x7f7734004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378309362 0 0) 0x7f77340065c0 con 0x7f7734002860 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient(hunting): my global_id is 214140 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f774c138810 con 0x7f7734002860 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 client.204351.objecter ms_handle_connect 0x7f7730041ac0 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3044006209 0 0) 0x7f774c138810 con 0x7f7734002860 2024-08-19T10:07:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77340065c0 con 0x7f7734002860 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7748004eb0 con 0x7f7734002860 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 10 monclient: dump: 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2444571767 0 0) 0x7f77340065c0 con 0x7f7734002860 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.997+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f773400a160 con 0x7f7734002860 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.998+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f77480063f0 con 0x7f7734002860 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.998+0000 7f77427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.998+0000 7f77427fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.998+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6790/0 conn(0x7f7734002860 legacy=0x7f7734004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:44.998+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6790/0 conn(0x7f7734002860 legacy=0x7f7734004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.004+0000 7f7753d12640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.004+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7734009e50 con 0x7f7734009110 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.006+0000 7f7751aae640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6790/0 conn(0x7f7734009110 legacy=0x7f77340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.006+0000 7f77427fc640 10 client.204351.objecter ms_handle_connect 0x7f7734009110 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.006+0000 7f77427fc640 10 client.204351.objecter resend_mon_ops 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1153548272 0 0) 0x7f7734009e50 con 0x7f7734009110 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient(hunting): my global_id is 214149 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77340065c0 con 0x7f7734009110 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3921566231 0 0) 0x7f77340065c0 con 0x7f7734009110 2024-08-19T10:07:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7734009e50 con 0x7f7734009110 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7748005460 con 0x7f7734009110 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient: dump: 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.025 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1296584742 0 0) 0x7f7734009e50 con 0x7f7734009110 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f773400c3a0 con 0x7f7734009110 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 96482883) 0x7f7748005870 con 0x7f7734009110 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6790/0 conn(0x7f7734009110 legacy=0x7f77340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.007+0000 7f77427fc640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6790/0 conn(0x7f7734009110 legacy=0x7f77340095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9881, 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:45.027 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:45.028 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.029 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:45.031 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:45.033 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.034 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:45.037 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:45.041 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.044 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:45.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.047 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.047 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:45.047 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 10 librados: watch_flush enter 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 10 librados: watch_flush exit 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 20 client.204351.objecter shutdown clearing up homeless session... 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 10 client.204351.objecter successfully canceled tick 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7730041ac0 legacy=0x7f7730043f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 10 monclient: shutdown 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 >> v1:172.21.15.1:6792/0 conn(0x7f774c0a46a0 legacy=0x7f774c135620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f77522af640 1 -- 172.21.15.1:0/3079329349 reap_dead start 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f77522af640 1 -- 172.21.15.1:0/3079329349 reap_dead start 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 shutdown_connections 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 >> 172.21.15.1:0/3079329349 conn(0x7f774c0a9020 msgr2=0x7f774c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 shutdown_connections 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 -- 172.21.15.1:0/3079329349 wait complete. 2024-08-19T10:07:45.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.008+0000 7f7753d12640 1 librados: shutdown 2024-08-19T10:07:45.050 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:45.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:07:45.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.149+0000 7f2f11104640 10 monclient: get_monmap_and_config 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.149+0000 7f2f11104640 10 monclient: build_initial_monmap 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.149+0000 7f2f11104640 10 monclient: monmap: 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:45.150799+0000 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:45.150799+0000 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.151 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.152 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 1 Processor -- start 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 1 -- start start 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 10 monclient: init 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 10 monclient: _add_conns ranks=[3,6,5,4,2,0,8,7,1] 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 10 monclient(hunting): picked mon.noname-g con 0x7f2f0c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 10 monclient(hunting): picked mon.noname-d con 0x7f2f0c0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 10 monclient(hunting): picked mon.noname-h con 0x7f2f0c0a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.150+0000 7f2f11104640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c0a3c70 con 0x7f2f0c0a6830 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.151+0000 7f2f11104640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c0a33d0 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.151+0000 7f2f11104640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c0a36b0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.151+0000 7f2f11104640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.151+0000 7f2f11104640 10 monclient(hunting): authenticate will time out at 1721.227058s 2024-08-19T10:07:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.151+0000 7f2f0a59c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f2f0c0694b0 0x7f2f0c0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:48202/0 (socket says 172.21.15.1:48202) 2024-08-19T10:07:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.151+0000 7f2f0a59c640 1 -- 172.21.15.1:0/2813102817 learned_addr learned my addr 172.21.15.1:0/2813102817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f0a59c640 1 -- 172.21.15.1:0/2813102817 >> v1:172.21.15.1:6792/0 conn(0x7f2f0c0694b0 legacy=0x7f2f0c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f0ad9d640 1 -- 172.21.15.1:0/2813102817 >> v1:172.21.15.82:6790/0 conn(0x7f2f0c0a6830 legacy=0x7f2f0c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f0b59e640 1 -- 172.21.15.1:0/2813102817 >> v1:172.21.15.82:6791/0 conn(0x7f2f0c0a46a0 legacy=0x7f2f0c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1760133030 0 0) 0x7f2f0c0a36b0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): my global_id is 204360 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ef8003520 con 0x7f2f0c0694b0 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111031507 0 0) 0x7f2f0c0a3c70 con 0x7f2f0c0a6830 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): my global_id is 214158 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c0a36b0 con 0x7f2f0c0a6830 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920977249 0 0) 0x7f2f0c0a33d0 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient(hunting): my global_id is 214124 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c0a3c70 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2237127381 0 0) 0x7f2ef8003520 con 0x7f2f0c0694b0 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 >> v1:172.21.15.82:6791/0 conn(0x7f2f0c0a46a0 legacy=0x7f2f0c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f0b59e640 1 --1- 172.21.15.1:0/2813102817 >> v1:172.21.15.82:6791/0 conn(0x7f2f0c0a46a0 0x7f2f0c141850 :-1 s=OPENED pgs=3853 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f2f0c0a3c70 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 >> v1:172.21.15.82:6790/0 conn(0x7f2f0c0a6830 legacy=0x7f2f0c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 monclient: found mon.noname-d 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:07:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0c06dbb0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ef4001ae0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: dump: 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f11104640 5 monclient: authenticate success, global_id 204360 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f11104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ef40048a0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 1 -- 172.21.15.1:0/2813102817 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ef4004e10 con 0x7f2f0c0694b0 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f11104640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f11104640 4 set_mon_vals no callback set 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.153+0000 7f2f09d9b640 10 monclient: dump: 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 10 monclient: shutdown 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 1 -- 172.21.15.1:0/2813102817 >> v1:172.21.15.1:6792/0 conn(0x7f2f0c0694b0 legacy=0x7f2f0c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 1 -- 172.21.15.1:0/2813102817 shutdown_connections 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 1 -- 172.21.15.1:0/2813102817 >> 172.21.15.1:0/2813102817 conn(0x7f2f0c0a9020 msgr2=0x7f2f0c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 1 -- 172.21.15.1:0/2813102817 shutdown_connections 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.154+0000 7f2f11104640 1 -- 172.21.15.1:0/2813102817 wait complete. 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient: build_initial_monmap 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient: monmap: 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 1 librados: starting msgr at 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 1 librados: starting objecter 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 1 Processor -- start 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 1 -- start start 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 1 librados: setting wanted keys 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 1 librados: calling monclient init 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient: init 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient: _add_conns ranks=[8,1,3,7,4,2,0,5,6] 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient(hunting): picked mon.noname-e con 0x7f2f0c0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.155+0000 7f2f11104640 10 monclient(hunting): picked mon.noname-f con 0x7f2f0c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f11104640 10 monclient(hunting): picked mon.noname-g con 0x7f2f0c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f11104640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c139520 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f11104640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c13b5a0 con 0x7f2f0c0a6830 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f11104640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c13b880 con 0x7f2f0c0694b0 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f11104640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f11104640 10 monclient(hunting): authenticate will time out at 1721.232083s 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f0ad9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f2f0c0694b0 0x7f2f0c1cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50680/0 (socket says 172.21.15.1:50680) 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.156+0000 7f2f0ad9d640 1 -- 172.21.15.1:0/301130106 learned_addr learned my addr 172.21.15.1:0/301130106 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f0ad9d640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.1:6793/0 conn(0x7f2f0c0694b0 legacy=0x7f2f0c1cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 client.?.objecter ms_handle_connect 0x7f2f0c0694b0 2024-08-19T10:07:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 client.?.objecter ms_handle_connect 0x7f2f0c0a46a0 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f0a59c640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6789/0 conn(0x7f2f0c0a46a0 legacy=0x7f2f0c141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f0b59e640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6790/0 conn(0x7f2f0c0a6830 legacy=0x7f2f0c1d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 client.?.objecter ms_handle_connect 0x7f2f0c0a6830 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253409981 0 0) 0x7f2f0c13b880 con 0x7f2f0c0694b0 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): my global_id is 204434 2024-08-19T10:07:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2eec004eb0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468350685 0 0) 0x7f2f0c139520 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): my global_id is 175141 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c13b880 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2749616006 0 0) 0x7f2f0c13b5a0 con 0x7f2f0c0a6830 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): my global_id is 214167 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c139520 con 0x7f2f0c0a6830 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2485060970 0 0) 0x7f2eec004eb0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f0c13b5a0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2efc003cd0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): dump: 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1079797766 0 0) 0x7f2f0c13b880 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2eec004eb0 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2257180521 0 0) 0x7f2f0c139520 con 0x7f2f0c0a6830 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f0c13b880 con 0x7f2f0c0a6830 2024-08-19T10:07:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ef4004d90 con 0x7f2f0c0a46a0 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): dump: 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2f04003950 con 0x7f2f0c0a6830 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient(hunting): dump: 2024-08-19T10:07:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1769715633 0 0) 0x7f2f0c13b5a0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6790/0 conn(0x7f2f0c0a6830 legacy=0x7f2f0c1d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6789/0 conn(0x7f2f0c0a46a0 legacy=0x7f2f0c141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 monclient: found mon.e 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0c142a20 con 0x7f2f0c0694b0 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f11104640 5 monclient: authenticate success, global_id 204434 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f11104640 10 monclient: _renew_subs 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f11104640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.158+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0c0a9f30 con 0x7f2f0c0694b0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 10 client.204434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 10 monclient: _renew_subs 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f0c1426f0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 1 librados: init done 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2efc001e30 con 0x7f2f0c0694b0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f037fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2efc0044e0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f037fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f08d99640 4 set_mon_vals no callback set 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f037fe640 10 monclient: dump: 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.159+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ef00065c0 con 0x7f2ef0002860 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2efc011e00 con 0x7f2f0c0694b0 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(654..654 src has 299..654) ==== 5979+0+0 (unknown 3667047492 0 0) 0x7f2efc04e8c0 con 0x7f2f0c0694b0 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f037fe640 10 client.204434.objecter ms_dispatch 0x7f2f0c000b90 osd_map(654..654 src has 299..654) 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f037fe640 3 client.204434.objecter handle_osd_map got epochs [654,654] > 0 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f037fe640 3 client.204434.objecter handle_osd_map decoding full epoch 654 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f0b59e640 10 monclient: get_auth_request con 0x7f2eec0418e0 auth_method 0 2024-08-19T10:07:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.160+0000 7f2f037fe640 20 client.204434.objecter dump_active .. 0 homeless 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f0a59c640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6791/0 conn(0x7f2ef0002860 legacy=0x7f2ef0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 client.204434.objecter ms_handle_connect 0x7f2ef0002860 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 client.204434.objecter resend_mon_ops 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527685369 0 0) 0x7f2ef00065c0 con 0x7f2ef0002860 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient(hunting): my global_id is 214133 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c13b5a0 con 0x7f2ef0002860 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3415326359 0 0) 0x7f2f0c13b5a0 con 0x7f2ef0002860 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ef00065c0 con 0x7f2ef0002860 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ef4001580 con 0x7f2ef0002860 2024-08-19T10:07:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient: dump: 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1091320580 0 0) 0x7f2ef00065c0 con 0x7f2ef0002860 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ef000a160 con 0x7f2ef0002860 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 client.204434.objecter ms_handle_connect 0x7f2eec0418e0 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2ef40048a0 con 0x7f2ef0002860 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.162+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6791/0 conn(0x7f2ef0002860 legacy=0x7f2ef0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.163+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6791/0 conn(0x7f2ef0002860 legacy=0x7f2ef0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.169+0000 7f2f11104640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.169+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ef0009e70 con 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f0a59c640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6791/0 conn(0x7f2ef0009130 legacy=0x7f2ef00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 client.204434.objecter ms_handle_connect 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 client.204434.objecter resend_mon_ops 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229150385 0 0) 0x7f2ef0009e70 con 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient(hunting): my global_id is 214142 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ef00065c0 con 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2747317749 0 0) 0x7f2ef00065c0 con 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ef0009e70 con 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2ef4005ad0 con 0x7f2ef0009130 2024-08-19T10:07:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.171+0000 7f2f037fe640 10 monclient: dump: 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2599408750 0 0) 0x7f2ef0009e70 con 0x7f2ef0009130 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ef000c3f0 con 0x7f2ef0009130 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 2973012394) 0x7f2ef4006fe0 con 0x7f2ef0009130 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6791/0 conn(0x7f2ef0009130 legacy=0x7f2ef00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f037fe640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6791/0 conn(0x7f2ef0009130 legacy=0x7f2ef00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 10046, 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:45.194 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.195 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.196 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:45.197 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.199 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:45.204 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:45.205 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.206 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.210 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:45.212 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:45.213 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 10 librados: watch_flush enter 2024-08-19T10:07:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 10 librados: watch_flush exit 2024-08-19T10:07:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 20 client.204434.objecter shutdown clearing up homeless session... 2024-08-19T10:07:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 10 client.204434.objecter successfully canceled tick 2024-08-19T10:07:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2eec0418e0 legacy=0x7f2eec043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 10 monclient: shutdown 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 >> v1:172.21.15.1:6793/0 conn(0x7f2f0c0694b0 legacy=0x7f2f0c1cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f0b59e640 1 -- 172.21.15.1:0/301130106 reap_dead start 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f0b59e640 1 -- 172.21.15.1:0/301130106 reap_dead start 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 shutdown_connections 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 >> 172.21.15.1:0/301130106 conn(0x7f2f0c0a9020 msgr2=0x7f2f0c0a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 shutdown_connections 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 -- 172.21.15.1:0/301130106 wait complete. 2024-08-19T10:07:45.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.172+0000 7f2f11104640 1 librados: shutdown 2024-08-19T10:07:45.217 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:45.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:07:45.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:07:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.315+0000 7f8872c20640 10 monclient: get_monmap_and_config 2024-08-19T10:07:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.315+0000 7f8872c20640 10 monclient: build_initial_monmap 2024-08-19T10:07:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.315+0000 7f8872c20640 10 monclient: monmap: 2024-08-19T10:07:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:45.316724+0000 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:45.316724+0000 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.318 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.316+0000 7f8872c20640 1 Processor -- start 2024-08-19T10:07:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.316+0000 7f8872c20640 1 -- start start 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.316+0000 7f8872c20640 10 monclient: init 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient: _add_conns ranks=[3,8,4,5,2,7,1,6,0] 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient(hunting): picked mon.noname-g con 0x7f886c0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient(hunting): picked mon.noname-e con 0x7f886c0a6ba0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient(hunting): picked mon.noname-c con 0x7f886c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c0a3b40 con 0x7f886c0a67c0 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c0a8130 con 0x7f886c13f5c0 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c06ede0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f8872c20640 10 monclient(hunting): authenticate will time out at 1721.393130s 2024-08-19T10:07:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f886bfff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f886c0a6ba0 0x7f886c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50694/0 (socket says 172.21.15.1:50694) 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f88711bd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f886c13f5c0 0x7f886c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:40902/0 (socket says 172.21.15.1:40902) 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.317+0000 7f886bfff640 1 -- 172.21.15.1:0/2322268749 learned_addr learned my addr 172.21.15.1:0/2322268749 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f88711bd640 1 -- 172.21.15.1:0/2322268749 >> v1:172.21.15.1:6791/0 conn(0x7f886c13f5c0 legacy=0x7f886c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886bfff640 1 -- 172.21.15.1:0/2322268749 >> v1:172.21.15.1:6793/0 conn(0x7f886c0a6ba0 legacy=0x7f886c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3227779018 0 0) 0x7f886c06ede0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): my global_id is 204443 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f88709bc640 1 -- 172.21.15.1:0/2322268749 >> v1:172.21.15.82:6790/0 conn(0x7f886c0a67c0 legacy=0x7f886c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8858003520 con 0x7f886c0a6ba0 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027834161 0 0) 0x7f886c0a8130 con 0x7f886c13f5c0 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): my global_id is 194602 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c06ede0 con 0x7f886c13f5c0 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382878921 0 0) 0x7f886c0a3b40 con 0x7f886c0a67c0 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient(hunting): my global_id is 214176 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c0a8130 con 0x7f886c0a67c0 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1209552609 0 0) 0x7f8858003520 con 0x7f886c0a6ba0 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 >> v1:172.21.15.1:6791/0 conn(0x7f886c13f5c0 legacy=0x7f886c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 >> v1:172.21.15.82:6790/0 conn(0x7f886c0a67c0 legacy=0x7f886c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 monclient: found mon.noname-e 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c0a92a0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8854001a00 con 0x7f886c0a6ba0 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f8872c20640 5 monclient: authenticate success, global_id 204443 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f8872c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f886b7fe640 10 monclient: dump: 2024-08-19T10:07:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.319+0000 7f8872c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8854001cd0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f886b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f886b7fe640 1 -- 172.21.15.1:0/2322268749 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8854004be0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f886b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f886b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f886b7fe640 10 monclient: dump: 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 4 set_mon_vals no callback set 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 10 monclient: shutdown 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 1 -- 172.21.15.1:0/2322268749 >> v1:172.21.15.1:6793/0 conn(0x7f886c0a6ba0 legacy=0x7f886c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 1 -- 172.21.15.1:0/2322268749 shutdown_connections 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 1 -- 172.21.15.1:0/2322268749 >> 172.21.15.1:0/2322268749 conn(0x7f886c0ac4b0 msgr2=0x7f886c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 1 -- 172.21.15.1:0/2322268749 shutdown_connections 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.320+0000 7f8872c20640 1 -- 172.21.15.1:0/2322268749 wait complete. 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 10 monclient: build_initial_monmap 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 10 monclient: monmap: 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 1 librados: starting msgr at 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 1 librados: starting objecter 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 1 Processor -- start 2024-08-19T10:07:45.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 1 -- start start 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 1 librados: setting wanted keys 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 1 librados: calling monclient init 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.321+0000 7f8872c20640 10 monclient: init 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient: _add_conns ranks=[8,5,0,4,6,2,1,7,3] 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient(hunting): picked mon.noname-e con 0x7f886c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient(hunting): picked mon.noname-h con 0x7f886c0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient(hunting): picked mon.noname-a con 0x7f886c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c13ba10 con 0x7f886c13f5c0 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c13bcf0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c13ae80 con 0x7f886c0a67c0 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f8872c20640 10 monclient(hunting): authenticate will time out at 1721.398192s 2024-08-19T10:07:45.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f88709bc640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f886c0a67c0 0x7f886c13c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50708/0 (socket says 172.21.15.1:50708) 2024-08-19T10:07:45.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.322+0000 7f88709bc640 1 -- 172.21.15.1:0/2147361685 learned_addr learned my addr 172.21.15.1:0/2147361685 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88709bc640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.1:6793/0 conn(0x7f886c0a67c0 legacy=0x7f886c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88711bd640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.1:6789/0 conn(0x7f886c13f5c0 legacy=0x7f886c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 client.?.objecter ms_handle_connect 0x7f886c0a67c0 2024-08-19T10:07:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 client.?.objecter ms_handle_connect 0x7f886c13f5c0 2024-08-19T10:07:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112596999 0 0) 0x7f886c13ae80 con 0x7f886c0a67c0 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f886bfff640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6791/0 conn(0x7f886c0a6ba0 legacy=0x7f886c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): my global_id is 204452 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f884c004ee0 con 0x7f886c0a67c0 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 client.?.objecter ms_handle_connect 0x7f886c0a6ba0 2024-08-19T10:07:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418755419 0 0) 0x7f886c13ba10 con 0x7f886c13f5c0 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): my global_id is 204408 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c13ae80 con 0x7f886c13f5c0 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603234574 0 0) 0x7f886c13bcf0 con 0x7f886c0a6ba0 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): my global_id is 214151 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c13ba10 con 0x7f886c0a6ba0 2024-08-19T10:07:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4073646913 0 0) 0x7f884c004ee0 con 0x7f886c0a67c0 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f886c13bcf0 con 0x7f886c0a67c0 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8864003610 con 0x7f886c0a67c0 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): dump: 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3062232122 0 0) 0x7f886c13ae80 con 0x7f886c13f5c0 2024-08-19T10:07:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f884c004ee0 con 0x7f886c13f5c0 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f885c002020 con 0x7f886c13f5c0 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 10 monclient(hunting): dump: 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3071804037 0 0) 0x7f886c13ba10 con 0x7f886c0a6ba0 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.324+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f886c13ae80 con 0x7f886c0a6ba0 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8854004b20 con 0x7f886c0a6ba0 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient(hunting): dump: 2024-08-19T10:07:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 927432172 0 0) 0x7f886c13bcf0 con 0x7f886c0a67c0 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6791/0 conn(0x7f886c0a6ba0 legacy=0x7f886c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.1:6789/0 conn(0x7f886c13f5c0 legacy=0x7f886c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 monclient: found mon.e 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c13e560 con 0x7f886c0a67c0 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 5 monclient: authenticate success, global_id 204452 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 10 monclient: _renew_subs 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f886c0a28b0 con 0x7f886c0a67c0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 10 client.204452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 10 monclient: _renew_subs 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f886c13d1f0 con 0x7f886c0a67c0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f8872c20640 1 librados: init done 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8864003bd0 con 0x7f886c0a67c0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8864004140 con 0x7f886c0a67c0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f886affd640 4 set_mon_vals no callback set 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.325+0000 7f88697fa640 10 monclient: dump: 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.353 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f8872c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88500065c0 con 0x7f8850002860 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f88640042e0 con 0x7f886c0a67c0 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(654..654 src has 299..654) ==== 5979+0+0 (unknown 3667047492 0 0) 0x7f886404e510 con 0x7f886c0a67c0 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88697fa640 10 client.204452.objecter ms_dispatch 0x7f886c000b90 osd_map(654..654 src has 299..654) 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88697fa640 3 client.204452.objecter handle_osd_map got epochs [654,654] > 0 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88697fa640 3 client.204452.objecter handle_osd_map decoding full epoch 654 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88697fa640 20 client.204452.objecter dump_active .. 0 homeless 2024-08-19T10:07:45.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.326+0000 7f88711bd640 10 monclient: get_auth_request con 0x7f884c041950 auth_method 0 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f886bfff640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6792/0 conn(0x7f8850002860 legacy=0x7f8850004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 client.204452.objecter ms_handle_connect 0x7f8850002860 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 client.204452.objecter resend_mon_ops 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656406625 0 0) 0x7f88500065c0 con 0x7f8850002860 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient(hunting): my global_id is 204388 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c13bcf0 con 0x7f8850002860 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 629274383 0 0) 0x7f886c13bcf0 con 0x7f8850002860 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88500065c0 con 0x7f8850002860 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8854004d00 con 0x7f8850002860 2024-08-19T10:07:45.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 monclient: dump: 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.356 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 10 client.204452.objecter ms_handle_connect 0x7f884c041950 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2575761525 0 0) 0x7f88500065c0 con 0x7f8850002860 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.328+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f885000a160 con 0x7f8850002860 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.329+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f88540052a0 con 0x7f8850002860 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.329+0000 7f88697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.329+0000 7f88697fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.329+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6792/0 conn(0x7f8850002860 legacy=0x7f8850004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.329+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6792/0 conn(0x7f8850002860 legacy=0x7f8850004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.335+0000 7f8872c20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:45.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.335+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8850009e50 con 0x7f8850009110 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.337+0000 7f886bfff640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6792/0 conn(0x7f8850009110 legacy=0x7f88500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.337+0000 7f88697fa640 10 client.204452.objecter ms_handle_connect 0x7f8850009110 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.337+0000 7f88697fa640 10 client.204452.objecter resend_mon_ops 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039270136 0 0) 0x7f8850009e50 con 0x7f8850009110 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient(hunting): my global_id is 204397 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88500065c0 con 0x7f8850009110 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2015580276 0 0) 0x7f88500065c0 con 0x7f8850009110 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8850009e50 con 0x7f8850009110 2024-08-19T10:07:45.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8854007b30 con 0x7f8850009110 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient: dump: 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.359 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1622543352 0 0) 0x7f8850009e50 con 0x7f8850009110 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f885000c3a0 con 0x7f8850009110 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3788992027) 0x7f8854007ee0 con 0x7f8850009110 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6792/0 conn(0x7f8850009110 legacy=0x7f88500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f88697fa640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6792/0 conn(0x7f8850009110 legacy=0x7f88500095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 131209, 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:45.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.362 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:45.363 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:45.364 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:45.365 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:45.366 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.367 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:45.368 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.369 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:45.372 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.373 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.374 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.376 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.377 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:45.379 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:45.380 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:45.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f8872c20640 10 librados: watch_flush enter 2024-08-19T10:07:45.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f8872c20640 10 librados: watch_flush exit 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f8872c20640 20 client.204452.objecter shutdown clearing up homeless session... 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.338+0000 7f8872c20640 10 client.204452.objecter successfully canceled tick 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.82:6812/2549104415 conn(0x7f884c041950 legacy=0x7f884c043e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 10 monclient: shutdown 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 >> v1:172.21.15.1:6793/0 conn(0x7f886c0a67c0 legacy=0x7f886c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f88711bd640 1 -- 172.21.15.1:0/2147361685 reap_dead start 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f88711bd640 1 -- 172.21.15.1:0/2147361685 reap_dead start 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 shutdown_connections 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 >> 172.21.15.1:0/2147361685 conn(0x7f886c0ac4b0 msgr2=0x7f886c13ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 shutdown_connections 2024-08-19T10:07:45.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 -- 172.21.15.1:0/2147361685 wait complete. 2024-08-19T10:07:45.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.339+0000 7f8872c20640 1 librados: shutdown 2024-08-19T10:07:45.384 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:45.384 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T10:07:45.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T10:07:45.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.481+0000 7f91d673a640 10 monclient: get_monmap_and_config 2024-08-19T10:07:45.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.481+0000 7f91d673a640 10 monclient: build_initial_monmap 2024-08-19T10:07:45.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.481+0000 7f91d673a640 10 monclient: monmap: 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:45.482383+0000 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:45.482383+0000 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.484 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.484 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 1 Processor -- start 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 1 -- start start 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient: init 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient: _add_conns ranks=[6,0,2,7,3,4,5,1,8] 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient(hunting): picked mon.noname-d con 0x7f91d00a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient(hunting): picked mon.noname-a con 0x7f91d00a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient(hunting): picked mon.noname-b con 0x7f91d013f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d00a3b40 con 0x7f91d00a6ba0 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d00a8130 con 0x7f91d013f5c0 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d006ede0 con 0x7f91d00a67c0 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d673a640 10 monclient(hunting): authenticate will time out at 1721.558740s 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91cffff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f91d00a67c0 0x7f91d00a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:48206/0 (socket says 172.21.15.1:48206) 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91cffff640 1 -- 172.21.15.1:0/2983453171 learned_addr learned my addr 172.21.15.1:0/2983453171 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91d4cd7640 1 --1- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6790/0 conn(0x7f91d013f5c0 0x7f91d01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:57378/0 (socket says 172.21.15.1:57378) 2024-08-19T10:07:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.482+0000 7f91cf7fe640 1 --1- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6789/0 conn(0x7f91d00a6ba0 0x7f91d00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54778/0 (socket says 172.21.15.1:54778) 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91cffff640 1 -- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6792/0 conn(0x7f91d00a67c0 legacy=0x7f91d00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91cf7fe640 1 -- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6789/0 conn(0x7f91d00a6ba0 legacy=0x7f91d00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91d4cd7640 1 -- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6790/0 conn(0x7f91d013f5c0 legacy=0x7f91d01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228004972 0 0) 0x7f91d006ede0 con 0x7f91d00a67c0 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): my global_id is 204369 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91b8003520 con 0x7f91d00a67c0 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589753059 0 0) 0x7f91d00a3b40 con 0x7f91d00a6ba0 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): my global_id is 204417 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91d006ede0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1564173552 0 0) 0x7f91d00a8130 con 0x7f91d013f5c0 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient(hunting): my global_id is 184781 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91d00a3b40 con 0x7f91d013f5c0 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91cffff640 1 --1- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6792/0 conn(0x7f91d00a67c0 0x7f91d00a3f60 :-1 s=OPENED pgs=3578 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f91c0001ad0 mon_map magic: 0 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4061107409 0 0) 0x7f91d006ede0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6792/0 conn(0x7f91d00a67c0 legacy=0x7f91d00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6790/0 conn(0x7f91d013f5c0 legacy=0x7f91d01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 monclient: found mon.noname-a 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d00a92a0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f91c4001ac0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91d673a640 5 monclient: authenticate success, global_id 204417 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91d673a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: dump: 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.493 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91d673a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f91c0001ad0 con 0x7f91d00a67c0 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91c40047e0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 1 -- 172.21.15.1:0/2983453171 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f91c4004d50 con 0x7f91d00a6ba0 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91d673a640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91d673a640 4 set_mon_vals no callback set 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.485+0000 7f91ceffd640 10 monclient: dump: 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 10 monclient: shutdown 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 -- 172.21.15.1:0/2983453171 >> v1:172.21.15.1:6789/0 conn(0x7f91d00a6ba0 legacy=0x7f91d00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 -- 172.21.15.1:0/2983453171 shutdown_connections 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 -- 172.21.15.1:0/2983453171 >> 172.21.15.1:0/2983453171 conn(0x7f91d00ac4b0 msgr2=0x7f91d00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 -- 172.21.15.1:0/2983453171 shutdown_connections 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 -- 172.21.15.1:0/2983453171 wait complete. 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 10 monclient: build_initial_monmap 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 10 monclient: monmap: 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:45.497 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 librados: starting msgr at 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.486+0000 7f91d673a640 1 librados: starting objecter 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 Processor -- start 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 -- start start 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 librados: setting wanted keys 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 librados: calling monclient init 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient: init 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient: _add_conns ranks=[8,4,7,5,1,3,2,6,0] 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient(hunting): picked mon.noname-e con 0x7f91d00a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient(hunting): picked mon.noname-c con 0x7f91d00a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient(hunting): picked mon.noname-i con 0x7f91d013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:45.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d013c5a0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d013a7d0 con 0x7f91d013f5c0 2024-08-19T10:07:45.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d013c880 con 0x7f91d00a67c0 2024-08-19T10:07:45.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:45.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91d673a640 10 monclient(hunting): authenticate will time out at 1721.563722s 2024-08-19T10:07:45.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91cffff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f91d00a67c0 0x7f91d01399b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:50712/0 (socket says 172.21.15.1:50712) 2024-08-19T10:07:45.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.487+0000 7f91cffff640 1 -- 172.21.15.1:0/566816257 learned_addr learned my addr 172.21.15.1:0/566816257 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:45.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.489+0000 7f91cf7fe640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.1:6791/0 conn(0x7f91d00a6ba0 legacy=0x7f91d013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.489+0000 7f91cffff640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.1:6793/0 conn(0x7f91d00a67c0 legacy=0x7f91d01399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.489+0000 7f91ccff9640 10 client.?.objecter ms_handle_connect 0x7f91d00a6ba0 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.489+0000 7f91ccff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.489+0000 7f91ccff9640 10 client.?.objecter ms_handle_connect 0x7f91d00a67c0 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.489+0000 7f91ccff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1390864697 0 0) 0x7f91d013c5a0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): my global_id is 194611 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91bc004eb0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194642536 0 0) 0x7f91d013c880 con 0x7f91d00a67c0 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): my global_id is 204461 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91d013c5a0 con 0x7f91d00a67c0 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 client.?.objecter ms_handle_connect 0x7f91d013f5c0 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d4cd7640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.82:6792/0 conn(0x7f91d013f5c0 legacy=0x7f91d01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3604275139 0 0) 0x7f91bc004eb0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91d013c880 con 0x7f91d00a6ba0 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f91c4004cd0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): dump: 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.505 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598380194 0 0) 0x7f91d013a7d0 con 0x7f91d013f5c0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): my global_id is 204406 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91bc004eb0 con 0x7f91d013f5c0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3381231634 0 0) 0x7f91d013c5a0 con 0x7f91d00a67c0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91d013a7d0 con 0x7f91d00a67c0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f91c0001990 con 0x7f91d00a67c0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:45.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient(hunting): dump: 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.508 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1986368058 0 0) 0x7f91d013c880 con 0x7f91d00a6ba0 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.1:6793/0 conn(0x7f91d00a67c0 legacy=0x7f91d01399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.82:6792/0 conn(0x7f91d013f5c0 legacy=0x7f91d01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: _finish_hunting 0 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 monclient: found mon.c 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d01429d0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: _finish_auth 0 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: _check_auth_tickets 2024-08-19T10:07:45.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 5 monclient: authenticate success, global_id 194611 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 10 monclient: _renew_subs 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f91d013c2c0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 10 client.194611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 10 monclient: _renew_subs 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:45.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f91d00a1e50 con 0x7f91d00a6ba0 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91d673a640 1 librados: init done 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91c4001d10 con 0x7f91d00a6ba0 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f91c4005680 con 0x7f91d00a6ba0 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ce7fc640 4 set_mon_vals no callback set 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.490+0000 7f91ccff9640 10 monclient: dump: 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f91c4011ae0 con 0x7f91d00a6ba0 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91d673a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91d673a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91d673a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b0005080 con 0x7f91d00a6ba0 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(654..654 src has 299..654) ==== 5979+0+0 (unknown 3667047492 0 0) 0x7f91c404e490 con 0x7f91d00a6ba0 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91ccff9640 10 client.194611.objecter ms_dispatch 0x7f91d0000b90 osd_map(654..654 src has 299..654) 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91ccff9640 3 client.194611.objecter handle_osd_map got epochs [654,654] > 0 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91ccff9640 3 client.194611.objecter handle_osd_map decoding full epoch 654 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.491+0000 7f91ccff9640 20 client.194611.objecter dump_active .. 0 homeless 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.492+0000 7f91cffff640 10 monclient: get_auth_request con 0x7f91bc041790 auth_method 0 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.494+0000 7f91ccff9640 10 client.194611.objecter ms_handle_connect 0x7f91bc041790 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.496+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f91c4017680 con 0x7f91d00a6ba0 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.496+0000 7f91ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.496+0000 7f91ccff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.749+0000 7f91d673a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T10:07:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.749+0000 7f91d673a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T10:07:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.749+0000 7f91d673a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.749+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f91b0005c40 con 0x7f91d00a6ba0 2024-08-19T10:07:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.751+0000 7f91ccff9640 1 -- 172.21.15.1:0/566816257 <== mon.4 v1:172.21.15.1:6791/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f91c4016f80 con 0x7f91d00a6ba0 2024-08-19T10:07:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f91ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T10:07:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f91ccff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:45.756 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.756 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.756 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=469079417,config=2586702516,health=3620876252,kv=2657046684,logm=1836637842}) 2024-08-19T10:07:45.756 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.756 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.756 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=232664299,config=2586702516,health=3620876252,kv=1455257465,logm=2648464894}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=25,config=2,health=11,kv=12,logm=50} crc {auth=2913850410,config=2586702516,health=3971658869,kv=1455257465,logm=3819867031}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=588364972,config=2586702516,health=2967847960,kv=1251488192,logm=3101574734}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=3766174750,config=2586702516,health=3620876252,kv=2971667195,logm=3952278214}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=1448885239,config=2586702516,health=3620876252,kv=1455257465,logm=3643138930}) 2024-08-19T10:07:45.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=158116814,config=2586702516,health=3620876252,kv=1895581218,logm=1764515803}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=12,logm=52} crc {auth=657115011,config=1228700967,health=1627243500,kv=1455257465,logm=2088711927}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=2453838742,config=2586702516,health=3620876252,kv=3256936250,logm=470706338}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2823230737}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645162292}) 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2184138929}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2612178643}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2129016800}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3796329763}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3821037992}) 2024-08-19T10:07:45.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239690927}) 2024-08-19T10:07:45.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1214637955}) 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.754+0000 7f91d673a640 10 librados: watch_flush enter 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 10 librados: watch_flush exit 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 20 client.194611.objecter shutdown clearing up homeless session... 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 10 client.194611.objecter successfully canceled tick 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.82:6812/2549104415 conn(0x7f91bc041790 legacy=0x7f91bc043c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 10 monclient: shutdown 2024-08-19T10:07:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:45.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 >> v1:172.21.15.1:6791/0 conn(0x7f91d00a6ba0 legacy=0x7f91d013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:45.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 shutdown_connections 2024-08-19T10:07:45.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 >> 172.21.15.1:0/566816257 conn(0x7f91d00ac4b0 msgr2=0x7f91d00ab120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:45.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 shutdown_connections 2024-08-19T10:07:45.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 -- 172.21.15.1:0/566816257 wait complete. 2024-08-19T10:07:45.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:45.755+0000 7f91d673a640 1 librados: shutdown 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748968024}) 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860248055}) 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2271666762}) 2024-08-19T10:07:45.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4276203693}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=216200444}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=347931202}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3339357542}) 2024-08-19T10:07:45.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665077274}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.756+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4114026468}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104094018}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917850998}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=907460293}) 2024-08-19T10:07:45.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=367869876}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1352352012}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.757+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3485844801}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.758+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.758+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.758+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1669514448}) 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.758+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.758+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094587258}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.758+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=872848317}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=25778663}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2166698705}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080619211}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1981112570}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1719719667}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1292146822}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2268149887}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345385853}) 2024-08-19T10:07:45.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.759+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4007190809}) 2024-08-19T10:07:45.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335089096}) 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079929425}) 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.961+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2568133202}) 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.964 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=618278783}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1225786595}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1732545493}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3716632941}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938832504}) 2024-08-19T10:07:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4248065803}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937594013}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=953142107}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1302452963}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.969 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1822185849}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2546868192}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4059508495}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=829487502}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432628364}) 2024-08-19T10:07:45.970 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:45.966+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3998556511}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879387424}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2092062590}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3235576391}) 2024-08-19T10:07:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2840994951}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1125844160}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3972555558}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=584229154}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961781495}) 2024-08-19T10:07:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3439365017}) 2024-08-19T10:07:46.735 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.735 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.735 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=894526884}) 2024-08-19T10:07:46.735 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.735 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.735 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167042237}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1988479204}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2503860167}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=986377403}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=708348684}) 2024-08-19T10:07:46.736 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2369874816}) 2024-08-19T10:07:46.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714117767}) 2024-08-19T10:07:46.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1236325465}) 2024-08-19T10:07:46.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.757 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3839762028}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2001978015}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=903102515}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4220449209}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1870331479}) 2024-08-19T10:07:46.758 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=783795099}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3772969315}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254800294}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1067276680}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4255072006}) 2024-08-19T10:07:46.759 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=291242125}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3711348572}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=409390055}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4254272571}) 2024-08-19T10:07:46.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1850629478}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=482496153}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583210756}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.736+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=260110127}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578872884}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.761 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369032065}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2075734665}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=111596405}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=462041513}) 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.762 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2457390395}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3271690503}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395367990}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.738+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1610567480}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=761894565}) 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.763 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251482805}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768550720}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.739+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4207111686}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=692438520}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.764 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4128887399}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1826497089}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315454359}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.740+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=123157213}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59818255}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.765 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=978261072}) 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=455026560}) 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388523822}) 2024-08-19T10:07:46.766 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3804631717}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3965748622}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=160795082}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703305927}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.741+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3979321494}) 2024-08-19T10:07:46.767 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586658937}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402151363}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=222115746}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2303435028}) 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.768 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2426450627}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=445944962}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=984248880}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268724758}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.743+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=41871642}) 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.769 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540509561}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3888995752}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3692699666}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2684808474}) 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.770 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=948473610}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3331885260}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=261219725}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126657623}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.744+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3858168337}) 2024-08-19T10:07:46.771 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3881187489}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309099663}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2068929080}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2327342658}) 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.772 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1159525147}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1572640794}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1658550948}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252310727}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.746+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2539015880}) 2024-08-19T10:07:46.773 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.747+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.747+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.747+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193214248}) 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.747+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.747+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1752317293}) 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3367127889}) 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.774 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1000267584}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=930570172}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3762140018}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2061269777}) 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.775 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775850384}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.748+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3026171297}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948591131}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429766517}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=508924315}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.776 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=887296511}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2720273519}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2355091071}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1263867003}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593702889}) 2024-08-19T10:07:46.777 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.749+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1953688163}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563704267}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286598292}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3046972756}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.778 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=75990461}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3635242470}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2302248156}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1224548401}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240047963}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.751+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4071568406}) 2024-08-19T10:07:46.779 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.752+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013600086}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977373324}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1369003942}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=566389543}) 2024-08-19T10:07:46.780 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3293331991}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1352025725}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=848574662}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195677818}) 2024-08-19T10:07:46.781 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.753+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3242034653}) 2024-08-19T10:07:46.782 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:07:46.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1086844848}) 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083965764}) 2024-08-19T10:07:46.804 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3725739347}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2279570351}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3088072732}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3175704446}) 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.805 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.762+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.763+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3490601374}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.763+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.763+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063819920}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.763+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1474461900}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686463646}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470392145}) 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.806 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3072346254}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=37056477}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3509169377}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1833516592}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.807 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2750666222}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701587674}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.764+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3662212745}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153431367}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1153651232}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=916204759}) 2024-08-19T10:07:46.808 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=950441717}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=982981496}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1103090554}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3600663657}) 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.765+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.809 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.766+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176616555}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.766+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1110378066}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349029579}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=555704433}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4230995918}) 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.810 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1501982410}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4236972827}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=809829580}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574382495}) 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.811 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770366943}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.767+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1364635517}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351776762}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3546193434}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1397426960}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.812 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1029634988}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2696397382}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3751181347}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2733363153}) 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.813 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.768+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742490848}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.769+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2658243326}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=361803422}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914471921}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3520445067}) 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.814 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=822470146}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=569519393}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=526924318}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3549961752}) 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.815 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013862914}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.770+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2049848626}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.771+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.771+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.771+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128302443}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.771+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.771+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.771+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1317797835}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1090405338}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.816 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3269320013}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=425714855}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1777650424}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=364253751}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598240437}) 2024-08-19T10:07:46.817 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.772+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2499033551}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4227449537}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3559396911}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3721472444}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3301755995}) 2024-08-19T10:07:46.818 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2019645816}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3444592147}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737480943}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405050117}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.773+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=960037408}) 2024-08-19T10:07:46.819 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234753574}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=741233827}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=280377658}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1688911008}) 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.820 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1341253144}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2042243619}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819317391}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644188703}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.775+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=232556813}) 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.821 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2421251532}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685196729}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2360502083}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1146867062}) 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.822 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.776+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3873483084}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1965494904}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4236959802}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807609728}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.777+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=22249679}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.823 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872447169}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830046476}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1446192706}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=117889309}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.824 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1063982596}) 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=721382038}) 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2099011457}) 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665193492}) 2024-08-19T10:07:46.825 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.778+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1729401825}) 2024-08-19T10:07:46.858 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.856+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.856+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.856+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816779581}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.856+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.856+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.856+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352083597}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1069943397}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.859 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4097265322}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=754457730}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=994274338}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1187232544}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250359922}) 2024-08-19T10:07:46.860 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.857+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3474370173}) 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.931+0000 7fc3d7fcc640 10 monclient: get_monmap_and_config 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.931+0000 7fc3d7fcc640 10 monclient: build_initial_monmap 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.931+0000 7fc3d7fcc640 10 monclient: monmap: 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:46.932163+0000 2024-08-19T10:07:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:46.932163+0000 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:46.933 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.931+0000 7fc3d7fcc640 1 Processor -- start 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 1 -- start start 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient: init 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient: _add_conns ranks=[7,8,5,0,3,1,4,6,2] 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient(hunting): picked mon.noname-i con 0x7fc3d00a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient(hunting): picked mon.noname-e con 0x7fc3d00694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient(hunting): picked mon.noname-h con 0x7fc3d00a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00a3c70 con 0x7fc3d00a46a0 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00a33d0 con 0x7fc3d00a6830 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00a36b0 con 0x7fc3d00694b0 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d7fcc640 10 monclient(hunting): authenticate will time out at 1723.008572s 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d5567640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc3d00694b0 0x7fc3d00a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:55330/0 (socket says 172.21.15.1:55330) 2024-08-19T10:07:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.932+0000 7fc3d5567640 1 -- 172.21.15.1:0/1486001072 learned_addr learned my addr 172.21.15.1:0/1486001072 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.934+0000 7fc3d5567640 1 -- 172.21.15.1:0/1486001072 >> v1:172.21.15.1:6793/0 conn(0x7fc3d00694b0 legacy=0x7fc3d00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.934+0000 7fc3d5d68640 1 -- 172.21.15.1:0/1486001072 >> v1:172.21.15.82:6792/0 conn(0x7fc3d00a6830 legacy=0x7fc3d00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.934+0000 7fc3d6569640 1 -- 172.21.15.1:0/1486001072 >> v1:172.21.15.82:6791/0 conn(0x7fc3d00a46a0 legacy=0x7fc3d0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.934+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293225848 0 0) 0x7fc3d00a36b0 con 0x7fc3d00694b0 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): _negotiate 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): my global_id is 204470 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3bc003520 con 0x7fc3d00694b0 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376417857 0 0) 0x7fc3d00a33d0 con 0x7fc3d00a6830 2024-08-19T10:07:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): _negotiate 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient(hunting): my global_id is 204415 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d00a36b0 con 0x7fc3d00a6830 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3500714484 0 0) 0x7fc3bc003520 con 0x7fc3d00694b0 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 >> v1:172.21.15.82:6792/0 conn(0x7fc3d00a6830 legacy=0x7fc3d00a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 >> v1:172.21.15.82:6791/0 conn(0x7fc3d00a46a0 legacy=0x7fc3d0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: _finish_hunting 0 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 monclient: found mon.noname-e 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d006dbb0 con 0x7fc3d00694b0 2024-08-19T10:07:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: _finish_auth 0 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: _check_auth_tickets 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc3c0001ae0 con 0x7fc3d00694b0 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: dump: 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.948 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 5 monclient: authenticate success, global_id 204470 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3c0001d30 con 0x7fc3d00694b0 2024-08-19T10:07:46.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 1 -- 172.21.15.1:0/1486001072 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc3c0004be0 con 0x7fc3d00694b0 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d4d66640 10 monclient: dump: 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.950 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 4 set_mon_vals no callback set 2024-08-19T10:07:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 10 monclient: shutdown 2024-08-19T10:07:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1486001072 >> v1:172.21.15.1:6793/0 conn(0x7fc3d00694b0 legacy=0x7fc3d00a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1486001072 shutdown_connections 2024-08-19T10:07:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.935+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1486001072 >> 172.21.15.1:0/1486001072 conn(0x7fc3d00a9020 msgr2=0x7fc3d0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1486001072 shutdown_connections 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1486001072 wait complete. 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 10 monclient: build_initial_monmap 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 10 monclient: monmap: 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:46.953 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 1 librados: starting msgr at 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 1 librados: starting objecter 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.936+0000 7fc3d7fcc640 1 Processor -- start 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 1 -- start start 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 1 librados: setting wanted keys 2024-08-19T10:07:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 1 librados: calling monclient init 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient: init 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient: _add_conns ranks=[1,3,5,6,2,0,4,7,8] 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient(hunting): picked mon.noname-f con 0x7fc3d00694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient(hunting): picked mon.noname-g con 0x7fc3d00a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient(hunting): picked mon.noname-h con 0x7fc3d00a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d0138080 con 0x7fc3d00694b0 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d0138360 con 0x7fc3d00a46a0 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d01425e0 con 0x7fc3d00a6830 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d7fcc640 10 monclient(hunting): authenticate will time out at 1723.013554s 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d6569640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7fc3d00a6830 0x7fc3d01d5e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:34438/0 (socket says 172.21.15.1:34438) 2024-08-19T10:07:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.937+0000 7fc3d6569640 1 -- 172.21.15.1:0/1076936236 learned_addr learned my addr 172.21.15.1:0/1076936236 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d5d68640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6789/0 conn(0x7fc3d00694b0 legacy=0x7fc3d0141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 client.?.objecter ms_handle_connect 0x7fc3d00694b0 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d6569640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6791/0 conn(0x7fc3d00a6830 legacy=0x7fc3d01d5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 client.?.objecter ms_handle_connect 0x7fc3d00a6830 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 client.?.objecter ms_handle_connect 0x7fc3d00a46a0 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d5567640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6790/0 conn(0x7fc3d00a46a0 legacy=0x7fc3d01cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101647565 0 0) 0x7fc3d0138080 con 0x7fc3d00694b0 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): my global_id is 175150 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b4004ee0 con 0x7fc3d00694b0 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658560143 0 0) 0x7fc3d01425e0 con 0x7fc3d00a6830 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): my global_id is 214169 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d0138080 con 0x7fc3d00a6830 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3119835059 0 0) 0x7fc3d0138360 con 0x7fc3d00a46a0 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): my global_id is 214185 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d01425e0 con 0x7fc3d00a46a0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4182840645 0 0) 0x7fc3b4004ee0 con 0x7fc3d00694b0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3d0138360 con 0x7fc3d00694b0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 218761006 0 0) 0x7fc3d01425e0 con 0x7fc3d00a46a0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3b4004ee0 con 0x7fc3d00a46a0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3330392226 0 0) 0x7fc3d0138080 con 0x7fc3d00a6830 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3d01425e0 con 0x7fc3d00a6830 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc3cc003610 con 0x7fc3d00694b0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): dump: 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:46.959 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc3c0001b00 con 0x7fc3d00a46a0 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): dump: 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc3c8003990 con 0x7fc3d00a6830 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient(hunting): dump: 2024-08-19T10:07:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2529734299 0 0) 0x7fc3d0138360 con 0x7fc3d00694b0 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6791/0 conn(0x7fc3d00a6830 legacy=0x7fc3d01d5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6790/0 conn(0x7fc3d00a46a0 legacy=0x7fc3d01cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 monclient: found mon.f 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d01ce170 con 0x7fc3d00694b0 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient: _finish_auth 0 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:46.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 5 monclient: authenticate success, global_id 175150 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 10 monclient: _renew_subs 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d01ce400 con 0x7fc3d00694b0 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 10 client.175150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 10 monclient: _renew_subs 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3d01d7700 con 0x7fc3d00694b0 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3d7fcc640 1 librados: init done 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.940+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3cc003be0 con 0x7fc3d00694b0 2024-08-19T10:07:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3c67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc3cc0040d0 con 0x7fc3d00694b0 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3c7fff640 4 set_mon_vals no callback set 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3c67fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3c67fc640 10 monclient: dump: 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:46.966 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3d7fcc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3d7fcc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3d7fcc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.941+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b8005080 con 0x7fc3d00694b0 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc3cc0042f0 con 0x7fc3d00694b0 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(655..655 src has 299..655) ==== 5606+0+0 (unknown 4130503094 0 0) 0x7fc3cc04e2d0 con 0x7fc3d00694b0 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3c67fc640 10 client.175150.objecter ms_dispatch 0x7fc3d0000b90 osd_map(655..655 src has 299..655) 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3c67fc640 3 client.175150.objecter handle_osd_map got epochs [655,655] > 0 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3c67fc640 3 client.175150.objecter handle_osd_map decoding full epoch 655 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3c67fc640 20 client.175150.objecter dump_active .. 0 homeless 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.942+0000 7fc3d5567640 10 monclient: get_auth_request con 0x7fc3b4041980 auth_method 0 2024-08-19T10:07:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.944+0000 7fc3c67fc640 10 client.175150.objecter ms_handle_connect 0x7fc3b4041980 2024-08-19T10:07:46.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.947+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fc3cc0175d0 con 0x7fc3d00694b0 2024-08-19T10:07:46.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.947+0000 7fc3c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:46.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:46.947+0000 7fc3c67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916268454}) 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2792373409}) 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1230301242}) 2024-08-19T10:07:46.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=549157476}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3719100865}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=159045892}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4258327143}) 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:46.981 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099297877}) 2024-08-19T10:07:46.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.977+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2769955510}) 2024-08-19T10:07:47.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597599586}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108096363}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2921016382}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=321964272}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2528025850}) 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1455782968}) 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3566276268}) 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190092504}) 2024-08-19T10:07:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:46.999+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4156249706}) 2024-08-19T10:07:47.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=298498358}) 2024-08-19T10:07:47.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207791322}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.011+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3255578587}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1792606972}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2786862848}) 2024-08-19T10:07:47.015 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2487264395}) 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=120122728}) 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307130554}) 2024-08-19T10:07:47.016 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.012+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3585983649}) 2024-08-19T10:07:47.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163332962}) 2024-08-19T10:07:47.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701477019}) 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2563224783}) 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2542370495}) 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3800723650}) 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=994695786}) 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=167296986}) 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680744447}) 2024-08-19T10:07:47.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=786852918}) 2024-08-19T10:07:47.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.226+0000 7fc3d7fcc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:07:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.226+0000 7fc3d7fcc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:07:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.226+0000 7fc3d7fcc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.82:6789/0 2024-08-19T10:07:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.226+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3b8005370 con 0x7fc3d00694b0 2024-08-19T10:07:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.227+0000 7fc3c67fc640 1 -- 172.21.15.1:0/1076936236 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 1203090166) 0x7fc3cc016ed0 con 0x7fc3d00694b0 2024-08-19T10:07:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.227+0000 7fc3c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:07:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.227+0000 7fc3c67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:47.229 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":138,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:07:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 10 librados: watch_flush enter 2024-08-19T10:07:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 10 librados: watch_flush exit 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 20 client.175150.objecter shutdown clearing up homeless session... 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 10 client.175150.objecter successfully canceled tick 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc3b4041980 legacy=0x7fc3b4043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 10 monclient: shutdown 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 >> v1:172.21.15.82:6789/0 conn(0x7fc3d00694b0 legacy=0x7fc3d0141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 shutdown_connections 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 >> 172.21.15.1:0/1076936236 conn(0x7fc3d00a9020 msgr2=0x7fc3d00a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 shutdown_connections 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 -- 172.21.15.1:0/1076936236 wait complete. 2024-08-19T10:07:47.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.229+0000 7fc3d7fcc640 1 librados: shutdown 2024-08-19T10:07:47.240 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:07:47.240 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T10:07:47.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:07:47.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.378+0000 7f8bc15a0640 10 monclient: get_monmap_and_config 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.378+0000 7f8bc15a0640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.378+0000 7f8bc15a0640 10 monclient: monmap: 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:47.379779+0000 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:47.379779+0000 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.381 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.379+0000 7f8bc15a0640 1 Processor -- start 2024-08-19T10:07:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.379+0000 7f8bc15a0640 1 -- start start 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.379+0000 7f8bc15a0640 10 monclient: init 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient: _add_conns ranks=[5,3,6,8,1,2,0,7,4] 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient(hunting): picked mon.noname-h con 0x7f8bbc0a8ce0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient(hunting): picked mon.noname-g con 0x7f8bbc0a5720 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient(hunting): picked mon.noname-d con 0x7f8bbc0a5b00 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc136660 con 0x7f8bbc0a5720 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc09ded0 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc09e1b0 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc15a0640 10 monclient(hunting): authenticate will time out at 1723.456142s 2024-08-19T10:07:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc0d9f640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f8bbc0a5b00 0x7f8bbc1418c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45644/0 (socket says 172.21.15.1:45644) 2024-08-19T10:07:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.380+0000 7f8bc0d9f640 1 -- 172.21.15.1:0/3026558408 learned_addr learned my addr 172.21.15.1:0/3026558408 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bc0d9f640 1 -- 172.21.15.1:0/3026558408 >> v1:172.21.15.1:6792/0 conn(0x7f8bbc0a5b00 legacy=0x7f8bbc1418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298542669 0 0) 0x7f8bbc09e1b0 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient(hunting): my global_id is 204378 2024-08-19T10:07:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba4003520 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbbfff640 1 -- 172.21.15.1:0/3026558408 >> v1:172.21.15.82:6791/0 conn(0x7f8bbc0a8ce0 legacy=0x7f8bbc135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbb7fe640 1 -- 172.21.15.1:0/3026558408 >> v1:172.21.15.82:6790/0 conn(0x7f8bbc0a5720 legacy=0x7f8bbc13e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3795406864 0 0) 0x7f8ba4003520 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 >> v1:172.21.15.82:6791/0 conn(0x7f8bbc0a8ce0 legacy=0x7f8bbc135f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 >> v1:172.21.15.82:6790/0 conn(0x7f8bbc0a5720 legacy=0x7f8bbc13e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 monclient: found mon.noname-d 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc0a5f50 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bb4001e60 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bbaffd640 10 monclient: dump: 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.395 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bc15a0640 5 monclient: authenticate success, global_id 204378 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.382+0000 7f8bc15a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bb4004c20 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bbaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bbaffd640 1 -- 172.21.15.1:0/3026558408 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bb4005190 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bbaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bbaffd640 10 monclient: dump: 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 4 set_mon_vals no callback set 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 10 monclient: shutdown 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 1 -- 172.21.15.1:0/3026558408 >> v1:172.21.15.1:6792/0 conn(0x7f8bbc0a5b00 legacy=0x7f8bbc1418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 1 -- 172.21.15.1:0/3026558408 shutdown_connections 2024-08-19T10:07:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 1 -- 172.21.15.1:0/3026558408 >> 172.21.15.1:0/3026558408 conn(0x7f8bbc0ac100 msgr2=0x7f8bbc0a85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 1 -- 172.21.15.1:0/3026558408 shutdown_connections 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.383+0000 7f8bc15a0640 1 -- 172.21.15.1:0/3026558408 wait complete. 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 10 monclient: monmap: 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 1 librados: starting msgr at 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 1 librados: starting objecter 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 1 Processor -- start 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 1 -- start start 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 1 librados: setting wanted keys 2024-08-19T10:07:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 1 librados: calling monclient init 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.384+0000 7f8bc15a0640 10 monclient: init 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient: _add_conns ranks=[0,8,4,3,2,6,5,7,1] 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bbc0a5720 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient(hunting): picked mon.noname-e con 0x7f8bbc0a5b00 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8bbc0a8ce0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc1321a0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc132480 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc0aa890 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bc15a0640 10 monclient(hunting): authenticate will time out at 1723.461098s 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bbbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8bbc0a5720 0x7f8bbc12eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41296/0 (socket says 172.21.15.1:41296) 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.385+0000 7f8bbbfff640 1 -- 172.21.15.1:0/4287463011 learned_addr learned my addr 172.21.15.1:0/4287463011 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bbb7fe640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6793/0 conn(0x7f8bbc0a5b00 legacy=0x7f8bbc12f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 client.?.objecter ms_handle_connect 0x7f8bbc0a5b00 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bbbfff640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8bbc0a5720 legacy=0x7f8bbc12eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 client.?.objecter ms_handle_connect 0x7f8bbc0a5720 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 client.?.objecter ms_handle_connect 0x7f8bbc0a8ce0 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bc0d9f640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6791/0 conn(0x7f8bbc0a8ce0 legacy=0x7f8bbc1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bbb7fe640 1 --1- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6793/0 conn(0x7f8bbc0a5b00 0x7f8bbc12f200 :-1 s=OPENED pgs=3712 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f8bbc0aa890 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 481020409 0 0) 0x7f8bbc1321a0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): my global_id is 204426 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba8004eb0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106749876 0 0) 0x7f8bbc132480 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): my global_id is 194620 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bbc1321a0 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 873154610 0 0) 0x7f8ba8004eb0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bbc132480 con 0x7f8bbc0a5720 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bac001ce0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): dump: 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.387+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 89760652 0 0) 0x7f8bbc1321a0 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ba8004eb0 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bb4005110 con 0x7f8bbc0a8ce0 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient(hunting): dump: 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 763464443 0 0) 0x7f8bbc132480 con 0x7f8bbc0a5720 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6793/0 conn(0x7f8bbc0a5b00 legacy=0x7f8bbc12f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6791/0 conn(0x7f8bbc0a8ce0 legacy=0x7f8bbc1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 monclient: found mon.a 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc0aa380 con 0x7f8bbc0a5720 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374001122 0 0) 0x7f8bbc0aa890 con 0x7f8bbc0a5b00 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 5 monclient: authenticate success, global_id 204426 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 10 monclient: _renew_subs 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc1353f0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 10 client.204426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 10 monclient: _renew_subs 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bbc131ec0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 1 librados: init done 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bac003a50 con 0x7f8bbc0a5720 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bac003f40 con 0x7f8bbc0a5720 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bba7fc640 4 set_mon_vals no callback set 2024-08-19T10:07:47.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bb8ff9640 10 monclient: dump: 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.388+0000 7f8bc15a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b9c0065c0 con 0x7f8b9c002860 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8bac0040e0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(656..656 src has 299..656) ==== 5979+0+0 (unknown 2284910107 0 0) 0x7f8bac04e3f0 con 0x7f8bbc0a5720 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bb8ff9640 10 client.204426.objecter ms_dispatch 0x7f8bbc000b90 osd_map(656..656 src has 299..656) 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bb8ff9640 3 client.204426.objecter handle_osd_map got epochs [656,656] > 0 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bb8ff9640 3 client.204426.objecter handle_osd_map decoding full epoch 656 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bb8ff9640 20 client.204426.objecter dump_active .. 0 homeless 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.389+0000 7f8bc0d9f640 10 monclient: get_auth_request con 0x7f8ba8041500 auth_method 0 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bbb7fe640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8b9c002860 legacy=0x7f8b9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 client.204426.objecter ms_handle_connect 0x7f8b9c002860 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 client.204426.objecter resend_mon_ops 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122991987 0 0) 0x7f8b9c0065c0 con 0x7f8b9c002860 2024-08-19T10:07:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient(hunting): my global_id is 204435 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bbc0aa890 con 0x7f8b9c002860 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3912143260 0 0) 0x7f8bbc0aa890 con 0x7f8b9c002860 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b9c0065c0 con 0x7f8b9c002860 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bb0001cb0 con 0x7f8b9c002860 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 monclient: dump: 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1072436626 0 0) 0x7f8b9c0065c0 con 0x7f8b9c002860 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b9c00a160 con 0x7f8b9c002860 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.391+0000 7f8bb8ff9640 10 client.204426.objecter ms_handle_connect 0x7f8ba8041500 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.392+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8bb00068b0 con 0x7f8b9c002860 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.392+0000 7f8bb8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.392+0000 7f8bb8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.392+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8b9c002860 legacy=0x7f8b9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.392+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8b9c002860 legacy=0x7f8b9c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.398+0000 7f8bc15a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:47.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.398+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b9c009e70 con 0x7f8b9c009130 2024-08-19T10:07:47.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 client.204426.objecter ms_handle_connect 0x7f8b9c009130 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bbb7fe640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8b9c009130 legacy=0x7f8b9c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 client.204426.objecter resend_mon_ops 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567869200 0 0) 0x7f8b9c009e70 con 0x7f8b9c009130 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient(hunting): my global_id is 204444 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b9c0065c0 con 0x7f8b9c009130 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 58891209 0 0) 0x7f8b9c0065c0 con 0x7f8b9c009130 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b9c009e70 con 0x7f8b9c009130 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8bb0003cc0 con 0x7f8b9c009130 2024-08-19T10:07:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 10 monclient: dump: 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1181777021 0 0) 0x7f8b9c009e70 con 0x7f8b9c009130 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.400+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8b9c00c410 con 0x7f8b9c009130 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7687 (unknown 0 0 3560010956) 0x7f8bb00040d0 con 0x7f8b9c009130 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bb8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bb8ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8b9c009130 legacy=0x7f8b9c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bb8ff9640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8b9c009130 legacy=0x7f8b9c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 133272, 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:47.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.423 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:47.424 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.425 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:47.426 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.427 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.428 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:47.429 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.430 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:47.434 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:47.435 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.438 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.439 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:47.440 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.441 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:47.442 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:47.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 10 librados: watch_flush enter 2024-08-19T10:07:47.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 10 librados: watch_flush exit 2024-08-19T10:07:47.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 20 client.204426.objecter shutdown clearing up homeless session... 2024-08-19T10:07:47.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 10 client.204426.objecter successfully canceled tick 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8ba8041500 legacy=0x7f8ba80439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 10 monclient: shutdown 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 >> v1:172.21.15.1:6789/0 conn(0x7f8bbc0a5720 legacy=0x7f8bbc12eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc0d9f640 1 -- 172.21.15.1:0/4287463011 reap_dead start 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc0d9f640 1 -- 172.21.15.1:0/4287463011 reap_dead start 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 shutdown_connections 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 >> 172.21.15.1:0/4287463011 conn(0x7f8bbc0ac100 msgr2=0x7f8bbc06d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 shutdown_connections 2024-08-19T10:07:47.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 -- 172.21.15.1:0/4287463011 wait complete. 2024-08-19T10:07:47.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.401+0000 7f8bc15a0640 1 librados: shutdown 2024-08-19T10:07:47.446 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:47.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:07:47.446 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.545+0000 7f031b73a640 10 monclient: get_monmap_and_config 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.545+0000 7f031b73a640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.545+0000 7f031b73a640 10 monclient: monmap: 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:47.546099+0000 2024-08-19T10:07:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:47.546099+0000 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.547 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.545+0000 7f031b73a640 1 Processor -- start 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.545+0000 7f031b73a640 1 -- start start 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.545+0000 7f031b73a640 10 monclient: init 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient: _add_conns ranks=[7,5,8,1,0,4,2,6,3] 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient(hunting): picked mon.noname-i con 0x7f03140a6720 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient(hunting): picked mon.noname-h con 0x7f03140a4590 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient(hunting): picked mon.noname-e con 0x7f031413f730 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03140a32c0 con 0x7f03140a4590 2024-08-19T10:07:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03140a35a0 con 0x7f03140a6720 2024-08-19T10:07:47.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03140a3880 con 0x7f031413f730 2024-08-19T10:07:47.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f031b73a640 10 monclient(hunting): authenticate will time out at 1723.622507s 2024-08-19T10:07:47.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f0319cd7640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f031413f730 0x7f0314141b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:55340/0 (socket says 172.21.15.1:55340) 2024-08-19T10:07:47.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.546+0000 7f0319cd7640 1 -- 172.21.15.1:0/2310170989 learned_addr learned my addr 172.21.15.1:0/2310170989 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.548+0000 7f0319cd7640 1 -- 172.21.15.1:0/2310170989 >> v1:172.21.15.1:6793/0 conn(0x7f031413f730 legacy=0x7f0314141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.548+0000 7f0318cd5640 1 -- 172.21.15.1:0/2310170989 >> v1:172.21.15.82:6791/0 conn(0x7f03140a4590 legacy=0x7f03140a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.548+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617907176 0 0) 0x7f03140a3880 con 0x7f031413f730 2024-08-19T10:07:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient(hunting): my global_id is 204488 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f03194d6640 1 -- 172.21.15.1:0/2310170989 >> v1:172.21.15.82:6792/0 conn(0x7f03140a6720 legacy=0x7f03140a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc003520 con 0x7f031413f730 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3283978691 0 0) 0x7f02fc003520 con 0x7f031413f730 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 >> v1:172.21.15.82:6792/0 conn(0x7f03140a6720 legacy=0x7f03140a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 >> v1:172.21.15.82:6791/0 conn(0x7f03140a4590 legacy=0x7f03140a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 monclient: found mon.noname-e 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03140a92b0 con 0x7f031413f730 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f030c001ae0 con 0x7f031413f730 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 5 monclient: authenticate success, global_id 204488 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: dump: 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.556 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f030c0048a0 con 0x7f031413f730 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 1 -- 172.21.15.1:0/2310170989 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f030c004dd0 con 0x7f031413f730 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 4 set_mon_vals no callback set 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f0313fff640 10 monclient: dump: 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 10 monclient: shutdown 2024-08-19T10:07:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 1 -- 172.21.15.1:0/2310170989 >> v1:172.21.15.1:6793/0 conn(0x7f031413f730 legacy=0x7f0314141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 1 -- 172.21.15.1:0/2310170989 shutdown_connections 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.549+0000 7f031b73a640 1 -- 172.21.15.1:0/2310170989 >> 172.21.15.1:0/2310170989 conn(0x7f03140ac410 msgr2=0x7f03140a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 -- 172.21.15.1:0/2310170989 shutdown_connections 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 -- 172.21.15.1:0/2310170989 wait complete. 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 10 monclient: monmap: 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 librados: starting msgr at 2024-08-19T10:07:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 librados: starting objecter 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 Processor -- start 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 -- start start 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 librados: setting wanted keys 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 1 librados: calling monclient init 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.550+0000 7f031b73a640 10 monclient: init 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient: _add_conns ranks=[2,5,6,7,3,1,4,8,0] 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient(hunting): picked mon.noname-b con 0x7f03140a4590 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient(hunting): picked mon.noname-h con 0x7f03140a6720 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient(hunting): picked mon.noname-d con 0x7f031413f730 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0314124c00 con 0x7f03140a4590 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03141251d0 con 0x7f03140a6720 2024-08-19T10:07:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03140a9e00 con 0x7f031413f730 2024-08-19T10:07:47.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f031b73a640 10 monclient(hunting): authenticate will time out at 1723.627373s 2024-08-19T10:07:47.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f0319cd7640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 0x7f03141d66c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45660/0 (socket says 172.21.15.1:45660) 2024-08-19T10:07:47.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f03194d6640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f03140a4590 0x7f0314122370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51906/0 (socket says 172.21.15.1:51906) 2024-08-19T10:07:47.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.551+0000 7f0319cd7640 1 -- 172.21.15.1:0/4107943687 learned_addr learned my addr 172.21.15.1:0/4107943687 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0319cd7640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 legacy=0x7f03141d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0319cd7640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 legacy=0x7f03141d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0319cd7640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 legacy=0x7f03141d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0319cd7640 1 --1- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 0x7f03141d66c0 :-1 s=OPENED pgs=3580 cs=1 l=1).handle_message read tag failed 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0319cd7640 1 --1- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 0x7f03141d66c0 :-1 s=OPENED pgs=3580 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 client.?.objecter ms_handle_connect 0x7f031413f730 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f03194d6640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f03140a4590 legacy=0x7f0314122370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 client.?.objecter ms_handle_connect 0x7f03140a4590 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6792/0 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0318cd5640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.82:6791/0 conn(0x7f03140a6720 legacy=0x7f0314122a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 client.?.objecter ms_handle_connect 0x7f03140a6720 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617591381 0 0) 0x7f0314124c00 con 0x7f03140a4590 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): my global_id is 184790 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0300004eb0 con 0x7f03140a4590 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1834294084 0 0) 0x7f0308001540 con 0x7f03140a6720 2024-08-19T10:07:47.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 10 monclient(hunting): my global_id is 214196 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.553+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0314124c00 con 0x7f03140a6720 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1198604901 0 0) 0x7f0300004eb0 con 0x7f03140a4590 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0308001540 con 0x7f03140a4590 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0304003990 con 0x7f03140a4590 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.570 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1827954170 0 0) 0x7f0314124c00 con 0x7f03140a6720 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0300004eb0 con 0x7f03140a6720 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0308003950 con 0x7f03140a6720 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2285747742 0 0) 0x7f0308001540 con 0x7f03140a4590 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6792/0 conn(0x7f031413f730 legacy=0x7f03141d66c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.82:6791/0 conn(0x7f03140a6720 legacy=0x7f0314122a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 monclient: found mon.b 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0314123190 con 0x7f03140a4590 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 5 monclient: authenticate success, global_id 184790 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 10 monclient: _renew_subs 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f031413e5c0 con 0x7f03140a4590 2024-08-19T10:07:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0304003e80 con 0x7f03140a4590 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f03040043f0 con 0x7f03140a4590 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f0311ffb640 10 monclient: dump: 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 10 client.184790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 10 monclient: _renew_subs 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0314125a70 con 0x7f03140a4590 2024-08-19T10:07:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f03137fe640 4 set_mon_vals no callback set 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.554+0000 7f031b73a640 1 librados: init done 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f0304004690 con 0x7f03140a4590 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f031b73a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0314142f30 con 0x7f0314124270 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(656..656 src has 299..656) ==== 5979+0+0 (unknown 2284910107 0 0) 0x7f030404e7b0 con 0x7f03140a4590 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0311ffb640 10 client.184790.objecter ms_dispatch 0x7f0314000b90 osd_map(656..656 src has 299..656) 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0311ffb640 3 client.184790.objecter handle_osd_map got epochs [656,656] > 0 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0311ffb640 3 client.184790.objecter handle_osd_map decoding full epoch 656 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0311ffb640 20 client.184790.objecter dump_active .. 0 homeless 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.555+0000 7f0319cd7640 10 monclient: get_auth_request con 0x7f0300041510 auth_method 0 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 client.184790.objecter ms_handle_connect 0x7f0314124270 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 client.184790.objecter resend_mon_ops 2024-08-19T10:07:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0318cd5640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f0314124270 legacy=0x7f031409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902413333 0 0) 0x7f0314142f30 con 0x7f0314124270 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient(hunting): my global_id is 184799 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0308001540 con 0x7f0314124270 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3393298679 0 0) 0x7f0308001540 con 0x7f0314124270 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0314142f30 con 0x7f0314124270 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0308003ee0 con 0x7f0314124270 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 monclient: dump: 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.577 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 10 client.184790.objecter ms_handle_connect 0x7f0300041510 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2724239567 0 0) 0x7f0314142f30 con 0x7f0314124270 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.557+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0314124650 con 0x7f0314124270 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.558+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0308004120 con 0x7f0314124270 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.558+0000 7f0311ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.558+0000 7f0311ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.558+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f0314124270 legacy=0x7f031409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.558+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f0314124270 legacy=0x7f031409da50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.564+0000 7f031b73a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:47.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.564+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0314068f60 con 0x7f031402e1d0 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.566+0000 7f0318cd5640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f031402e1d0 legacy=0x7f031409e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.566+0000 7f0311ffb640 10 client.184790.objecter ms_handle_connect 0x7f031402e1d0 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.566+0000 7f0311ffb640 10 client.184790.objecter resend_mon_ops 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828622483 0 0) 0x7f0314068f60 con 0x7f031402e1d0 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient(hunting): my global_id is 184808 2024-08-19T10:07:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0314142f30 con 0x7f031402e1d0 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 911791063 0 0) 0x7f0314142f30 con 0x7f031402e1d0 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0314068f60 con 0x7f031402e1d0 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f0308005b00 con 0x7f031402e1d0 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient: dump: 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1264503319 0 0) 0x7f0314068f60 con 0x7f031402e1d0 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f031402e5b0 con 0x7f031402e1d0 2024-08-19T10:07:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 1584288133) 0x7f0308005e90 con 0x7f031402e1d0 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f031402e1d0 legacy=0x7f031409e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f0311ffb640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f031402e1d0 legacy=0x7f031409e160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:47.584 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 384815, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.585 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.586 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.587 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:47.588 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.590 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:47.591 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:47.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:47.594 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:47.600 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.601 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.603 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:47.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:47.605 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:47.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.567+0000 7f031b73a640 10 librados: watch_flush enter 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 10 librados: watch_flush exit 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 20 client.184790.objecter shutdown clearing up homeless session... 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 10 client.184790.objecter successfully canceled tick 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.82:6812/2549104415 conn(0x7f0300041510 legacy=0x7f03000439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 10 monclient: shutdown 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 >> v1:172.21.15.1:6790/0 conn(0x7f03140a4590 legacy=0x7f0314122370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f0319cd7640 1 -- 172.21.15.1:0/4107943687 reap_dead start 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f0319cd7640 1 -- 172.21.15.1:0/4107943687 reap_dead start 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 shutdown_connections 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 >> 172.21.15.1:0/4107943687 conn(0x7f03140ac410 msgr2=0x7f031413eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 shutdown_connections 2024-08-19T10:07:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 -- 172.21.15.1:0/4107943687 wait complete. 2024-08-19T10:07:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.568+0000 7f031b73a640 1 librados: shutdown 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434321726}) 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337088855}) 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3608486249}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=995758055}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1065731749}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1379458038}) 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.570+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.571+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3041972862}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.571+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.571+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953605949}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.571+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1854049222}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082120773}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1216416441}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.622 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1684638646}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4237564449}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1286473805}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3646061744}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3805957902}) 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.623 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090362623}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.572+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3628580048}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3616134751}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622164914}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2367021698}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.624 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3375118989}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=199387319}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.573+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2101210626}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4198237028}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415650003}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.574+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2478829816}) 2024-08-19T10:07:47.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209075191}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480207518}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1862636283}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3108855175}) 2024-08-19T10:07:47.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=277056471}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3675763397}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2558965796}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370548335}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.576+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2452554872}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2161979259}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107806122}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4224339980}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1648760018}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.628 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=28698000}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1720316684}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1139440533}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473619602}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.577+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1253364187}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055737661}) 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.629 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335687347}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2120844095}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3410645978}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2805682746}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.630 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1738769098}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=829967508}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.578+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.579+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195344748}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.579+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3969934654}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339178248}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3130649530}) 2024-08-19T10:07:47.631 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.632 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.632 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35470878}) 2024-08-19T10:07:47.632 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.633 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:47.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:07:47.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3866423335}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=734537652}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1247853035}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.668 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3047937566}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846876899}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.580+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1263201159}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1458100524}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1508405412}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2696828288}) 2024-08-19T10:07:47.669 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1123078768}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1598827407}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2558015603}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3033750592}) 2024-08-19T10:07:47.670 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580544878}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.581+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4184166646}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=694387155}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=906325124}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812228904}) 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.671 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3399132342}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3925818825}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2442346401}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2556003576}) 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.672 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385619406}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.591+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=916439002}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3083365510}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950458009}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.592+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=150322664}) 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.673 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3946497913}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1015461334}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4292598646}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=548253232}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041765796}) 2024-08-19T10:07:47.674 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.593+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2667355146}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3858333251}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717713900}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=797763363}) 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.675 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=674264862}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2773971526}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=171136008}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1118065414}) 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.676 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018911743}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.594+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3608087982}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992891081}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1764165356}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1009558988}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.677 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1464064106}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=102243518}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2829729945}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=268229484}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135594908}) 2024-08-19T10:07:47.678 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.595+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=590638468}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135413510}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448906707}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3631877373}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=296752468}) 2024-08-19T10:07:47.679 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=653017255}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2989321332}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1968144998}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160230725}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.597+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1799648203}) 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.680 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612458387}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456066056}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2561477897}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3576461024}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=34607095}) 2024-08-19T10:07:47.681 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1531104754}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1469483027}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276376195}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.598+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=741987791}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.599+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.599+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.599+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2845345132}) 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.599+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.682 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.599+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=572946574}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3228731361}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1349731356}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1582056581}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3202145658}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276882273}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480251263}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.600+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3587356894}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194054045}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.684 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647766023}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3293613319}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1442729218}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1219496134}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3141484801}) 2024-08-19T10:07:47.685 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2865962876}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951197295}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.601+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4246760096}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464336117}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618868446}) 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.686 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4065871338}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=880491364}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=120263833}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1844240961}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2767676348}) 2024-08-19T10:07:47.687 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132130013}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.602+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=85534954}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.603+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.603+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.603+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2667107537}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3341620284}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3405080948}) 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.688 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=777295182}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1142741330}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=796004668}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3437001015}) 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.689 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263326724}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.604+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3768113213}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763862058}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592794348}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=464143282}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.690 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=870773169}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1459824123}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4065551461}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2138528252}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155600134}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.605+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1860400844}) 2024-08-19T10:07:47.691 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667069544}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3252805805}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3632175773}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1212587585}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=957478459}) 2024-08-19T10:07:47.692 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3440476879}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3429324047}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085176754}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.617+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3328620093}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2668124213}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.693 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328705418}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2525159784}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3109297567}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1695767199}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.694 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3998372587}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3805774826}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283444702}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.619+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=209384091}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284954499}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.695 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137036623}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1969538697}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663570005}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=422056691}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1800573970}) 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.696 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3705579650}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120106242}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.620+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2661421977}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540545270}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729678837}) 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.697 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1346810092}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3001835580}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2256433740}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=622972660}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.621+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.698 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.622+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=278777102}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.622+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.622+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839315930}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.622+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=46559939}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580322757}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2455212007}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.699 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=391098559}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=478942819}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=665713038}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1878401044}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2299821867}) 2024-08-19T10:07:47.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380731595}) 2024-08-19T10:07:47.701 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.623+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2740745454}) 2024-08-19T10:07:47.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.800+0000 7f6fd858a640 10 monclient: get_monmap_and_config 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.800+0000 7f6fd858a640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.800+0000 7f6fd858a640 10 monclient: monmap: 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:47.801169+0000 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:47.801169+0000 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.802 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.800+0000 7f6fd858a640 1 Processor -- start 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.800+0000 7f6fd858a640 1 -- start start 2024-08-19T10:07:47.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.800+0000 7f6fd858a640 10 monclient: init 2024-08-19T10:07:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient: _add_conns ranks=[5,3,8,6,1,2,4,7,0] 2024-08-19T10:07:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient(hunting): picked mon.noname-h con 0x7f6fd00a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient(hunting): picked mon.noname-g con 0x7f6fd00a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient(hunting): picked mon.noname-e con 0x7f6fd013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd00a3b40 con 0x7f6fd00a6ba0 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd00a8130 con 0x7f6fd00a67c0 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd006ede0 con 0x7f6fd013f5c0 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd858a640 10 monclient(hunting): authenticate will time out at 1723.877564s 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd6b27640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f6fd013f5c0 0x7f6fd01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:55344/0 (socket says 172.21.15.1:55344) 2024-08-19T10:07:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.801+0000 7f6fd6b27640 1 -- 172.21.15.1:0/3485704632 learned_addr learned my addr 172.21.15.1:0/3485704632 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.803+0000 7f6fd6b27640 1 -- 172.21.15.1:0/3485704632 >> v1:172.21.15.1:6793/0 conn(0x7f6fd013f5c0 legacy=0x7f6fd01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.803+0000 7f6fd6326640 1 -- 172.21.15.1:0/3485704632 >> v1:172.21.15.82:6791/0 conn(0x7f6fd00a67c0 legacy=0x7f6fd00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5b25640 1 -- 172.21.15.1:0/3485704632 >> v1:172.21.15.82:6790/0 conn(0x7f6fd00a6ba0 legacy=0x7f6fd00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659965942 0 0) 0x7f6fd006ede0 con 0x7f6fd013f5c0 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): my global_id is 204497 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fbc003520 con 0x7f6fd013f5c0 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953602566 0 0) 0x7f6fd00a8130 con 0x7f6fd00a67c0 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): my global_id is 214205 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd006ede0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991553560 0 0) 0x7f6fd00a3b40 con 0x7f6fd00a6ba0 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient(hunting): my global_id is 214203 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd00a8130 con 0x7f6fd00a6ba0 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2050556436 0 0) 0x7f6fbc003520 con 0x7f6fd013f5c0 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 >> v1:172.21.15.82:6791/0 conn(0x7f6fd00a67c0 legacy=0x7f6fd00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 >> v1:172.21.15.82:6790/0 conn(0x7f6fd00a6ba0 legacy=0x7f6fd00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 monclient: found mon.noname-e 2024-08-19T10:07:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd00a92a0 con 0x7f6fd013f5c0 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8001e50 con 0x7f6fd013f5c0 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 5 monclient: authenticate success, global_id 204497 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: dump: 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fc8004c20 con 0x7f6fd013f5c0 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 1 -- 172.21.15.1:0/3485704632 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc8005190 con 0x7f6fd013f5c0 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 4 set_mon_vals no callback set 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd5324640 10 monclient: dump: 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 10 monclient: shutdown 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.804+0000 7f6fd858a640 1 -- 172.21.15.1:0/3485704632 >> v1:172.21.15.1:6793/0 conn(0x7f6fd013f5c0 legacy=0x7f6fd01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 -- 172.21.15.1:0/3485704632 shutdown_connections 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 -- 172.21.15.1:0/3485704632 >> 172.21.15.1:0/3485704632 conn(0x7f6fd00ac4b0 msgr2=0x7f6fd00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 -- 172.21.15.1:0/3485704632 shutdown_connections 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 -- 172.21.15.1:0/3485704632 wait complete. 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 10 monclient: monmap: 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 librados: starting msgr at 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 librados: starting objecter 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.805+0000 7f6fd858a640 1 Processor -- start 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 1 -- start start 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 1 librados: setting wanted keys 2024-08-19T10:07:47.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 1 librados: calling monclient init 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient: init 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient: _add_conns ranks=[2,3,7,0,4,6,5,1,8] 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd00a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient(hunting): picked mon.noname-g con 0x7f6fd00a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient(hunting): picked mon.noname-i con 0x7f6fd013f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd013d0f0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd013bdd0 con 0x7f6fd00a6ba0 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd0139a70 con 0x7f6fd013f5c0 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd858a640 10 monclient(hunting): authenticate will time out at 1723.882524s 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd6326640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f6fd00a67c0 0x7f6fd013c2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51938/0 (socket says 172.21.15.1:51938) 2024-08-19T10:07:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.806+0000 7f6fd6326640 1 -- 172.21.15.1:0/202664479 learned_addr learned my addr 172.21.15.1:0/202664479 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fd6326640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6790/0 conn(0x7f6fd00a67c0 legacy=0x7f6fd013c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd00a67c0 2024-08-19T10:07:47.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd00a6ba0 2024-08-19T10:07:47.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd013f5c0 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fd5b25640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.82:6790/0 conn(0x7f6fd00a6ba0 legacy=0x7f6fd013c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fd6b27640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.82:6792/0 conn(0x7f6fd013f5c0 legacy=0x7f6fd01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183304409 0 0) 0x7f6fd013d0f0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 184817 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.808+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fb4004ee0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2260230751 0 0) 0x7f6fd013bdd0 con 0x7f6fd00a6ba0 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 214212 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd013d0f0 con 0x7f6fd00a6ba0 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165889423 0 0) 0x7f6fd0139a70 con 0x7f6fd013f5c0 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 204433 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd013bdd0 con 0x7f6fd013f5c0 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3980900323 0 0) 0x7f6fb4004ee0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd0139a70 con 0x7f6fd00a67c0 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fcc003610 con 0x7f6fd00a67c0 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 727566086 0 0) 0x7f6fd0139a70 con 0x7f6fd00a67c0 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.82:6792/0 conn(0x7f6fd013f5c0 legacy=0x7f6fd01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.82:6790/0 conn(0x7f6fd00a6ba0 legacy=0x7f6fd013c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 monclient: found mon.b 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd013baf0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 5 monclient: authenticate success, global_id 184817 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 10 monclient: _renew_subs 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd013a9a0 con 0x7f6fd00a67c0 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 10 client.184817.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 10 monclient: _renew_subs 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6fd01d7b80 con 0x7f6fd00a67c0 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd858a640 1 librados: init done 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fcc003c50 con 0x7f6fd00a67c0 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fcc004180 con 0x7f6fd00a67c0 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fd4b23640 4 set_mon_vals no callback set 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.809+0000 7f6fc6ffd640 10 monclient: dump: 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fd858a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fb80065c0 con 0x7f6fb8002860 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6fcc004320 con 0x7f6fd00a67c0 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(656..656 src has 299..656) ==== 5979+0+0 (unknown 2284910107 0 0) 0x7f6fcc04e500 con 0x7f6fd00a67c0 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fc6ffd640 10 client.184817.objecter ms_dispatch 0x7f6fd0000b90 osd_map(656..656 src has 299..656) 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fc6ffd640 3 client.184817.objecter handle_osd_map got epochs [656,656] > 0 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fc6ffd640 3 client.184817.objecter handle_osd_map decoding full epoch 656 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fc6ffd640 20 client.184817.objecter dump_active .. 0 homeless 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.810+0000 7f6fd6b27640 10 monclient: get_auth_request con 0x7f6fb4040fb0 auth_method 0 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fd5b25640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6791/0 conn(0x7f6fb8002860 legacy=0x7f6fb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 client.184817.objecter ms_handle_connect 0x7f6fb8002860 2024-08-19T10:07:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 client.184817.objecter resend_mon_ops 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541792709 0 0) 0x7f6fb80065c0 con 0x7f6fb8002860 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 194629 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd0139a70 con 0x7f6fb8002860 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 817339537 0 0) 0x7f6fd0139a70 con 0x7f6fb8002860 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fb80065c0 con 0x7f6fb8002860 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc0003e50 con 0x7f6fb8002860 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 10 monclient: dump: 2024-08-19T10:07:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.836 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 940828555 0 0) 0x7f6fb80065c0 con 0x7f6fb8002860 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.812+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fb800a160 con 0x7f6fb8002860 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.813+0000 7f6fc6ffd640 10 client.184817.objecter ms_handle_connect 0x7f6fb4040fb0 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.813+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6fc0004190 con 0x7f6fb8002860 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.813+0000 7f6fc6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.813+0000 7f6fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.813+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6791/0 conn(0x7f6fb8002860 legacy=0x7f6fb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.813+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6791/0 conn(0x7f6fb8002860 legacy=0x7f6fb8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.819+0000 7f6fd858a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.819+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fb8009e50 con 0x7f6fb8009110 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fd5b25640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6791/0 conn(0x7f6fb8009110 legacy=0x7f6fb80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 10 client.184817.objecter ms_handle_connect 0x7f6fb8009110 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 10 client.184817.objecter resend_mon_ops 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 51233237 0 0) 0x7f6fb8009e50 con 0x7f6fb8009110 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 10 monclient(hunting): my global_id is 194638 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.821+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fb80065c0 con 0x7f6fb8009110 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1281158207 0 0) 0x7f6fb80065c0 con 0x7f6fb8009110 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fb8009e50 con 0x7f6fb8009110 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6fc0005b00 con 0x7f6fb8009110 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 10 monclient: dump: 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.839 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.840 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.840 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.840 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.840 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.840 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3208879702 0 0) 0x7f6fb8009e50 con 0x7f6fb8009110 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fb800c3a0 con 0x7f6fb8009110 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3622785405) 0x7f6fc0005e10 con 0x7f6fb8009110 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6791/0 conn(0x7f6fb8009110 legacy=0x7f6fb80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fc6ffd640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6791/0 conn(0x7f6fb8009110 legacy=0x7f6fb80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:47.841 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 255509, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:47.842 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 6, 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.843 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.844 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:47.845 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:47.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:47.849 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:47.850 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.851 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:47.854 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.855 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:47.859 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:47.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:47.862 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 10 librados: watch_flush enter 2024-08-19T10:07:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 10 librados: watch_flush exit 2024-08-19T10:07:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 20 client.184817.objecter shutdown clearing up homeless session... 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 10 client.184817.objecter successfully canceled tick 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6fb4040fb0 legacy=0x7f6fb4043470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 10 monclient: shutdown 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 >> v1:172.21.15.1:6790/0 conn(0x7f6fd00a67c0 legacy=0x7f6fd013c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd6b27640 1 -- 172.21.15.1:0/202664479 reap_dead start 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.822+0000 7f6fd6b27640 1 -- 172.21.15.1:0/202664479 reap_dead start 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.823+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 shutdown_connections 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.823+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 >> 172.21.15.1:0/202664479 conn(0x7f6fd00ac4b0 msgr2=0x7f6fd0141e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.823+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 shutdown_connections 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.823+0000 7f6fd858a640 1 -- 172.21.15.1:0/202664479 wait complete. 2024-08-19T10:07:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.823+0000 7f6fd858a640 1 librados: shutdown 2024-08-19T10:07:47.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:47.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:07:47.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:07:47.932 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434391245}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475094309}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4199226415}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2781365932}) 2024-08-19T10:07:47.933 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2216738125}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3570197667}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1864969658}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263930650}) 2024-08-19T10:07:47.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642615305}) 2024-08-19T10:07:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.965+0000 7f71eaf41640 10 monclient: get_monmap_and_config 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.965+0000 7f71eaf41640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.965+0000 7f71eaf41640 10 monclient: monmap: 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:47.966447+0000 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:47.966447+0000 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.968 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 1 Processor -- start 2024-08-19T10:07:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 1 -- start start 2024-08-19T10:07:47.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient: init 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient: _add_conns ranks=[0,4,2,5,7,3,1,6,8] 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient(hunting): picked mon.noname-a con 0x7f71e40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient(hunting): picked mon.noname-c con 0x7f71e40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient(hunting): picked mon.noname-b con 0x7f71e413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e40a3b40 con 0x7f71e40a67c0 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e40a8130 con 0x7f71e413f5c0 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e406ede0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.966+0000 7f71eaf41640 10 monclient(hunting): authenticate will time out at 1724.042968s 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.967+0000 7f71e3fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f71e40a6ba0 0x7f71e40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48606/0 (socket says 172.21.15.1:48606) 2024-08-19T10:07:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.967+0000 7f71e8cdd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f71e40a67c0 0x7f71e40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41332/0 (socket says 172.21.15.1:41332) 2024-08-19T10:07:47.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.967+0000 7f71e3fff640 1 -- 172.21.15.1:0/2577606304 learned_addr learned my addr 172.21.15.1:0/2577606304 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e3fff640 1 -- 172.21.15.1:0/2577606304 >> v1:172.21.15.1:6791/0 conn(0x7f71e40a6ba0 legacy=0x7f71e40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e94de640 1 -- 172.21.15.1:0/2577606304 >> v1:172.21.15.1:6790/0 conn(0x7f71e413f5c0 legacy=0x7f71e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e8cdd640 1 -- 172.21.15.1:0/2577606304 >> v1:172.21.15.1:6789/0 conn(0x7f71e40a67c0 legacy=0x7f71e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522861881 0 0) 0x7f71e406ede0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): my global_id is 194647 2024-08-19T10:07:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71cc003520 con 0x7f71e40a6ba0 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 678437578 0 0) 0x7f71e40a8130 con 0x7f71e413f5c0 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): my global_id is 184826 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e406ede0 con 0x7f71e413f5c0 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2542537966 0 0) 0x7f71e40a3b40 con 0x7f71e40a67c0 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient(hunting): my global_id is 204453 2024-08-19T10:07:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e40a8130 con 0x7f71e40a67c0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1249221198 0 0) 0x7f71cc003520 con 0x7f71e40a6ba0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 >> v1:172.21.15.1:6790/0 conn(0x7f71e413f5c0 legacy=0x7f71e41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 >> v1:172.21.15.1:6789/0 conn(0x7f71e40a67c0 legacy=0x7f71e40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 monclient: found mon.noname-c 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e40a92a0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d8001ad0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: dump: 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d4001e00 con 0x7f71e40a67c0 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71eaf41640 5 monclient: authenticate success, global_id 194647 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71eaf41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71d80048a0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 1 -- 172.21.15.1:0/2577606304 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d8004dd0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71e37fe640 10 monclient: dump: 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71eaf41640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.969+0000 7f71eaf41640 4 set_mon_vals no callback set 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 10 monclient: shutdown 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 1 -- 172.21.15.1:0/2577606304 >> v1:172.21.15.1:6791/0 conn(0x7f71e40a6ba0 legacy=0x7f71e40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 1 -- 172.21.15.1:0/2577606304 shutdown_connections 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 1 -- 172.21.15.1:0/2577606304 >> 172.21.15.1:0/2577606304 conn(0x7f71e40ac4b0 msgr2=0x7f71e40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 1 -- 172.21.15.1:0/2577606304 shutdown_connections 2024-08-19T10:07:47.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 1 -- 172.21.15.1:0/2577606304 wait complete. 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 10 monclient: build_initial_monmap 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.970+0000 7f71eaf41640 10 monclient: monmap: 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 librados: starting msgr at 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 librados: starting objecter 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 Processor -- start 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 -- start start 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 librados: setting wanted keys 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 librados: calling monclient init 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient: init 2024-08-19T10:07:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient: _add_conns ranks=[0,4,1,3,7,6,8,5,2] 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient(hunting): picked mon.noname-a con 0x7f71e40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient(hunting): picked mon.noname-c con 0x7f71e40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient(hunting): picked mon.noname-f con 0x7f71e413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e40a1730 con 0x7f71e40a67c0 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e40aa080 con 0x7f71e413f5c0 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e40a9cc0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.971+0000 7f71eaf41640 10 monclient(hunting): authenticate will time out at 1724.047970s 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.972+0000 7f71e8cdd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f71e40a67c0 0x7f71e409fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41346/0 (socket says 172.21.15.1:41346) 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.972+0000 7f71e8cdd640 1 -- 172.21.15.1:0/1629091617 learned_addr learned my addr 172.21.15.1:0/1629091617 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.972+0000 7f71e3fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f71e40a6ba0 0x7f71e40a03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48608/0 (socket says 172.21.15.1:48608) 2024-08-19T10:07:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e8cdd640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6789/0 conn(0x7f71e40a67c0 legacy=0x7f71e409fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 client.?.objecter ms_handle_connect 0x7f71e40a67c0 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e3fff640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6791/0 conn(0x7f71e40a6ba0 legacy=0x7f71e40a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 client.?.objecter ms_handle_connect 0x7f71e40a6ba0 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744995383 0 0) 0x7f71e40a9cc0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): my global_id is 194656 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d0004eb0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296996330 0 0) 0x7f71e40a1730 con 0x7f71e40a67c0 2024-08-19T10:07:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): my global_id is 204462 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e40a9cc0 con 0x7f71e40a67c0 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 client.?.objecter ms_handle_connect 0x7f71e413f5c0 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e94de640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.82:6789/0 conn(0x7f71e413f5c0 legacy=0x7f71e40a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2983331484 0 0) 0x7f71d0004eb0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71e40a1730 con 0x7f71e40a6ba0 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d8004d90 con 0x7f71e40a6ba0 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:07:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): dump: 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996245688 0 0) 0x7f71e40aa080 con 0x7f71e413f5c0 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): my global_id is 175159 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d0004eb0 con 0x7f71e413f5c0 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 65146231 0 0) 0x7f71e40a9cc0 con 0x7f71e40a67c0 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71e40aa080 con 0x7f71e40a67c0 2024-08-19T10:07:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d40016a0 con 0x7f71e40a67c0 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient(hunting): dump: 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1488116253 0 0) 0x7f71e40a1730 con 0x7f71e40a6ba0 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.82:6789/0 conn(0x7f71e413f5c0 legacy=0x7f71e40a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6789/0 conn(0x7f71e40a67c0 legacy=0x7f71e409fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient: _finish_hunting 0 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 monclient: found mon.c 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:47.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e413a800 con 0x7f71e40a6ba0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient: _finish_auth 0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 10 monclient: _check_auth_tickets 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71eaf41640 5 monclient: authenticate success, global_id 194656 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71eaf41640 10 monclient: _renew_subs 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71eaf41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.974+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71e40a2750 con 0x7f71e40a6ba0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 10 client.194656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 10 monclient: _renew_subs 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f71e4139b30 con 0x7f71e40a6ba0 2024-08-19T10:07:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 1 librados: init done 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71d8001d20 con 0x7f71e40a6ba0 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d8001a70 con 0x7f71e40a6ba0 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e2ffd640 4 set_mon_vals no callback set 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 10 monclient: dump: 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71c40065c0 con 0x7f71c4002860 2024-08-19T10:07:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.975+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f71d80119c0 con 0x7f71e40a6ba0 2024-08-19T10:07:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.976+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(656..656 src has 299..656) ==== 5979+0+0 (unknown 2284910107 0 0) 0x7f71d804e340 con 0x7f71e40a6ba0 2024-08-19T10:07:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.976+0000 7f71e17fa640 10 client.194656.objecter ms_dispatch 0x7f71e4000b90 osd_map(656..656 src has 299..656) 2024-08-19T10:07:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.976+0000 7f71e17fa640 3 client.194656.objecter handle_osd_map got epochs [656,656] > 0 2024-08-19T10:07:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.976+0000 7f71e17fa640 3 client.194656.objecter handle_osd_map decoding full epoch 656 2024-08-19T10:07:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.976+0000 7f71e17fa640 20 client.194656.objecter dump_active .. 0 homeless 2024-08-19T10:07:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.976+0000 7f71e94de640 10 monclient: get_auth_request con 0x7f71d00418b0 auth_method 0 2024-08-19T10:07:47.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 client.194656.objecter ms_handle_connect 0x7f71c4002860 2024-08-19T10:07:47.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e8cdd640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6792/0 conn(0x7f71c4002860 legacy=0x7f71c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 client.194656.objecter resend_mon_ops 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157526289 0 0) 0x7f71c40065c0 con 0x7f71c4002860 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient(hunting): my global_id is 204387 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e40a1730 con 0x7f71c4002860 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1922034420 0 0) 0x7f71e40a1730 con 0x7f71c4002860 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71c40065c0 con 0x7f71c4002860 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d40053d0 con 0x7f71c4002860 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient: dump: 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.000 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 443308633 0 0) 0x7f71c40065c0 con 0x7f71c4002860 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71c400a160 con 0x7f71c4002860 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 client.194656.objecter ms_handle_connect 0x7f71d00418b0 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f71d40075f0 con 0x7f71c4002860 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6792/0 conn(0x7f71c4002860 legacy=0x7f71c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.978+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6792/0 conn(0x7f71c4002860 legacy=0x7f71c4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1682853693}) 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732470741}) 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=227172160}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2580857932}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=127078434}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3582197131}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2874942186}) 2024-08-19T10:07:48.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753647958}) 2024-08-19T10:07:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1030643458}) 2024-08-19T10:07:48.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.986+0000 7f71eaf41640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:48.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.986+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71c4009e70 con 0x7f71c4009130 2024-08-19T10:07:48.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.988+0000 7f71e8cdd640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6792/0 conn(0x7f71c4009130 legacy=0x7f71c40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.988+0000 7f71e17fa640 10 client.194656.objecter ms_handle_connect 0x7f71c4009130 2024-08-19T10:07:48.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:47.988+0000 7f71e17fa640 10 client.194656.objecter resend_mon_ops 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3224108334 0 0) 0x7f71c4009e70 con 0x7f71c4009130 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient(hunting): my global_id is 204396 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c40065c0 con 0x7f71c4009130 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 556546639 0 0) 0x7f71c40065c0 con 0x7f71c4009130 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71c4009e70 con 0x7f71c4009130 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f71d4006e60 con 0x7f71c4009130 2024-08-19T10:07:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient: dump: 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3065688452 0 0) 0x7f71c4009e70 con 0x7f71c4009130 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71c400c410 con 0x7f71c4009130 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 2217743952) 0x7f71d4007270 con 0x7f71c4009130 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.008+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6792/0 conn(0x7f71c4009130 legacy=0x7f71c40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71e17fa640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6792/0 conn(0x7f71c4009130 legacy=0x7f71c40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 133879, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:48.013 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:48.015 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.016 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:48.017 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.018 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.021 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:48.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.024 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:48.033 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:48.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 10 librados: watch_flush enter 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 10 librados: watch_flush exit 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 20 client.194656.objecter shutdown clearing up homeless session... 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 10 client.194656.objecter successfully canceled tick 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.82:6812/2549104415 conn(0x7f71d00418b0 legacy=0x7f71d0043d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 10 monclient: shutdown 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 >> v1:172.21.15.1:6791/0 conn(0x7f71e40a6ba0 legacy=0x7f71e40a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.034 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71e94de640 1 -- 172.21.15.1:0/1629091617 reap_dead start 2024-08-19T10:07:48.035 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71e94de640 1 -- 172.21.15.1:0/1629091617 reap_dead start 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4080894814}) 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943378555}) 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.036 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1233782543}) 2024-08-19T10:07:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 shutdown_connections 2024-08-19T10:07:48.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.037 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 >> 172.21.15.1:0/1629091617 conn(0x7f71e40ac4b0 msgr2=0x7f71e40a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2543613188}) 2024-08-19T10:07:48.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1821210034}) 2024-08-19T10:07:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 shutdown_connections 2024-08-19T10:07:48.038 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f71eaf41640 1 -- 172.21.15.1:0/1629091617 wait complete. 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1508287506}) 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=884461546}) 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227574034}) 2024-08-19T10:07:48.040 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.009+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3560651244}) 2024-08-19T10:07:48.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.010+0000 7f71eaf41640 1 librados: shutdown 2024-08-19T10:07:48.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531363031}) 2024-08-19T10:07:48.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.045 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2377386386}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=718908866}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2401224530}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4039194813}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.046 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=468145655}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1027888776}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660017755}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3420613114}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799435211}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608368341}) 2024-08-19T10:07:48.047 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1152568459}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=942042443}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2549614432}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=679805294}) 2024-08-19T10:07:48.048 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2584918767}) 2024-08-19T10:07:48.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523441894}) 2024-08-19T10:07:48.049 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1928672662}) 2024-08-19T10:07:48.050 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:48.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:07:48.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:07:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.152+0000 7f93ecaff640 10 monclient: get_monmap_and_config 2024-08-19T10:07:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.152+0000 7f93ecaff640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.152+0000 7f93ecaff640 10 monclient: monmap: 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:48.153477+0000 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:48.153477+0000 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.155 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.155 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.155 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.155 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.155 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 1 Processor -- start 2024-08-19T10:07:48.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 1 -- start start 2024-08-19T10:07:48.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient: init 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient: _add_conns ranks=[1,3,5,6,4,8,2,7,0] 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient(hunting): picked mon.noname-f con 0x7f93e80a6180 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient(hunting): picked mon.noname-g con 0x7f93e80a5cd0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient(hunting): picked mon.noname-h con 0x7f93e813f840 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e813a200 con 0x7f93e80a6180 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e80a28e0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e806f520 con 0x7f93e813f840 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.153+0000 7f93ecaff640 10 monclient(hunting): authenticate will time out at 1724.229825s 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.154+0000 7f93e6d9d640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f93e813f840 0x7f93e8141c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:34466/0 (socket says 172.21.15.1:34466) 2024-08-19T10:07:48.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.154+0000 7f93e6d9d640 1 -- 172.21.15.1:0/4030229876 learned_addr learned my addr 172.21.15.1:0/4030229876 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e6d9d640 1 -- 172.21.15.1:0/4030229876 >> v1:172.21.15.82:6791/0 conn(0x7f93e813f840 legacy=0x7f93e8141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e5d9b640 1 -- 172.21.15.1:0/4030229876 >> v1:172.21.15.82:6790/0 conn(0x7f93e80a5cd0 legacy=0x7f93e8139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e659c640 1 -- 172.21.15.1:0/4030229876 >> v1:172.21.15.82:6789/0 conn(0x7f93e80a6180 legacy=0x7f93e81393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1636568677 0 0) 0x7f93e806f520 con 0x7f93e813f840 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): my global_id is 214214 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93d0003520 con 0x7f93e813f840 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3026441473 0 0) 0x7f93e80a28e0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): my global_id is 214221 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e806f520 con 0x7f93e80a5cd0 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447112965 0 0) 0x7f93e813a200 con 0x7f93e80a6180 2024-08-19T10:07:48.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient(hunting): my global_id is 175168 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e80a28e0 con 0x7f93e80a6180 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4148876143 0 0) 0x7f93d0003520 con 0x7f93e813f840 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 >> v1:172.21.15.82:6790/0 conn(0x7f93e80a5cd0 legacy=0x7f93e8139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 >> v1:172.21.15.82:6789/0 conn(0x7f93e80a6180 legacy=0x7f93e81393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 monclient: found mon.noname-h 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e806e9c0 con 0x7f93e813f840 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93e0001f20 con 0x7f93e813f840 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93ecaff640 5 monclient: authenticate success, global_id 214214 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93ecaff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: dump: 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.164 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93ecaff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93e0004c20 con 0x7f93e813f840 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93ecaff640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 1 -- 172.21.15.1:0/4030229876 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93e0005190 con 0x7f93e813f840 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93ecaff640 4 set_mon_vals no callback set 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.156+0000 7f93e559a640 10 monclient: dump: 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 10 monclient: shutdown 2024-08-19T10:07:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 1 -- 172.21.15.1:0/4030229876 >> v1:172.21.15.82:6791/0 conn(0x7f93e813f840 legacy=0x7f93e8141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 1 -- 172.21.15.1:0/4030229876 shutdown_connections 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 1 -- 172.21.15.1:0/4030229876 >> 172.21.15.1:0/4030229876 conn(0x7f93e80a6700 msgr2=0x7f93e80a6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 1 -- 172.21.15.1:0/4030229876 shutdown_connections 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.157+0000 7f93ecaff640 1 -- 172.21.15.1:0/4030229876 wait complete. 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient: monmap: 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 librados: starting msgr at 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 librados: starting objecter 2024-08-19T10:07:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 Processor -- start 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 -- start start 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 librados: setting wanted keys 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 librados: calling monclient init 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient: init 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient: _add_conns ranks=[6,1,4,0,5,3,2,8,7] 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient(hunting): picked mon.noname-d con 0x7f93e80a5cd0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient(hunting): picked mon.noname-f con 0x7f93e80a6180 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 10 monclient(hunting): picked mon.noname-c con 0x7f93e813f840 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.158+0000 7f93ecaff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e80a7b90 con 0x7f93e80a6180 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.159+0000 7f93ecaff640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e8142e50 con 0x7f93e813f840 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.159+0000 7f93ecaff640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e813d7a0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.159+0000 7f93ecaff640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.159+0000 7f93ecaff640 10 monclient(hunting): authenticate will time out at 1724.235020s 2024-08-19T10:07:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.159+0000 7f93e659c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f93e80a5cd0 0x7f93e813bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45676/0 (socket says 172.21.15.1:45676) 2024-08-19T10:07:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.159+0000 7f93e659c640 1 -- 172.21.15.1:0/4096894029 learned_addr learned my addr 172.21.15.1:0/4096894029 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93e659c640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6792/0 conn(0x7f93e80a5cd0 legacy=0x7f93e813bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93e6d9d640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6791/0 conn(0x7f93e813f840 legacy=0x7f93e81d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 client.?.objecter ms_handle_connect 0x7f93e80a5cd0 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 client.?.objecter ms_handle_connect 0x7f93e813f840 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 client.?.objecter ms_handle_connect 0x7f93e80a6180 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93e5d9b640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.82:6789/0 conn(0x7f93e80a6180 legacy=0x7f93e813c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768059837 0 0) 0x7f93e813d7a0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 204405 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93c8004ee0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4194909387 0 0) 0x7f93e8142e50 con 0x7f93e813f840 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 194665 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e813d7a0 con 0x7f93e813f840 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242941054 0 0) 0x7f93dc0015d0 con 0x7f93e80a6180 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 175177 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e8142e50 con 0x7f93e80a6180 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2676483100 0 0) 0x7f93c8004ee0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93dc0015d0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93d8003610 con 0x7f93e80a5cd0 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1752296913 0 0) 0x7f93e8142e50 con 0x7f93e80a6180 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93c8004ee0 con 0x7f93e80a6180 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93dc003950 con 0x7f93e80a6180 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.182 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2121417534 0 0) 0x7f93e813d7a0 con 0x7f93e813f840 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93e8142e50 con 0x7f93e813f840 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93e0001ee0 con 0x7f93e813f840 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1807249628 0 0) 0x7f93dc0015d0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6791/0 conn(0x7f93e813f840 legacy=0x7f93e81d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.82:6789/0 conn(0x7f93e80a6180 legacy=0x7f93e813c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 monclient: found mon.d 2024-08-19T10:07:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e8121530 con 0x7f93e80a5cd0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93ecaff640 5 monclient: authenticate success, global_id 204405 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93ecaff640 10 monclient: _renew_subs 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93ecaff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.161+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93e81206a0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 10 client.204405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 10 monclient: _renew_subs 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f93e813cde0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 1 librados: init done 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93d8003ae0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93d6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93d8003fd0 con 0x7f93e80a5cd0 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93d6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93d6ffd640 10 monclient: dump: 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93e4d99640 4 set_mon_vals no callback set 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.162+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f93e8051b00 con 0x7f93e813fc20 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f93d8004170 con 0x7f93e80a5cd0 2024-08-19T10:07:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7f93d804e400 con 0x7f93e80a5cd0 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93d6ffd640 10 client.204405.objecter ms_dispatch 0x7f93e8000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93d6ffd640 3 client.204405.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93d6ffd640 3 client.204405.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93d6ffd640 20 client.204405.objecter dump_active .. 0 homeless 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.163+0000 7f93e6d9d640 10 monclient: get_auth_request con 0x7f93c80419e0 auth_method 0 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93e5d9b640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6793/0 conn(0x7f93e813fc20 legacy=0x7f93e8140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 client.204405.objecter ms_handle_connect 0x7f93e813fc20 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 client.204405.objecter resend_mon_ops 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147939850 0 0) 0x7f93e8051b00 con 0x7f93e813fc20 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 204506 2024-08-19T10:07:48.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93dc0015d0 con 0x7f93e813fc20 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3444936429 0 0) 0x7f93dc0015d0 con 0x7f93e813fc20 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93e8051b00 con 0x7f93e813fc20 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93dc0040e0 con 0x7f93e813fc20 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient: dump: 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 804496288 0 0) 0x7f93e8051b00 con 0x7f93e813fc20 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93e813d0d0 con 0x7f93e813fc20 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 client.204405.objecter ms_handle_connect 0x7f93c80419e0 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f93dc004400 con 0x7f93e813fc20 2024-08-19T10:07:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6793/0 conn(0x7f93e813fc20 legacy=0x7f93e8140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.165+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6793/0 conn(0x7f93e813fc20 legacy=0x7f93e8140000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.172+0000 7f93ecaff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.172+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f93e8068ef0 con 0x7f93e8140710 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93e5d9b640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6793/0 conn(0x7f93e8140710 legacy=0x7f93e81dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 client.204405.objecter ms_handle_connect 0x7f93e8140710 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 client.204405.objecter resend_mon_ops 2024-08-19T10:07:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941365226 0 0) 0x7f93e8068ef0 con 0x7f93e8140710 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 204515 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e8051b00 con 0x7f93e8140710 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 605207303 0 0) 0x7f93e8051b00 con 0x7f93e8140710 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93e8068ef0 con 0x7f93e8140710 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f93dc005d70 con 0x7f93e8140710 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 10 monclient: dump: 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.193 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 493692571 0 0) 0x7f93e8068ef0 con 0x7f93e8140710 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.174+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93e8140af0 con 0x7f93e8140710 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1266323117) 0x7f93dc008050 con 0x7f93e8140710 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93d6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93d6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6793/0 conn(0x7f93e8140710 legacy=0x7f93e81dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93d6ffd640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6793/0 conn(0x7f93e8140710 legacy=0x7f93e81dbbe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.194 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 134047, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:48.195 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.196 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.197 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:48.198 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:48.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.201 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.205 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.208 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.209 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.212 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.213 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.214 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:48.214 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:48.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909893363}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216209087}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1974009481}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2919661471}) 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=837926561}) 2024-08-19T10:07:48.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3405291487}) 2024-08-19T10:07:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 10 librados: watch_flush enter 2024-08-19T10:07:48.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1164158594}) 2024-08-19T10:07:48.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 10 librados: watch_flush exit 2024-08-19T10:07:48.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 20 client.204405.objecter shutdown clearing up homeless session... 2024-08-19T10:07:48.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 10 client.204405.objecter successfully canceled tick 2024-08-19T10:07:48.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336974229}) 2024-08-19T10:07:48.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1553421964}) 2024-08-19T10:07:48.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.82:6812/2549104415 conn(0x7f93c80419e0 legacy=0x7f93c8043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 10 monclient: shutdown 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 >> v1:172.21.15.1:6792/0 conn(0x7f93e80a5cd0 legacy=0x7f93e813bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93e6d9d640 1 -- 172.21.15.1:0/4096894029 reap_dead start 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93e6d9d640 1 -- 172.21.15.1:0/4096894029 reap_dead start 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.175+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 shutdown_connections 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.176+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 >> 172.21.15.1:0/4096894029 conn(0x7f93e80a6700 msgr2=0x7f93e8072d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.176+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 shutdown_connections 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.176+0000 7f93ecaff640 1 -- 172.21.15.1:0/4096894029 wait complete. 2024-08-19T10:07:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.176+0000 7f93ecaff640 1 librados: shutdown 2024-08-19T10:07:48.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:48.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:07:48.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.321+0000 7f8dfb7de640 10 monclient: get_monmap_and_config 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.321+0000 7f8dfb7de640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.321+0000 7f8dfb7de640 10 monclient: monmap: 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:48.322027+0000 2024-08-19T10:07:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:48.322027+0000 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.323 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.321+0000 7f8dfb7de640 1 Processor -- start 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.321+0000 7f8dfb7de640 1 -- start start 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.321+0000 7f8dfb7de640 10 monclient: init 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient: _add_conns ranks=[2,1,7,6,0,3,8,4,5] 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient(hunting): picked mon.noname-b con 0x7f8df40a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient(hunting): picked mon.noname-f con 0x7f8df40694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient(hunting): picked mon.noname-i con 0x7f8df40a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df40a3c70 con 0x7f8df40694b0 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df40a33d0 con 0x7f8df40a6830 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df40a36b0 con 0x7f8df40a46a0 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8dfb7de640 10 monclient(hunting): authenticate will time out at 1724.398374s 2024-08-19T10:07:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8df957a640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f8df40a6830 0x7f8df40a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51956/0 (socket says 172.21.15.1:51956) 2024-08-19T10:07:48.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.322+0000 7f8df957a640 1 -- 172.21.15.1:0/2480685873 learned_addr learned my addr 172.21.15.1:0/2480685873 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df957a640 1 -- 172.21.15.1:0/2480685873 >> v1:172.21.15.1:6790/0 conn(0x7f8df40a6830 legacy=0x7f8df40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df9d7b640 1 -- 172.21.15.1:0/2480685873 >> v1:172.21.15.82:6792/0 conn(0x7f8df40a46a0 legacy=0x7f8df4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df8d79640 1 -- 172.21.15.1:0/2480685873 >> v1:172.21.15.82:6789/0 conn(0x7f8df40694b0 legacy=0x7f8df40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2814802943 0 0) 0x7f8df40a33d0 con 0x7f8df40a6830 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): my global_id is 184835 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc003520 con 0x7f8df40a6830 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039209370 0 0) 0x7f8df40a36b0 con 0x7f8df40a46a0 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): my global_id is 204442 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df40a33d0 con 0x7f8df40a46a0 2024-08-19T10:07:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345827313 0 0) 0x7f8df40a3c70 con 0x7f8df40694b0 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 10 monclient(hunting): my global_id is 175186 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.324+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df40a36b0 con 0x7f8df40694b0 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2185267577 0 0) 0x7f8ddc003520 con 0x7f8df40a6830 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 >> v1:172.21.15.82:6792/0 conn(0x7f8df40a46a0 legacy=0x7f8df4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 >> v1:172.21.15.82:6789/0 conn(0x7f8df40694b0 legacy=0x7f8df40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 monclient: found mon.noname-b 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:07:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df406dbb0 con 0x7f8df40a6830 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de4001e60 con 0x7f8df40a6830 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 5 monclient: authenticate success, global_id 184835 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: dump: 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de4004c20 con 0x7f8df40a6830 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 1 -- 172.21.15.1:0/2480685873 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de4005190 con 0x7f8df40a6830 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 4 set_mon_vals no callback set 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8df3fff640 10 monclient: dump: 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 10 monclient: shutdown 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 1 -- 172.21.15.1:0/2480685873 >> v1:172.21.15.1:6790/0 conn(0x7f8df40a6830 legacy=0x7f8df40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 1 -- 172.21.15.1:0/2480685873 shutdown_connections 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 1 -- 172.21.15.1:0/2480685873 >> 172.21.15.1:0/2480685873 conn(0x7f8df40a9020 msgr2=0x7f8df4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 1 -- 172.21.15.1:0/2480685873 shutdown_connections 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.325+0000 7f8dfb7de640 1 -- 172.21.15.1:0/2480685873 wait complete. 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 10 monclient: monmap: 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 1 librados: starting msgr at 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 1 librados: starting objecter 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 1 Processor -- start 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 1 -- start start 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 1 librados: setting wanted keys 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 1 librados: calling monclient init 2024-08-19T10:07:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.326+0000 7f8dfb7de640 10 monclient: init 2024-08-19T10:07:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient: _add_conns ranks=[4,0,5,2,3,7,1,8,6] 2024-08-19T10:07:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient(hunting): picked mon.noname-c con 0x7f8df40694b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient(hunting): picked mon.noname-a con 0x7f8df40a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient(hunting): picked mon.noname-h con 0x7f8df40a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df40a9ec0 con 0x7f8df40a46a0 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df4138bb0 con 0x7f8df40694b0 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df4138e90 con 0x7f8df40a6830 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8dfb7de640 10 monclient(hunting): authenticate will time out at 1724.403365s 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8df957a640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f8df40694b0 0x7f8df41419f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48616/0 (socket says 172.21.15.1:48616) 2024-08-19T10:07:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8df8d79640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8df40a46a0 0x7f8df41d3880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41350/0 (socket says 172.21.15.1:41350) 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.327+0000 7f8df957a640 1 -- 172.21.15.1:0/141768963 learned_addr learned my addr 172.21.15.1:0/141768963 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df957a640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.1:6791/0 conn(0x7f8df40694b0 legacy=0x7f8df41419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 client.?.objecter ms_handle_connect 0x7f8df40694b0 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 client.?.objecter ms_handle_connect 0x7f8df40a46a0 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df8d79640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.1:6789/0 conn(0x7f8df40a46a0 legacy=0x7f8df41d3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df9d7b640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6791/0 conn(0x7f8df40a6830 legacy=0x7f8df41d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 client.?.objecter ms_handle_connect 0x7f8df40a6830 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870152773 0 0) 0x7f8df4138bb0 con 0x7f8df40694b0 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): my global_id is 194674 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8de0004eb0 con 0x7f8df40694b0 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506507283 0 0) 0x7f8df40a9ec0 con 0x7f8df40a46a0 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): my global_id is 204471 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df4138bb0 con 0x7f8df40a46a0 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867223487 0 0) 0x7f8df4138e90 con 0x7f8df40a6830 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 10 monclient(hunting): my global_id is 214223 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.329+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df40a9ec0 con 0x7f8df40a6830 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1468885204 0 0) 0x7f8df4138bb0 con 0x7f8df40a46a0 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df4138e90 con 0x7f8df40a46a0 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2527452903 0 0) 0x7f8de0004eb0 con 0x7f8df40694b0 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df4138bb0 con 0x7f8df40694b0 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de8001c80 con 0x7f8df40a46a0 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1333390896 0 0) 0x7f8df40a9ec0 con 0x7f8df40a6830 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8de0004eb0 con 0x7f8df40a6830 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de40020b0 con 0x7f8df40694b0 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8dec003610 con 0x7f8df40a6830 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient(hunting): dump: 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2667374740 0 0) 0x7f8df4138e90 con 0x7f8df40a46a0 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6791/0 conn(0x7f8df40a6830 legacy=0x7f8df41d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.1:6791/0 conn(0x7f8df40694b0 legacy=0x7f8df41419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 monclient: found mon.a 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df4137c50 con 0x7f8df40a46a0 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 5 monclient: authenticate success, global_id 204471 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 10 monclient: _renew_subs 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8df4137130 con 0x7f8df40a46a0 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de80038c0 con 0x7f8df40a46a0 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de8003df0 con 0x7f8df40a46a0 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df1ffb640 10 monclient: dump: 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.355 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 10 client.204471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 10 monclient: _renew_subs 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8df37fe640 4 set_mon_vals no callback set 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8df41422e0 con 0x7f8df40a46a0 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.330+0000 7f8dfb7de640 1 librados: init done 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8de80040c0 con 0x7f8df40a46a0 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8dfb7de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8df40a3c70 con 0x7f8df41ce520 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7f8de804e290 con 0x7f8df40a46a0 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df1ffb640 10 client.204471.objecter ms_dispatch 0x7f8df4000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df1ffb640 3 client.204471.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df1ffb640 3 client.204471.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:07:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df1ffb640 20 client.204471.objecter dump_active .. 0 homeless 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.331+0000 7f8df9d7b640 10 monclient: get_auth_request con 0x7f8de0041b10 auth_method 0 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 client.204471.objecter ms_handle_connect 0x7f8df41ce520 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 client.204471.objecter resend_mon_ops 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df957a640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6789/0 conn(0x7f8df41ce520 legacy=0x7f8df409e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 941439244 0 0) 0x7f8df406dbb0 con 0x7f8df41ce520 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient(hunting): my global_id is 175195 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df4138e90 con 0x7f8df41ce520 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 client.204471.objecter ms_handle_connect 0x7f8de0041b10 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3284977783 0 0) 0x7f8df4138e90 con 0x7f8df41ce520 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df406dbb0 con 0x7f8df41ce520 2024-08-19T10:07:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de4004c20 con 0x7f8df41ce520 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.333+0000 7f8df1ffb640 10 monclient: dump: 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2778687710 0 0) 0x7f8df406dbb0 con 0x7f8df41ce520 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8df4142bb0 con 0x7f8df41ce520 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8de4007990 con 0x7f8df41ce520 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6789/0 conn(0x7f8df41ce520 legacy=0x7f8df409e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.334+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6789/0 conn(0x7f8df41ce520 legacy=0x7f8df409e880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.340+0000 7f8dfb7de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.340+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8df40aa720 con 0x7f8df402e1d0 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df957a640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6789/0 conn(0x7f8df402e1d0 legacy=0x7f8df409ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 client.204471.objecter ms_handle_connect 0x7f8df402e1d0 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 client.204471.objecter resend_mon_ops 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973879860 0 0) 0x7f8de4005b80 con 0x7f8df402e1d0 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient(hunting): my global_id is 175204 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df406dbb0 con 0x7f8df402e1d0 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1107007407 0 0) 0x7f8df406dbb0 con 0x7f8df402e1d0 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8de4005b80 con 0x7f8df402e1d0 2024-08-19T10:07:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de4007410 con 0x7f8df402e1d0 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient: dump: 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 308872248 0 0) 0x7f8de4005b80 con 0x7f8df402e1d0 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8df40a5d40 con 0x7f8df402e1d0 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7852 (unknown 0 0 3603302246) 0x7f8de40077c0 con 0x7f8df402e1d0 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6789/0 conn(0x7f8df402e1d0 legacy=0x7f8df409ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.343+0000 7f8df1ffb640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6789/0 conn(0x7f8df402e1d0 legacy=0x7f8df409ef90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 518401, 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:48.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.364 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.365 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.366 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:48.367 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.368 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.370 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:48.383 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 10 librados: watch_flush enter 2024-08-19T10:07:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 10 librados: watch_flush exit 2024-08-19T10:07:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 20 client.204471.objecter shutdown clearing up homeless session... 2024-08-19T10:07:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 10 client.204471.objecter successfully canceled tick 2024-08-19T10:07:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8de0041b10 legacy=0x7f8de0043fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 10 monclient: shutdown 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 >> v1:172.21.15.1:6789/0 conn(0x7f8df40a46a0 legacy=0x7f8df41d3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8df9d7b640 1 -- 172.21.15.1:0/141768963 reap_dead start 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8df9d7b640 1 -- 172.21.15.1:0/141768963 reap_dead start 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 shutdown_connections 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 >> 172.21.15.1:0/141768963 conn(0x7f8df40a9020 msgr2=0x7f8df40a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 shutdown_connections 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 -- 172.21.15.1:0/141768963 wait complete. 2024-08-19T10:07:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.344+0000 7f8dfb7de640 1 librados: shutdown 2024-08-19T10:07:48.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:48.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:07:48.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.485+0000 7fa891215640 10 monclient: get_monmap_and_config 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.485+0000 7fa891215640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.485+0000 7fa891215640 10 monclient: monmap: 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:48.486765+0000 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:48.486765+0000 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.487 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.488 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 1 Processor -- start 2024-08-19T10:07:48.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 1 -- start start 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient: init 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient: _add_conns ranks=[7,2,0,1,6,3,5,4,8] 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient(hunting): picked mon.noname-i con 0x7fa88c0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient(hunting): picked mon.noname-b con 0x7fa88c0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient(hunting): picked mon.noname-a con 0x7fa88c13f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c0a3b40 con 0x7fa88c13f5c0 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c0a8130 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c06ede0 con 0x7fa88c0a67c0 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.486+0000 7fa891215640 10 monclient(hunting): authenticate will time out at 1724.562987s 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.487+0000 7fa88a59c640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa88c0a6ba0 0x7fa88c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51960/0 (socket says 172.21.15.1:51960) 2024-08-19T10:07:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.487+0000 7fa88a59c640 1 -- 172.21.15.1:0/1893645703 learned_addr learned my addr 172.21.15.1:0/1893645703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.487+0000 7fa88b59e640 1 --1- 172.21.15.1:0/1893645703 >> v1:172.21.15.1:6789/0 conn(0x7fa88c13f5c0 0x7fa88c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41354/0 (socket says 172.21.15.1:41354) 2024-08-19T10:07:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa88b59e640 1 -- 172.21.15.1:0/1893645703 >> v1:172.21.15.1:6789/0 conn(0x7fa88c13f5c0 legacy=0x7fa88c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa88a59c640 1 -- 172.21.15.1:0/1893645703 >> v1:172.21.15.1:6790/0 conn(0x7fa88c0a6ba0 legacy=0x7fa88c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa88ad9d640 1 -- 172.21.15.1:0/1893645703 >> v1:172.21.15.82:6792/0 conn(0x7fa88c0a67c0 legacy=0x7fa88c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108631469 0 0) 0x7fa88c0a3b40 con 0x7fa88c13f5c0 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): my global_id is 204480 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa878003520 con 0x7fa88c13f5c0 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841178599 0 0) 0x7fa88c0a8130 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): my global_id is 184844 2024-08-19T10:07:48.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c0a3b40 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168420080 0 0) 0x7fa88c06ede0 con 0x7fa88c0a67c0 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient(hunting): my global_id is 204451 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c0a8130 con 0x7fa88c0a67c0 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1583261247 0 0) 0x7fa878003520 con 0x7fa88c13f5c0 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 >> v1:172.21.15.82:6792/0 conn(0x7fa88c0a67c0 legacy=0x7fa88c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 >> v1:172.21.15.1:6790/0 conn(0x7fa88c0a6ba0 legacy=0x7fa88c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 monclient: found mon.noname-a 2024-08-19T10:07:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa88b59e640 1 --1- 172.21.15.1:0/1893645703 >> v1:172.21.15.1:6789/0 conn(0x7fa88c13f5c0 0x7fa88c1419b0 :-1 s=OPENED pgs=835 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fa884001ba0 mon_map magic: 0 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88c0a92a0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa891215640 5 monclient: authenticate success, global_id 204480 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa891215640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa884001ba0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: dump: 2024-08-19T10:07:48.498 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.499 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8840047e0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa891215640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa889d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa891215640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.489+0000 7fa891215640 4 set_mon_vals no callback set 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 10 monclient: shutdown 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 1 -- 172.21.15.1:0/1893645703 >> v1:172.21.15.1:6789/0 conn(0x7fa88c13f5c0 legacy=0x7fa88c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa889d9b640 1 -- 172.21.15.1:0/1893645703 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa884004d40 con 0x7fa88c13f5c0 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa889d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:07:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 1 -- 172.21.15.1:0/1893645703 shutdown_connections 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 1 -- 172.21.15.1:0/1893645703 >> 172.21.15.1:0/1893645703 conn(0x7fa88c0ac4b0 msgr2=0x7fa88c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 1 -- 172.21.15.1:0/1893645703 shutdown_connections 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.490+0000 7fa891215640 1 -- 172.21.15.1:0/1893645703 wait complete. 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 10 monclient: monmap: 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 1 librados: starting msgr at 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 1 librados: starting objecter 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 1 Processor -- start 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 1 -- start start 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 1 librados: setting wanted keys 2024-08-19T10:07:48.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 1 librados: calling monclient init 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.491+0000 7fa891215640 10 monclient: init 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient: _add_conns ranks=[5,4,6,8,0,7,1,2,3] 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient(hunting): picked mon.noname-h con 0x7fa88c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient(hunting): picked mon.noname-c con 0x7fa88c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient(hunting): picked mon.noname-d con 0x7fa88c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c13bc60 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c1395a0 con 0x7fa88c0a67c0 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c142d60 con 0x7fa88c13f5c0 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa891215640 10 monclient(hunting): authenticate will time out at 1724.568245s 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa88a59c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fa88c0a6ba0 0x7fa88c138e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48622/0 (socket says 172.21.15.1:48622) 2024-08-19T10:07:48.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.492+0000 7fa88a59c640 1 -- 172.21.15.1:0/2158056286 learned_addr learned my addr 172.21.15.1:0/2158056286 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa88b59e640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.1:6792/0 conn(0x7fa88c13f5c0 legacy=0x7fa88c1d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa8837fe640 10 client.?.objecter ms_handle_connect 0x7fa88c13f5c0 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa88a59c640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.1:6791/0 conn(0x7fa88c0a6ba0 legacy=0x7fa88c138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa8837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa8837fe640 10 client.?.objecter ms_handle_connect 0x7fa88c0a6ba0 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa8837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa88ad9d640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6791/0 conn(0x7fa88c0a67c0 legacy=0x7fa88c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.494+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 client.?.objecter ms_handle_connect 0x7fa88c0a67c0 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3576403035 0 0) 0x7fa88c13bc60 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): my global_id is 194683 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa86c004eb0 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4060239248 0 0) 0x7fa88c142d60 con 0x7fa88c13f5c0 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): my global_id is 204414 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c13bc60 con 0x7fa88c13f5c0 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159404035 0 0) 0x7fa88c1395a0 con 0x7fa88c0a67c0 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 10 monclient(hunting): my global_id is 214232 2024-08-19T10:07:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88c142d60 con 0x7fa88c0a67c0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.496+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3233038029 0 0) 0x7fa86c004eb0 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa88c1395a0 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1151846572 0 0) 0x7fa88c13bc60 con 0x7fa88c13f5c0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa86c004eb0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa874001a20 con 0x7fa88c0a6ba0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): dump: 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa884001ba0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): dump: 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2158517170 0 0) 0x7fa88c142d60 con 0x7fa88c0a67c0 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa88c13bc60 con 0x7fa88c0a67c0 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa87c002080 con 0x7fa88c0a67c0 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient(hunting): dump: 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 909585886 0 0) 0x7fa86c004eb0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6791/0 conn(0x7fa88c0a67c0 legacy=0x7fa88c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.1:6791/0 conn(0x7fa88c0a6ba0 legacy=0x7fa88c138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 monclient: found mon.d 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88c13b980 con 0x7fa88c13f5c0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 5 monclient: authenticate success, global_id 204414 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 10 monclient: _renew_subs 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa88c13a8a0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 10 client.204414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 10 monclient: _renew_subs 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa88c13e510 con 0x7fa88c13f5c0 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa891215640 1 librados: init done 2024-08-19T10:07:48.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa884001580 con 0x7fa88c13f5c0 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa884001d40 con 0x7fa88c13f5c0 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa88959a640 4 set_mon_vals no callback set 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.497+0000 7fa8837fe640 10 monclient: dump: 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa891215640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8700065c0 con 0x7fa870002860 2024-08-19T10:07:48.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa884011a60 con 0x7fa88c13f5c0 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7fa88404e1b0 con 0x7fa88c13f5c0 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa8837fe640 10 client.204414.objecter ms_dispatch 0x7fa88c000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa8837fe640 3 client.204414.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa8837fe640 3 client.204414.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa8837fe640 20 client.204414.objecter dump_active .. 0 homeless 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.498+0000 7fa88a59c640 10 monclient: get_auth_request con 0x7fa86c041910 auth_method 0 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa88ad9d640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6790/0 conn(0x7fa870002860 legacy=0x7fa870004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 client.204414.objecter ms_handle_connect 0x7fa870002860 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 client.204414.objecter resend_mon_ops 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682763135 0 0) 0x7fa8700065c0 con 0x7fa870002860 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient(hunting): my global_id is 214230 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa86c004eb0 con 0x7fa870002860 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 client.204414.objecter ms_handle_connect 0x7fa86c041910 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2434410385 0 0) 0x7fa86c004eb0 con 0x7fa870002860 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8700065c0 con 0x7fa870002860 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa87c003f40 con 0x7fa870002860 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.500+0000 7fa8837fe640 10 monclient: dump: 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.521 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3877287972 0 0) 0x7fa8700065c0 con 0x7fa870002860 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa87000a160 con 0x7fa870002860 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa87c005b40 con 0x7fa870002860 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6790/0 conn(0x7fa870002860 legacy=0x7fa870004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.501+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6790/0 conn(0x7fa870002860 legacy=0x7fa870004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.507+0000 7fa891215640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.507+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa870009e70 con 0x7fa870009130 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa88ad9d640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6790/0 conn(0x7fa870009130 legacy=0x7fa8700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 client.204414.objecter ms_handle_connect 0x7fa870009130 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 client.204414.objecter resend_mon_ops 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139338084 0 0) 0x7fa870009e70 con 0x7fa870009130 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient(hunting): my global_id is 214239 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8700065c0 con 0x7fa870009130 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 61437533 0 0) 0x7fa8700065c0 con 0x7fa870009130 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa870009e70 con 0x7fa870009130 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa87c005cf0 con 0x7fa870009130 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 10 monclient: dump: 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3239984513 0 0) 0x7fa870009e70 con 0x7fa870009130 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.510+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa87000c410 con 0x7fa870009130 2024-08-19T10:07:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 3912825658) 0x7fa87c007d70 con 0x7fa870009130 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa8837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa8837fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6790/0 conn(0x7fa870009130 legacy=0x7fa8700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa8837fe640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6790/0 conn(0x7fa870009130 legacy=0x7fa8700095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13384, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:07:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:48.548 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 10 librados: watch_flush enter 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 10 librados: watch_flush exit 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 20 client.204414.objecter shutdown clearing up homeless session... 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 10 client.204414.objecter successfully canceled tick 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa86c041910 legacy=0x7fa86c043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 10 monclient: shutdown 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 >> v1:172.21.15.1:6792/0 conn(0x7fa88c13f5c0 legacy=0x7fa88c1d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa88b59e640 1 -- 172.21.15.1:0/2158056286 reap_dead start 2024-08-19T10:07:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa88b59e640 1 -- 172.21.15.1:0/2158056286 reap_dead start 2024-08-19T10:07:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 shutdown_connections 2024-08-19T10:07:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 >> 172.21.15.1:0/2158056286 conn(0x7fa88c0ac4b0 msgr2=0x7fa88c141bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 shutdown_connections 2024-08-19T10:07:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 -- 172.21.15.1:0/2158056286 wait complete. 2024-08-19T10:07:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.511+0000 7fa891215640 1 librados: shutdown 2024-08-19T10:07:48.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:48.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:07:48.552 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:07:48.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.553 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1582512127}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3695155864}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1429935719}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.546+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1970481783}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.554 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2768347242}) 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1792128896}) 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=688899663}) 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860249761}) 2024-08-19T10:07:48.555 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.547+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3411953549}) 2024-08-19T10:07:48.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.653+0000 7f7470981640 10 monclient: get_monmap_and_config 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.653+0000 7f7470981640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.653+0000 7f7470981640 10 monclient: monmap: 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:48.654524+0000 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:48.654524+0000 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.656 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.656 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.656 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.656 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.656 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 1 Processor -- start 2024-08-19T10:07:48.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 1 -- start start 2024-08-19T10:07:48.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 10 monclient: init 2024-08-19T10:07:48.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 10 monclient: _add_conns ranks=[5,0,1,3,2,7,8,6,4] 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 10 monclient(hunting): picked mon.noname-h con 0x7f746c0a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 10 monclient(hunting): picked mon.noname-a con 0x7f746c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 10 monclient(hunting): picked mon.noname-f con 0x7f746c13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0a3b40 con 0x7f746c0a6ba0 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0a8130 con 0x7f746c13f5c0 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.654+0000 7f7470981640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c06ede0 con 0x7f746c0a67c0 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.655+0000 7f7470981640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.655+0000 7f7470981640 10 monclient(hunting): authenticate will time out at 1724.731017s 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.655+0000 7f7463fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f746c0a6ba0 0x7f746c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41366/0 (socket says 172.21.15.1:41366) 2024-08-19T10:07:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.655+0000 7f7463fff640 1 -- 172.21.15.1:0/3031951236 learned_addr learned my addr 172.21.15.1:0/3031951236 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7463fff640 1 -- 172.21.15.1:0/3031951236 >> v1:172.21.15.1:6789/0 conn(0x7f746c0a6ba0 legacy=0x7f746c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193842175 0 0) 0x7f746c0a3b40 con 0x7f746c0a6ba0 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): my global_id is 204489 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7458003520 con 0x7f746c0a6ba0 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f746a59c640 1 -- 172.21.15.1:0/3031951236 >> v1:172.21.15.82:6791/0 conn(0x7f746c0a67c0 legacy=0x7f746c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f746ad9d640 1 -- 172.21.15.1:0/3031951236 >> v1:172.21.15.82:6789/0 conn(0x7f746c13f5c0 legacy=0x7f746c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013021546 0 0) 0x7f746c06ede0 con 0x7f746c0a67c0 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient(hunting): my global_id is 214241 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c0a3b40 con 0x7f746c0a67c0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 551589834 0 0) 0x7f7458003520 con 0x7f746c0a6ba0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 >> v1:172.21.15.82:6791/0 conn(0x7f746c0a67c0 legacy=0x7f746c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 >> v1:172.21.15.82:6789/0 conn(0x7f746c13f5c0 legacy=0x7f746c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 monclient: found mon.noname-a 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0a92a0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7454001ae0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7470981640 5 monclient: authenticate success, global_id 204489 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7470981640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: dump: 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7470981640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74540048a0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 1 -- 172.21.15.1:0/3031951236 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7454004e10 con 0x7f746c0a6ba0 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7470981640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7470981640 4 set_mon_vals no callback set 2024-08-19T10:07:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.657+0000 7f7469d9b640 10 monclient: dump: 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 10 monclient: shutdown 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 1 -- 172.21.15.1:0/3031951236 >> v1:172.21.15.1:6789/0 conn(0x7f746c0a6ba0 legacy=0x7f746c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 1 -- 172.21.15.1:0/3031951236 shutdown_connections 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 1 -- 172.21.15.1:0/3031951236 >> 172.21.15.1:0/3031951236 conn(0x7f746c0ac4b0 msgr2=0x7f746c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 1 -- 172.21.15.1:0/3031951236 shutdown_connections 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 1 -- 172.21.15.1:0/3031951236 wait complete. 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.658+0000 7f7470981640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient: monmap: 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.672 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 librados: starting msgr at 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 librados: starting objecter 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 Processor -- start 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 -- start start 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 librados: setting wanted keys 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 librados: calling monclient init 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient: init 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient: _add_conns ranks=[2,0,3,6,5,8,4,7,1] 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient(hunting): picked mon.noname-b con 0x7f746c0a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient(hunting): picked mon.noname-a con 0x7f746c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient(hunting): picked mon.noname-g con 0x7f746c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c13a770 con 0x7f746c0a6ba0 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c13aa50 con 0x7f746c0a67c0 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c13f330 con 0x7f746c13f5c0 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7470981640 10 monclient(hunting): authenticate will time out at 1724.735843s 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7463fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f746c0a6ba0 0x7f746c0a0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41376/0 (socket says 172.21.15.1:41376) 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f7463fff640 1 -- 172.21.15.1:0/3200062798 learned_addr learned my addr 172.21.15.1:0/3200062798 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.659+0000 7f746a59c640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f746c0a67c0 0x7f746c09ff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51968/0 (socket says 172.21.15.1:51968) 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f746a59c640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.1:6790/0 conn(0x7f746c0a67c0 legacy=0x7f746c09ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7463fff640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.1:6789/0 conn(0x7f746c0a6ba0 legacy=0x7f746c0a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 client.?.objecter ms_handle_connect 0x7f746c0a67c0 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 client.?.objecter ms_handle_connect 0x7f746c0a6ba0 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f746a59c640 1 --1- 172.21.15.1:0/3200062798 >> v1:172.21.15.1:6790/0 conn(0x7f746c0a67c0 0x7f746c09ff80 :-1 s=OPENED pgs=3630 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f746c13aa50 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3251421406 0 0) 0x7f746c13a770 con 0x7f746c0a6ba0 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): my global_id is 204498 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f746ad9d640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6790/0 conn(0x7f746c13f5c0 legacy=0x7f746c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f744c004eb0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 client.?.objecter ms_handle_connect 0x7f746c13f5c0 2024-08-19T10:07:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2892235726 0 0) 0x7f744c004eb0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f746c13a770 con 0x7f746c0a6ba0 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7454004d90 con 0x7f746c0a6ba0 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): dump: 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834720545 0 0) 0x7f746c13f330 con 0x7f746c13f5c0 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient(hunting): my global_id is 214248 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f744c004eb0 con 0x7f746c13f5c0 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 351704061 0 0) 0x7f746c13a770 con 0x7f746c0a6ba0 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6790/0 conn(0x7f746c13f5c0 legacy=0x7f746c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.1:6790/0 conn(0x7f746c0a67c0 legacy=0x7f746c09ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 monclient: found mon.a 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0a28b0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 5 monclient: authenticate success, global_id 204498 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204288691 0 0) 0x7f746c13aa50 con 0x7f746c0a67c0 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 10 monclient: _renew_subs 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f746c0a16e0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 10 client.204498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 10 monclient: _renew_subs 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f746c0a0da0 con 0x7f746c0a6ba0 2024-08-19T10:07:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7470981640 1 librados: init done 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7454001d30 con 0x7f746c0a6ba0 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7454001b00 con 0x7f746c0a6ba0 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f7462ffd640 10 monclient: dump: 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.662+0000 7f746959a640 4 set_mon_vals no callback set 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7470981640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74500065c0 con 0x7f7450002860 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7454011a00 con 0x7f746c0a6ba0 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7f745404e620 con 0x7f746c0a6ba0 2024-08-19T10:07:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7462ffd640 10 client.204498.objecter ms_dispatch 0x7f746c000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:07:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7462ffd640 3 client.204498.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:07:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7462ffd640 3 client.204498.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:07:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f7462ffd640 20 client.204498.objecter dump_active .. 0 homeless 2024-08-19T10:07:48.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.663+0000 7f746ad9d640 10 monclient: get_auth_request con 0x7f744c040e30 auth_method 0 2024-08-19T10:07:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f746a59c640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6791/0 conn(0x7f7450002860 legacy=0x7f7450004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 10 client.204498.objecter ms_handle_connect 0x7f7450002860 2024-08-19T10:07:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 10 client.204498.objecter resend_mon_ops 2024-08-19T10:07:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640746327 0 0) 0x7f74500065c0 con 0x7f7450002860 2024-08-19T10:07:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.665+0000 7f7462ffd640 10 monclient(hunting): my global_id is 214250 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c13aa50 con 0x7f7450002860 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3470889487 0 0) 0x7f746c13aa50 con 0x7f7450002860 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74500065c0 con 0x7f7450002860 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f745c001d30 con 0x7f7450002860 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 10 monclient: dump: 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 10 client.204498.objecter ms_handle_connect 0x7f744c040e30 2024-08-19T10:07:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1974535606 0 0) 0x7f74500065c0 con 0x7f7450002860 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f745000a160 con 0x7f7450002860 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f745c0057c0 con 0x7f7450002860 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6791/0 conn(0x7f7450002860 legacy=0x7f7450004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.666+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6791/0 conn(0x7f7450002860 legacy=0x7f7450004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.673+0000 7f7470981640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.673+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7450009e70 con 0x7f7450009130 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f746a59c640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6791/0 conn(0x7f7450009130 legacy=0x7f74500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 client.204498.objecter ms_handle_connect 0x7f7450009130 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 client.204498.objecter resend_mon_ops 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935935755 0 0) 0x7f7450009e70 con 0x7f7450009130 2024-08-19T10:07:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient(hunting): my global_id is 214259 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74500065c0 con 0x7f7450009130 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1325179681 0 0) 0x7f74500065c0 con 0x7f7450009130 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7450009e70 con 0x7f7450009130 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f745c003ef0 con 0x7f7450009130 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.675+0000 7f7462ffd640 10 monclient: dump: 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.689 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3216311671 0 0) 0x7f7450009e70 con 0x7f7450009130 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f745000c410 con 0x7f7450009130 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7517 (unknown 0 0 1002505352) 0x7f745c0042c0 con 0x7f7450009130 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6791/0 conn(0x7f7450009130 legacy=0x7f74500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7462ffd640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6791/0 conn(0x7f7450009130 legacy=0x7f74500095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13550, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 7, 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:48.711 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:48.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 10 librados: watch_flush enter 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 10 librados: watch_flush exit 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 20 client.204498.objecter shutdown clearing up homeless session... 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 10 client.204498.objecter successfully canceled tick 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.82:6812/2549104415 conn(0x7f744c040e30 legacy=0x7f744c0432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 10 monclient: shutdown 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 >> v1:172.21.15.1:6789/0 conn(0x7f746c0a6ba0 legacy=0x7f746c0a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f746ad9d640 1 -- 172.21.15.1:0/3200062798 reap_dead start 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f746ad9d640 1 -- 172.21.15.1:0/3200062798 reap_dead start 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 shutdown_connections 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 >> 172.21.15.1:0/3200062798 conn(0x7f746c0ac4b0 msgr2=0x7f746c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 shutdown_connections 2024-08-19T10:07:48.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 -- 172.21.15.1:0/3200062798 wait complete. 2024-08-19T10:07:48.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.676+0000 7f7470981640 1 librados: shutdown 2024-08-19T10:07:48.714 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:48.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:07:48.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:07:48.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.737 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1579446206}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1908719611}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1711267843}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=251663624}) 2024-08-19T10:07:48.738 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2613591503}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2925090922}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4064885075}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249321172}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.729+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1505759629}) 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.730+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.739 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.730+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2748115426}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225943084}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4065354850}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3016492524}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.740 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=390890567}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1241765716}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2285051747}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183636138}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.731+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2029732776}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4203340397}) 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225688983}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=964791576}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3742692143}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2506820149}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.742 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1684132826}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3505190848}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94948953}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.732+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1882634226}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034536662}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1043763113}) 2024-08-19T10:07:48.743 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1692389733}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2886495380}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2860469132}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3959231579}) 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.744 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4203465891}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954666229}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.733+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=239214709}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407593106}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680551249}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.734+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.745 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=929191705}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1075637382}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3415481788}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1854143424}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2503718521}) 2024-08-19T10:07:48.746 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001178787}) 2024-08-19T10:07:48.747 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.735+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4220000821}) 2024-08-19T10:07:48.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.817+0000 7fa831ae5640 10 monclient: get_monmap_and_config 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.817+0000 7fa831ae5640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.817+0000 7fa831ae5640 10 monclient: monmap: 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:48.818769+0000 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:48.818769+0000 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.820 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.818+0000 7fa831ae5640 1 Processor -- start 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.818+0000 7fa831ae5640 1 -- start start 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.818+0000 7fa831ae5640 10 monclient: init 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient: _add_conns ranks=[0,7,8,3,1,5,4,2,6] 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient(hunting): picked mon.noname-a con 0x7fa82c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient(hunting): picked mon.noname-i con 0x7fa82c0a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient(hunting): picked mon.noname-e con 0x7fa82c13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c0a3b40 con 0x7fa82c0a67c0 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c0a8130 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c06ede0 con 0x7fa82c13f5c0 2024-08-19T10:07:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa831ae5640 10 monclient(hunting): authenticate will time out at 1724.895163s 2024-08-19T10:07:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa82bfff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fa82c13f5c0 0x7fa82c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:55368/0 (socket says 172.21.15.1:55368) 2024-08-19T10:07:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa82b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa82c0a67c0 0x7fa82c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41388/0 (socket says 172.21.15.1:41388) 2024-08-19T10:07:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.819+0000 7fa82bfff640 1 -- 172.21.15.1:0/1317252489 learned_addr learned my addr 172.21.15.1:0/1317252489 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82b7fe640 1 -- 172.21.15.1:0/1317252489 >> v1:172.21.15.1:6789/0 conn(0x7fa82c0a67c0 legacy=0x7fa82c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82bfff640 1 -- 172.21.15.1:0/1317252489 >> v1:172.21.15.1:6793/0 conn(0x7fa82c13f5c0 legacy=0x7fa82c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82affd640 1 -- 172.21.15.1:0/1317252489 >> v1:172.21.15.82:6792/0 conn(0x7fa82c0a6ba0 legacy=0x7fa82c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962770747 0 0) 0x7fa82c0a3b40 con 0x7fa82c0a67c0 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): my global_id is 204507 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa814003520 con 0x7fa82c0a67c0 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520169598 0 0) 0x7fa82c06ede0 con 0x7fa82c13f5c0 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): my global_id is 204524 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c0a3b40 con 0x7fa82c13f5c0 2024-08-19T10:07:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677872774 0 0) 0x7fa82c0a8130 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient(hunting): my global_id is 204460 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c06ede0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2820114702 0 0) 0x7fa814003520 con 0x7fa82c0a67c0 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 >> v1:172.21.15.1:6793/0 conn(0x7fa82c13f5c0 legacy=0x7fa82c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 >> v1:172.21.15.82:6792/0 conn(0x7fa82c0a6ba0 legacy=0x7fa82c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 monclient: found mon.noname-a 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0a92a0 con 0x7fa82c0a67c0 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa831ae5640 5 monclient: authenticate success, global_id 204507 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa81c001ae0 con 0x7fa82c0a67c0 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa831ae5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa82a7fc640 10 monclient: dump: 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.821+0000 7fa831ae5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa81c0047e0 con 0x7fa82c0a67c0 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa82a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa82a7fc640 1 -- 172.21.15.1:0/1317252489 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa81c004d50 con 0x7fa82c0a67c0 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 10 monclient: get_monmap_and_config success 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 4 set_mon_vals no callback set 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa82a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa82a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa82a7fc640 10 monclient: dump: 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 10 monclient: shutdown 2024-08-19T10:07:48.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 1 -- 172.21.15.1:0/1317252489 >> v1:172.21.15.1:6789/0 conn(0x7fa82c0a67c0 legacy=0x7fa82c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 1 -- 172.21.15.1:0/1317252489 shutdown_connections 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 1 -- 172.21.15.1:0/1317252489 >> 172.21.15.1:0/1317252489 conn(0x7fa82c0ac4b0 msgr2=0x7fa82c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 1 -- 172.21.15.1:0/1317252489 shutdown_connections 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.822+0000 7fa831ae5640 1 -- 172.21.15.1:0/1317252489 wait complete. 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 10 monclient: build_initial_monmap 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 10 monclient: monmap: 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 1 librados: starting msgr at 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 1 librados: starting objecter 2024-08-19T10:07:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 1 Processor -- start 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 1 -- start start 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 1 librados: setting wanted keys 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 1 librados: calling monclient init 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.823+0000 7fa831ae5640 10 monclient: init 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient: _add_conns ranks=[3,2,7,5,4,8,6,1,0] 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient(hunting): picked mon.noname-g con 0x7fa82c0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient(hunting): picked mon.noname-b con 0x7fa82c0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient(hunting): picked mon.noname-i con 0x7fa82c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c139ad0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c13a990 con 0x7fa82c0a67c0 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c142e70 con 0x7fa82c13f5c0 2024-08-19T10:07:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa831ae5640 10 monclient(hunting): authenticate will time out at 1724.900162s 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa82affd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa82c0a6ba0 0x7fa82c0a0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51972/0 (socket says 172.21.15.1:51972) 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.824+0000 7fa82affd640 1 -- 172.21.15.1:0/1829320681 learned_addr learned my addr 172.21.15.1:0/1829320681 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa82affd640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.1:6790/0 conn(0x7fa82c0a6ba0 legacy=0x7fa82c0a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 client.?.objecter ms_handle_connect 0x7fa82c0a6ba0 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 client.?.objecter ms_handle_connect 0x7fa82c13f5c0 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa82bfff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa82c13f5c0 legacy=0x7fa82c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 client.?.objecter ms_handle_connect 0x7fa82c0a67c0 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa82b7fe640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6790/0 conn(0x7fa82c0a67c0 legacy=0x7fa82c0a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214646575 0 0) 0x7fa82c139ad0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): my global_id is 184862 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa818004eb0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432361566 0 0) 0x7fa82c142e70 con 0x7fa82c13f5c0 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): my global_id is 204469 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c139ad0 con 0x7fa82c13f5c0 2024-08-19T10:07:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777325282 0 0) 0x7fa82c13a990 con 0x7fa82c0a67c0 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): my global_id is 214257 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c142e70 con 0x7fa82c0a67c0 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2148954645 0 0) 0x7fa818004eb0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa82c13a990 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa820003610 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): dump: 2024-08-19T10:07:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2039996267 0 0) 0x7fa82c13a990 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa82c13f5c0 legacy=0x7fa82c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6790/0 conn(0x7fa82c0a67c0 legacy=0x7fa82c0a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.826+0000 7fa813fff640 10 monclient: _finish_hunting 0 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 1 monclient: found mon.b 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0a17d0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: _finish_auth 0 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: _check_auth_tickets 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 5 monclient: authenticate success, global_id 184862 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 10 monclient: _renew_subs 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c13a6b0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 10 client.184862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 10 monclient: _renew_subs 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa82c0a0e30 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 1 librados: init done 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa820003bb0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa820004120 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa829ffb640 4 set_mon_vals no callback set 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 10 monclient: dump: 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa80c0065c0 con 0x7fa80c002860 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.827+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fa8200042c0 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.828+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7fa82004e550 con 0x7fa82c0a6ba0 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.828+0000 7fa813fff640 10 client.184862.objecter ms_dispatch 0x7fa82c000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.828+0000 7fa813fff640 3 client.184862.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.828+0000 7fa813fff640 3 client.184862.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.828+0000 7fa813fff640 20 client.184862.objecter dump_active .. 0 homeless 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.828+0000 7fa82bfff640 10 monclient: get_auth_request con 0x7fa818041120 auth_method 0 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 client.184862.objecter ms_handle_connect 0x7fa80c002860 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 client.184862.objecter resend_mon_ops 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa82b7fe640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa80c002860 legacy=0x7fa80c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1063870647 0 0) 0x7fa80c0065c0 con 0x7fa80c002860 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient(hunting): my global_id is 204478 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c13a990 con 0x7fa80c002860 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2081646238 0 0) 0x7fa82c13a990 con 0x7fa80c002860 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa80c0065c0 con 0x7fa80c002860 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa81c001d30 con 0x7fa80c002860 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 monclient: dump: 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 10 client.184862.objecter ms_handle_connect 0x7fa818041120 2024-08-19T10:07:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 488504042 0 0) 0x7fa80c0065c0 con 0x7fa80c002860 2024-08-19T10:07:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.830+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa80c00a160 con 0x7fa80c002860 2024-08-19T10:07:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.831+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa81c004ad0 con 0x7fa80c002860 2024-08-19T10:07:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.831+0000 7fa813fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:07:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.831+0000 7fa813fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:07:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.831+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa80c002860 legacy=0x7fa80c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.831+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa80c002860 legacy=0x7fa80c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.837+0000 7fa831ae5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.837+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa80c009e70 con 0x7fa80c009130 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa82b7fe640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa80c009130 legacy=0x7fa80c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 10 client.184862.objecter ms_handle_connect 0x7fa80c009130 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 10 client.184862.objecter resend_mon_ops 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1049978154 0 0) 0x7fa80c009e70 con 0x7fa80c009130 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 10 monclient(hunting): _negotiate 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 10 monclient(hunting): my global_id is 204487 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.839+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa80c0065c0 con 0x7fa80c009130 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1456941736 0 0) 0x7fa80c0065c0 con 0x7fa80c009130 2024-08-19T10:07:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa80c009e70 con 0x7fa80c009130 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fa81c0053d0 con 0x7fa80c009130 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 10 monclient: dump: 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:07:48.856 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2287647194 0 0) 0x7fa80c009e70 con 0x7fa80c009130 2024-08-19T10:07:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa80c00c3f0 con 0x7fa80c009130 2024-08-19T10:07:48.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 300171390) 0x7fa81c008540 con 0x7fa80c009130 2024-08-19T10:07:48.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:07:48.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:07:48.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa80c009130 legacy=0x7fa80c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa813fff640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6792/0 conn(0x7fa80c009130 legacy=0x7fa80c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:07:48.859 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:07:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:07:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 134711, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:07:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 8, 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:07:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:07:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:07:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:07:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:07:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:07:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:07:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:07:48.880 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:07:48.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 10 librados: watch_flush enter 2024-08-19T10:07:48.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 10 librados: watch_flush exit 2024-08-19T10:07:48.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 20 client.184862.objecter shutdown clearing up homeless session... 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 10 client.184862.objecter successfully canceled tick 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.82:6812/2549104415 conn(0x7fa818041120 legacy=0x7fa8180435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 10 monclient: shutdown 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 >> v1:172.21.15.1:6790/0 conn(0x7fa82c0a6ba0 legacy=0x7fa82c0a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa82bfff640 1 -- 172.21.15.1:0/1829320681 reap_dead start 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.840+0000 7fa82bfff640 1 -- 172.21.15.1:0/1829320681 reap_dead start 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.841+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 shutdown_connections 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.841+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 >> 172.21.15.1:0/1829320681 conn(0x7fa82c0ac4b0 msgr2=0x7fa82c0aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.841+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 shutdown_connections 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.841+0000 7fa831ae5640 1 -- 172.21.15.1:0/1829320681 wait complete. 2024-08-19T10:07:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:48.841+0000 7fa831ae5640 1 librados: shutdown 2024-08-19T10:07:48.884 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:07:48.884 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['f'] 2024-08-19T10:07:48.884 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-19T10:07:48.884 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-08-19T10:07:48.884 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-08-19T10:07:48.885 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T10:07:48.885 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:07:48.902 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T10:07:48.902 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T10:07:48.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:07:48.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4229812539}) 2024-08-19T10:07:48.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646379663}) 2024-08-19T10:07:48.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2946834586}) 2024-08-19T10:07:48.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=764627820}) 2024-08-19T10:07:48.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.930+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=582066998}) 2024-08-19T10:07:48.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2155743342}) 2024-08-19T10:07:48.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3548377652}) 2024-08-19T10:07:48.942 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:07:48.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906546758}) 2024-08-19T10:07:48.943 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:07:48.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2858923788}) 2024-08-19T10:07:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.021+0000 7f20efb2f640 10 monclient: get_monmap_and_config 2024-08-19T10:07:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.021+0000 7f20efb2f640 10 monclient: build_initial_monmap 2024-08-19T10:07:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.021+0000 7f20efb2f640 10 monclient: monmap: 2024-08-19T10:07:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:07:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:07:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:07:49.022127+0000 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:07:49.022127+0000 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:07:49.023 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:07:49.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.021+0000 7f20efb2f640 1 Processor -- start 2024-08-19T10:07:49.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 1 -- start start 2024-08-19T10:07:49.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient: init 2024-08-19T10:07:49.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient: _reopen_session rank -1 2024-08-19T10:07:49.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient: _add_conns ranks=[5,3,7,0,2,8,1,4,6] 2024-08-19T10:07:49.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient(hunting): picked mon.noname-h con 0x7f20e80a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:07:49.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient(hunting): picked mon.noname-g con 0x7f20e80694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:49.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient(hunting): picked mon.noname-i con 0x7f20e80a46a0 addr v1:172.21.15.82:6792/0 2024-08-19T10:07:49.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e80a3c70 con 0x7f20e80694b0 2024-08-19T10:07:49.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e80a33d0 con 0x7f20e80a6830 2024-08-19T10:07:49.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e80a36b0 con 0x7f20e80a46a0 2024-08-19T10:07:49.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:49.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20efb2f640 10 monclient(hunting): authenticate will time out at 1725.098549s 2024-08-19T10:07:49.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20ed8cb640 1 --1- >> v1:172.21.15.82:6791/0 conn(0x7f20e80a6830 0x7f20e80a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6791/0 says I am v1:172.21.15.1:34518/0 (socket says 172.21.15.1:34518) 2024-08-19T10:07:49.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.022+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 learned_addr learned my addr 172.21.15.1:0/3238359273 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:07:49.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.025+0000 7f20ed0ca640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6790/0 conn(0x7f20e80694b0 legacy=0x7f20e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:49.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.025+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6791/0 conn(0x7f20e80a6830 legacy=0x7f20e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:49.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:49.025+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6792/0 conn(0x7f20e80a46a0 legacy=0x7f20e8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): tick 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 monclient(hunting): continuing hunt 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6792/0 conn(0x7f20e80a46a0 legacy=0x7f20e8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6791/0 conn(0x7f20e80a6830 legacy=0x7f20e80a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:52.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6790/0 conn(0x7f20e80694b0 legacy=0x7f20e80a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient: _add_conns ranks=[1,6,8,0,4,2] 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-f con 0x7f20cc003960 addr v1:172.21.15.82:6789/0 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-d con 0x7f20cc007530 addr v1:172.21.15.1:6792/0 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-e con 0x7f20cc00b280 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e806dbb0 con 0x7f20cc003960 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc010150 con 0x7f20cc007530 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc0113f0 con 0x7f20cc00b280 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20df7fe640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:52.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.022+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc003960 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:07:52.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.025+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6793/0 conn(0x7f20cc00b280 legacy=0x7f20cc00d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:52.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.025+0000 7f20ed0ca640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6792/0 conn(0x7f20cc007530 legacy=0x7f20cc0099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:52.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.224+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc003960 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:07:52.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:52.625+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc003960 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:07:53.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:53.426+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc003960 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): tick 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 monclient(hunting): continuing hunt 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6793/0 conn(0x7f20cc00b280 legacy=0x7f20cc00d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6792/0 conn(0x7f20cc007530 legacy=0x7f20cc0099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc003960 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient: _add_conns ranks=[4,0,2] 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 reap_dead start 2024-08-19T10:07:55.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f20cc010680 addr v1:172.21.15.1:6791/0 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 reap_dead start 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f20cc00df10 addr v1:172.21.15.1:6789/0 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc00e310 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc0126f0 con 0x7f20cc00df10 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e806dbb0 con 0x7f20cc00e310 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc00e950 con 0x7f20cc010680 2024-08-19T10:07:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.023+0000 7f20df7fe640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:55.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.025+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6791/0 conn(0x7f20cc010680 legacy=0x7f20cc010a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:55.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.025+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6790/0 conn(0x7f20cc00e310 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:55.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:55.025+0000 7f20ed0ca640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6789/0 conn(0x7f20cc00df10 legacy=0x7f20cc0099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:58.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 10 monclient(hunting): tick 2024-08-19T10:07:58.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:07:58.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 1 monclient(hunting): continuing hunt 2024-08-19T10:07:58.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:07:58.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6791/0 conn(0x7f20cc010680 legacy=0x7f20cc010a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:58.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6790/0 conn(0x7f20cc00e310 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.023+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6789/0 conn(0x7f20cc00df10 legacy=0x7f20cc0099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 10 monclient: _add_conns ranks=[3,2,8,1,6,5,4,7,0] 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-g con 0x7f20cc011790 addr v1:172.21.15.82:6790/0 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc00e310 addr v1:172.21.15.1:6790/0 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-e con 0x7f20cc011c10 addr v1:172.21.15.1:6793/0 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc00ebe0 con 0x7f20cc00e310 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc0121d0 con 0x7f20cc011790 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc01b190 con 0x7f20cc011c10 2024-08-19T10:07:58.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.024+0000 7f20df7fe640 10 monclient(hunting): _renew_subs 2024-08-19T10:07:58.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.026+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6790/0 conn(0x7f20cc011790 legacy=0x7f20cc00d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:58.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.027+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6793/0 conn(0x7f20cc011c10 legacy=0x7f20cc017a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:07:58.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:07:58.027+0000 7f20ed0ca640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6790/0 conn(0x7f20cc00e310 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): tick 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): _check_auth_tickets 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 monclient(hunting): continuing hunt 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6793/0 conn(0x7f20cc011c10 legacy=0x7f20cc017a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6790/0 conn(0x7f20cc011790 legacy=0x7f20cc00d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6790/0 conn(0x7f20cc00e310 legacy=0x7f20cc005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient: _add_conns ranks=[1,5,6,7,4,0] 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 reap_dead start 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-f con 0x7f20cc01b600 addr v1:172.21.15.82:6789/0 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-h con 0x7f20cc00df10 addr v1:172.21.15.82:6791/0 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc01b600 legacy=0x7f20cc01ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): picked mon.noname-d con 0x7f20cc00e310 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc01c370 con 0x7f20cc01b600 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc010660 con 0x7f20cc00df10 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc010830 con 0x7f20cc00e310 2024-08-19T10:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.025+0000 7f20df7fe640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:01.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.027+0000 7f20ee0cc640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6792/0 conn(0x7f20cc00e310 legacy=0x7f20cc005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:01.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.027+0000 7f20ed0ca640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6791/0 conn(0x7f20cc00df10 legacy=0x7f20cc00d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:01.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.226+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc01b600 legacy=0x7f20cc01ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:01.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:01.627+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc01b600 legacy=0x7f20cc01ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:02.428+0000 7f20ed8cb640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc01b600 legacy=0x7f20cc01ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889851410 0 0) 0x7f20cc010660 con 0x7f20cc00df10 2024-08-19T10:08:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient(hunting): _negotiate 2024-08-19T10:08:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient(hunting): my global_id is 214268 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d4003520 con 0x7f20cc00df10 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3279598578 0 0) 0x7f20d4003520 con 0x7f20cc00df10 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.1:6792/0 conn(0x7f20cc00e310 legacy=0x7f20cc005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6789/0 conn(0x7f20cc01b600 legacy=0x7f20cc01ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: _finish_hunting 0 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 monclient: found mon.noname-h 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.82:6791/0 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc0099c0 con 0x7f20cc00df10 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: _finish_auth 0 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: _check_auth_tickets 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f20d8001af0 con 0x7f20cc00df10 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20ec8c9640 10 monclient: dump: 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20efb2f640 5 monclient: authenticate success, global_id 214268 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.830+0000 7f20efb2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20d8001c90 con 0x7f20cc00df10 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20ec8c9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20ec8c9640 1 -- 172.21.15.1:0/3238359273 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f20d8003950 con 0x7f20cc00df10 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20ec8c9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20ec8c9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20ec8c9640 10 monclient: dump: 2024-08-19T10:08:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:03.838 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20efb2f640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.831+0000 7f20efb2f640 4 set_mon_vals no callback set 2024-08-19T10:08:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 10 monclient: shutdown 2024-08-19T10:08:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 1 -- 172.21.15.1:0/3238359273 >> v1:172.21.15.82:6791/0 conn(0x7f20cc00df10 legacy=0x7f20cc00d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 1 -- 172.21.15.1:0/3238359273 shutdown_connections 2024-08-19T10:08:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 1 -- 172.21.15.1:0/3238359273 >> 172.21.15.1:0/3238359273 conn(0x7f20e80a9020 msgr2=0x7f20e8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 1 -- 172.21.15.1:0/3238359273 shutdown_connections 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.832+0000 7f20efb2f640 1 -- 172.21.15.1:0/3238359273 wait complete. 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.833+0000 7f20efb2f640 10 monclient: build_initial_monmap 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.833+0000 7f20efb2f640 10 monclient: monmap: 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 1 librados: starting msgr at 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 1 librados: starting objecter 2024-08-19T10:08:03.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 1 Processor -- start 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 1 -- start start 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 1 librados: setting wanted keys 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 1 librados: calling monclient init 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.834+0000 7f20efb2f640 10 monclient: init 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient: _add_conns ranks=[2,6,0,1,8,4,3,5,7] 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc00df10 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient(hunting): picked mon.noname-d con 0x7f20cc00e310 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient(hunting): picked mon.noname-a con 0x7f20cc01b600 addr v1:172.21.15.1:6789/0 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e8126970 con 0x7f20cc01b600 2024-08-19T10:08:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e80aa770 con 0x7f20cc00df10 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e8124eb0 con 0x7f20cc00e310 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20efb2f640 10 monclient(hunting): authenticate will time out at 1739.911383s 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20ed8cb640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f20cc00df10 0x7f20e8139850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53682/0 (socket says 172.21.15.1:53682) 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20ed8cb640 1 -- 172.21.15.1:0/1270499863 learned_addr learned my addr 172.21.15.1:0/1270499863 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.835+0000 7f20ed0ca640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f20cc00e310 0x7f20e81d2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:57274/0 (socket says 172.21.15.1:57274) 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20ed8cb640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.1:6790/0 conn(0x7f20cc00df10 legacy=0x7f20e8139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20ed0ca640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.1:6792/0 conn(0x7f20cc00e310 legacy=0x7f20e81d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20de7fc640 10 client.?.objecter ms_handle_connect 0x7f20cc00df10 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20de7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20de7fc640 10 client.?.objecter ms_handle_connect 0x7f20cc00e310 2024-08-19T10:08:03.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20de7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20de7fc640 10 client.?.objecter ms_handle_connect 0x7f20cc01b600 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20de7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.837+0000 7f20ee0cc640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.1:6789/0 conn(0x7f20cc01b600 legacy=0x7f20e81d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990528820 0 0) 0x7f20cc010830 con 0x7f20cc01b600 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): my global_id is 204516 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20cc010640 con 0x7f20cc01b600 2024-08-19T10:08:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4250372616 0 0) 0x7f20cc010640 con 0x7f20cc01b600 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20cc010830 con 0x7f20cc01b600 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f20e0001f50 con 0x7f20cc01b600 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): dump: 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:03.849 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.838+0000 7f20de7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2431983437 0 0) 0x7f20cc010830 con 0x7f20cc01b600 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.1:6792/0 conn(0x7f20cc00e310 legacy=0x7f20e81d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.1:6790/0 conn(0x7f20cc00df10 legacy=0x7f20e8139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 monclient: found mon.a 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e8127210 con 0x7f20cc01b600 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: _finish_auth 0 2024-08-19T10:08:03.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 5 monclient: authenticate success, global_id 204516 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 10 monclient: _renew_subs 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20e8125a10 con 0x7f20cc01b600 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 10 client.204516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 10 monclient: _renew_subs 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:03.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f20e813a140 con 0x7f20cc01b600 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20efb2f640 1 librados: init done 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20e0003c40 con 0x7f20cc01b600 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f20e0004130 con 0x7f20cc01b600 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20dffff640 4 set_mon_vals no callback set 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.839+0000 7f20de7fc640 10 monclient: dump: 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:03.853 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.840+0000 7f20efb2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.840+0000 7f20efb2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.840+0000 7f20efb2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.840+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d0005080 con 0x7f20cc01b600 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.840+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f20e00042d0 con 0x7f20cc01b600 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.841+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7f20e004e2a0 con 0x7f20cc01b600 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.841+0000 7f20de7fc640 10 client.204516.objecter ms_dispatch 0x7f20e8000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.841+0000 7f20de7fc640 3 client.204516.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:08:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.841+0000 7f20de7fc640 3 client.204516.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:08:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.841+0000 7f20de7fc640 20 client.204516.objecter dump_active .. 0 homeless 2024-08-19T10:08:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.841+0000 7f20ed8cb640 10 monclient: get_auth_request con 0x7f20cc044950 auth_method 0 2024-08-19T10:08:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.843+0000 7f20de7fc640 10 client.204516.objecter ms_handle_connect 0x7f20cc044950 2024-08-19T10:08:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.846+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f20e8126970 con 0x7f20cc01b600 2024-08-19T10:08:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.846+0000 7f20de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:03.846+0000 7f20de7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:04.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20efb2f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20efb2f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20efb2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f20d0005370 con 0x7f20cc01b600 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20de7fc640 1 -- 172.21.15.1:0/1270499863 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2697 (unknown 254746657 0 1632298564) 0x7f20e00174a0 con 0x7f20cc01b600 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.110+0000 7f20de7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:04.112 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":146,"quorum":[0,2,3,4,5,6,7,8],"quorum_names":["a","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:08:04.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.112+0000 7f20efb2f640 10 librados: watch_flush enter 2024-08-19T10:08:04.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.112+0000 7f20efb2f640 10 librados: watch_flush exit 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 20 client.204516.objecter shutdown clearing up homeless session... 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 10 client.204516.objecter successfully canceled tick 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.82:6812/2549104415 conn(0x7f20cc044950 legacy=0x7f20cc046e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 10 monclient: shutdown 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 >> v1:172.21.15.1:6789/0 conn(0x7f20cc01b600 legacy=0x7f20e81d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 shutdown_connections 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 >> 172.21.15.1:0/1270499863 conn(0x7f20e80a9020 msgr2=0x7f20e80a5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 shutdown_connections 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 -- 172.21.15.1:0/1270499863 wait complete. 2024-08-19T10:08:04.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.113+0000 7f20efb2f640 1 librados: shutdown 2024-08-19T10:08:04.125 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T10:08:04.125 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-08-19T10:08:04.125 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.264+0000 7f7459e38640 10 monclient: get_monmap_and_config 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.264+0000 7f7459e38640 10 monclient: build_initial_monmap 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.264+0000 7f7459e38640 10 monclient: monmap: 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:04.265095+0000 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:04.265095+0000 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:04.265 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:04.266 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.264+0000 7f7459e38640 1 Processor -- start 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.264+0000 7f7459e38640 1 -- start start 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.264+0000 7f7459e38640 10 monclient: init 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient: _add_conns ranks=[2,6,0,5,7,8,4,1,3] 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient(hunting): picked mon.noname-b con 0x7f74540a4590 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient(hunting): picked mon.noname-d con 0x7f74540a5080 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient(hunting): picked mon.noname-a con 0x7f745413fcd0 addr v1:172.21.15.1:6789/0 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7454074790 con 0x7f745413fcd0 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f745406cdf0 con 0x7f74540a4590 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74540a2ce0 con 0x7f74540a5080 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7459e38640 10 monclient(hunting): authenticate will time out at 1740.341444s 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7452ffd640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f74540a5080 0x7f745413e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:57288/0 (socket says 172.21.15.1:57288) 2024-08-19T10:08:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f74537fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f74540a4590 0x7f74540a4970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53698/0 (socket says 172.21.15.1:53698) 2024-08-19T10:08:04.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7452ffd640 1 -- 172.21.15.1:0/1632163339 learned_addr learned my addr 172.21.15.1:0/1632163339 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:04.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.265+0000 7f7453fff640 1 --1- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6789/0 conn(0x7f745413fcd0 0x7f74541420c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34496/0 (socket says 172.21.15.1:34496) 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f7452ffd640 1 -- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6792/0 conn(0x7f74540a5080 legacy=0x7f745413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f7453fff640 1 -- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6789/0 conn(0x7f745413fcd0 legacy=0x7f74541420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74537fe640 1 -- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6790/0 conn(0x7f74540a4590 legacy=0x7f74540a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435862486 0 0) 0x7f744c001700 con 0x7f745413fcd0 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74527fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74527fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74527fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74527fc640 10 monclient(hunting): my global_id is 204525 2024-08-19T10:08:04.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.267+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f743c003520 con 0x7f745413fcd0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3795184174 0 0) 0x7f743c003520 con 0x7f745413fcd0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6792/0 conn(0x7f74540a5080 legacy=0x7f745413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6790/0 conn(0x7f74540a4590 legacy=0x7f74540a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: _finish_hunting 0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 monclient: found mon.noname-a 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74540a2fc0 con 0x7f745413fcd0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: _finish_auth 0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: _check_auth_tickets 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f744c001f60 con 0x7f745413fcd0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: dump: 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.274 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 5 monclient: authenticate success, global_id 204525 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f744c004ba0 con 0x7f745413fcd0 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 4 set_mon_vals no callback set 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 1 -- 172.21.15.1:0/1632163339 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f744c005150 con 0x7f745413fcd0 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f74527fc640 10 monclient: dump: 2024-08-19T10:08:04.275 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.276 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 10 monclient: shutdown 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 1 -- 172.21.15.1:0/1632163339 >> v1:172.21.15.1:6789/0 conn(0x7f745413fcd0 legacy=0x7f74541420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 1 -- 172.21.15.1:0/1632163339 shutdown_connections 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 1 -- 172.21.15.1:0/1632163339 >> 172.21.15.1:0/1632163339 conn(0x7f74540a5f60 msgr2=0x7f7454068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 1 -- 172.21.15.1:0/1632163339 shutdown_connections 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.268+0000 7f7459e38640 1 -- 172.21.15.1:0/1632163339 wait complete. 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 10 monclient: build_initial_monmap 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 10 monclient: monmap: 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:04.277 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 1 librados: starting msgr at 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 1 librados: starting objecter 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 1 Processor -- start 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 1 -- start start 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 1 librados: setting wanted keys 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 1 librados: calling monclient init 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.269+0000 7f7459e38640 10 monclient: init 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient: _add_conns ranks=[5,7,3,2,8,0,4,1,6] 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient(hunting): picked mon.noname-h con 0x7f74540a4590 addr v1:172.21.15.82:6791/0 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient(hunting): picked mon.noname-i con 0x7f74540a5080 addr v1:172.21.15.82:6792/0 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient(hunting): picked mon.noname-g con 0x7f745413fcd0 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7454126db0 con 0x7f745413fcd0 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7454124a50 con 0x7f74540a4590 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7454126510 con 0x7f74540a5080 2024-08-19T10:08:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7459e38640 10 monclient(hunting): authenticate will time out at 1740.346365s 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7452ffd640 1 --1- >> v1:172.21.15.82:6792/0 conn(0x7f74540a5080 0x7f74541d3250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6792/0 says I am v1:172.21.15.1:59784/0 (socket says 172.21.15.1:59784) 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.270+0000 7f7452ffd640 1 -- 172.21.15.1:0/2424091568 learned_addr learned my addr 172.21.15.1:0/2424091568 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f7452ffd640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6792/0 conn(0x7f74540a5080 legacy=0x7f74541d3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f7453fff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6790/0 conn(0x7f745413fcd0 legacy=0x7f74541d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f743bfff640 10 client.?.objecter ms_handle_connect 0x7f74540a5080 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f743bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:04.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f743bfff640 10 client.?.objecter ms_handle_connect 0x7f745413fcd0 2024-08-19T10:08:04.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f743bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:04.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f743bfff640 10 client.?.objecter ms_handle_connect 0x7f74540a4590 2024-08-19T10:08:04.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f743bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:04.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.272+0000 7f74537fe640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6791/0 conn(0x7f74540a4590 legacy=0x7f745413b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747580009 0 0) 0x7f7454126510 con 0x7f74540a5080 2024-08-19T10:08:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): my global_id is 204496 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7440004eb0 con 0x7f74540a5080 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104533239 0 0) 0x7f7454124a50 con 0x7f74540a4590 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): my global_id is 214277 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7454126510 con 0x7f74540a4590 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550716812 0 0) 0x7f7454126db0 con 0x7f745413fcd0 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 10 monclient(hunting): my global_id is 214266 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.794+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7454124a50 con 0x7f745413fcd0 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2020097318 0 0) 0x7f7440004eb0 con 0x7f74540a5080 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7454126db0 con 0x7f74540a5080 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2157283982 0 0) 0x7f7454126510 con 0x7f74540a4590 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7440004eb0 con 0x7f74540a4590 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7448001d00 con 0x7f74540a5080 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): dump: 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.799 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:04.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 823227956 0 0) 0x7f7454124a50 con 0x7f745413fcd0 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7454126510 con 0x7f745413fcd0 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7444001c30 con 0x7f74540a4590 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): dump: 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f744c001660 con 0x7f745413fcd0 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient(hunting): dump: 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3293993219 0 0) 0x7f7440004eb0 con 0x7f74540a4590 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6792/0 conn(0x7f74540a5080 legacy=0x7f74541d3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6790/0 conn(0x7f745413fcd0 legacy=0x7f74541d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient: _finish_hunting 0 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 monclient: found mon.h 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745413bb20 con 0x7f74540a4590 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient: _finish_auth 0 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 10 monclient: _check_auth_tickets 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f743bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f7459e38640 5 monclient: authenticate success, global_id 214277 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f7459e38640 10 monclient: _renew_subs 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f7459e38640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:08:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.795+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f74541cdf30 con 0x7f74540a4590 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7444004b00 con 0x7f74540a4590 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f743bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f7459e38640 10 client.214277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f7459e38640 10 monclient: _renew_subs 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7444005070 con 0x7f74540a4590 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f7459e38640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f7454123fa0 con 0x7f74540a4590 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f7459e38640 1 librados: init done 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.796+0000 7f7451ffb640 4 set_mon_vals no callback set 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.797+0000 7f743bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.797+0000 7f743bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.797+0000 7f743bfff640 10 monclient: dump: 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.810 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.797+0000 7f7459e38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.797+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74540008d0 con 0x7f745411f1d0 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.798+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7444005210 con 0x7f74540a4590 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.798+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(657..657 src has 299..657) ==== 5992+0+0 (unknown 1432688115 0 0) 0x7f744404dc10 con 0x7f74540a4590 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.798+0000 7f743bfff640 10 client.214277.objecter ms_dispatch 0x7f7454000b90 osd_map(657..657 src has 299..657) 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.798+0000 7f743bfff640 3 client.214277.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.798+0000 7f743bfff640 3 client.214277.objecter handle_osd_map decoding full epoch 657 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.798+0000 7f743bfff640 20 client.214277.objecter dump_active .. 0 homeless 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.799+0000 7f7453fff640 10 monclient: get_auth_request con 0x7f74400418c0 auth_method 0 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.799+0000 7f743bfff640 10 client.214277.objecter ms_handle_connect 0x7f745411f1d0 2024-08-19T10:08:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.799+0000 7f743bfff640 10 client.214277.objecter resend_mon_ops 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.799+0000 7f7452ffd640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.1:6789/0 conn(0x7f745411f1d0 legacy=0x7f745411f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789506443 0 0) 0x7f74540008d0 con 0x7f745411f1d0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 10 monclient(hunting): my global_id is 204534 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7440004eb0 con 0x7f745411f1d0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.801+0000 7f743bfff640 10 client.214277.objecter ms_handle_connect 0x7f74400418c0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3518759883 0 0) 0x7f7440004eb0 con 0x7f745411f1d0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74540008d0 con 0x7f745411f1d0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7448004ef0 con 0x7f745411f1d0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:08:04.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 10 monclient: dump: 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3121375736 0 0) 0x7f74540008d0 con 0x7f745411f1d0 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.802+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7454124330 con 0x7f745411f1d0 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.803+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f74480051e0 con 0x7f745411f1d0 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.803+0000 7f743bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.803+0000 7f743bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.803+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.1:6789/0 conn(0x7f745411f1d0 legacy=0x7f745411f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.803+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.1:6789/0 conn(0x7f745411f1d0 legacy=0x7f745411f5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.818+0000 7f7459e38640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.818+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74541200a0 con 0x7f745411fcc0 2024-08-19T10:08:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.821+0000 7f743bfff640 10 client.214277.objecter ms_handle_connect 0x7f745411fcc0 2024-08-19T10:08:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.821+0000 7f743bfff640 10 client.214277.objecter resend_mon_ops 2024-08-19T10:08:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.821+0000 7f7452ffd640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.1:6789/0 conn(0x7f745411fcc0 legacy=0x7f74541dc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3406977784 0 0) 0x7f74541200a0 con 0x7f745411fcc0 2024-08-19T10:08:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient(hunting): my global_id is 204543 2024-08-19T10:08:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74540008d0 con 0x7f745411fcc0 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 764680250 0 0) 0x7f74540008d0 con 0x7f745411fcc0 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74541200a0 con 0x7f745411fcc0 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7448005bf0 con 0x7f745411fcc0 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 10 monclient: dump: 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1534665731 0 0) 0x7f74541200a0 con 0x7f745411fcc0 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.824+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7454126880 con 0x7f745411fcc0 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7676 (unknown 0 0 2974981880) 0x7f7448007e10 con 0x7f745411fcc0 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f743bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f743bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.1:6789/0 conn(0x7f745411fcc0 legacy=0x7f74541dc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f743bfff640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.1:6789/0 conn(0x7f745411fcc0 legacy=0x7f74541dc0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 150697, 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:04.829 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:04.830 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:04.831 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:04.832 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:04.833 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:04.838 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.842 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.843 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:04.848 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:04.849 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:04.849 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:04.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f7459e38640 10 librados: watch_flush enter 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f7459e38640 10 librados: watch_flush exit 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f7459e38640 20 client.214277.objecter shutdown clearing up homeless session... 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.825+0000 7f7459e38640 10 client.214277.objecter successfully canceled tick 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6812/2549104415 conn(0x7f74400418c0 legacy=0x7f7440043d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 10 monclient: shutdown 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 >> v1:172.21.15.82:6791/0 conn(0x7f74540a4590 legacy=0x7f745413b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7453fff640 1 -- 172.21.15.1:0/2424091568 reap_dead start 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7453fff640 1 -- 172.21.15.1:0/2424091568 reap_dead start 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 shutdown_connections 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 >> 172.21.15.1:0/2424091568 conn(0x7f74540a5f60 msgr2=0x7f745413e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 shutdown_connections 2024-08-19T10:08:04.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 -- 172.21.15.1:0/2424091568 wait complete. 2024-08-19T10:08:04.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.826+0000 7f7459e38640 1 librados: shutdown 2024-08-19T10:08:04.852 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:04.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T10:08:04.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:08:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.957+0000 7fe9675c5640 10 monclient: get_monmap_and_config 2024-08-19T10:08:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.957+0000 7fe9675c5640 10 monclient: build_initial_monmap 2024-08-19T10:08:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.957+0000 7fe9675c5640 10 monclient: monmap: 2024-08-19T10:08:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:04.958099+0000 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:04.958099+0000 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:04.959 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.957+0000 7fe9675c5640 1 Processor -- start 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 1 -- start start 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient: init 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient: _add_conns ranks=[8,3,4,2,1,7,6,5,0] 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient(hunting): picked mon.noname-e con 0x7fe9600a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient(hunting): picked mon.noname-g con 0x7fe9600a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient(hunting): picked mon.noname-c con 0x7fe96013f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9600a3b40 con 0x7fe9600a6ba0 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9600a8130 con 0x7fe96013f5c0 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96006ede0 con 0x7fe9600a67c0 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe9675c5640 10 monclient(hunting): authenticate will time out at 1741.034648s 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe965361640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe9600a67c0 0x7fe9600a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:51604/0 (socket says 172.21.15.1:51604) 2024-08-19T10:08:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.958+0000 7fe965361640 1 -- 172.21.15.1:0/4110623852 learned_addr learned my addr 172.21.15.1:0/4110623852 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.960+0000 7fe965361640 1 -- 172.21.15.1:0/4110623852 >> v1:172.21.15.1:6793/0 conn(0x7fe9600a67c0 legacy=0x7fe9600a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.960+0000 7fe965b62640 1 -- 172.21.15.1:0/4110623852 >> v1:172.21.15.1:6791/0 conn(0x7fe96013f5c0 legacy=0x7fe9601419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 906808999 0 0) 0x7fe9600a8130 con 0x7fe96013f5c0 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): my global_id is 194692 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe948003520 con 0x7fe96013f5c0 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235993230 0 0) 0x7fe96006ede0 con 0x7fe9600a67c0 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): my global_id is 204533 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9600a8130 con 0x7fe9600a67c0 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe964b60640 1 -- 172.21.15.1:0/4110623852 >> v1:172.21.15.82:6790/0 conn(0x7fe9600a6ba0 legacy=0x7fe9600a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553639287 0 0) 0x7fe9600a3b40 con 0x7fe9600a6ba0 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient(hunting): my global_id is 214275 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe96006ede0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2800752602 0 0) 0x7fe948003520 con 0x7fe96013f5c0 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 >> v1:172.21.15.1:6793/0 conn(0x7fe9600a67c0 legacy=0x7fe9600a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 >> v1:172.21.15.82:6790/0 conn(0x7fe9600a6ba0 legacy=0x7fe9600a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:08:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 monclient: found mon.noname-c 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9600a92a0 con 0x7fe96013f5c0 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: _finish_auth 0 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe950001de0 con 0x7fe96013f5c0 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 5 monclient: authenticate success, global_id 194692 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: dump: 2024-08-19T10:08:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9500020b0 con 0x7fe96013f5c0 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 1 -- 172.21.15.1:0/4110623852 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe950003cd0 con 0x7fe96013f5c0 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 4 set_mon_vals no callback set 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe95ffff640 10 monclient: dump: 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.970 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 10 monclient: shutdown 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 1 -- 172.21.15.1:0/4110623852 >> v1:172.21.15.1:6791/0 conn(0x7fe96013f5c0 legacy=0x7fe9601419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 1 -- 172.21.15.1:0/4110623852 shutdown_connections 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.961+0000 7fe9675c5640 1 -- 172.21.15.1:0/4110623852 >> 172.21.15.1:0/4110623852 conn(0x7fe9600ac4b0 msgr2=0x7fe9600a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 -- 172.21.15.1:0/4110623852 shutdown_connections 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 -- 172.21.15.1:0/4110623852 wait complete. 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 10 monclient: build_initial_monmap 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 10 monclient: monmap: 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 librados: starting msgr at 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 librados: starting objecter 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 Processor -- start 2024-08-19T10:08:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 -- start start 2024-08-19T10:08:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 librados: setting wanted keys 2024-08-19T10:08:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 1 librados: calling monclient init 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.962+0000 7fe9675c5640 10 monclient: init 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient: _add_conns ranks=[2,7,5,1,0,6,8,3,4] 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient(hunting): picked mon.noname-b con 0x7fe9600a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient(hunting): picked mon.noname-i con 0x7fe9600a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient(hunting): picked mon.noname-h con 0x7fe96013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96013bc00 con 0x7fe9600a67c0 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96013bee0 con 0x7fe96013f5c0 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9600a9de0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe9675c5640 10 monclient(hunting): authenticate will time out at 1741.039416s 2024-08-19T10:08:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe965361640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fe9600a67c0 0x7fe9601ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53710/0 (socket says 172.21.15.1:53710) 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.963+0000 7fe965361640 1 -- 172.21.15.1:0/1196138424 learned_addr learned my addr 172.21.15.1:0/1196138424 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe965361640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe9600a67c0 legacy=0x7fe9601ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 client.?.objecter ms_handle_connect 0x7fe9600a67c0 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 client.?.objecter ms_handle_connect 0x7fe9600a6ba0 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe964b60640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.82:6792/0 conn(0x7fe9600a6ba0 legacy=0x7fe9601d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe965361640 1 --1- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe9600a67c0 0x7fe9601ce4a0 :-1 s=OPENED pgs=3650 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fe96013bc00 auth_reply(proto 2 0 (0) Success) 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 client.?.objecter ms_handle_connect 0x7fe96013f5c0 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe965b62640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.82:6791/0 conn(0x7fe96013f5c0 legacy=0x7fe9601d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650971134 0 0) 0x7fe9600a9de0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): my global_id is 204505 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe94c004eb0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744145332 0 0) 0x7fe96013bee0 con 0x7fe96013f5c0 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 10 monclient(hunting): my global_id is 214286 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.965+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9600a9de0 con 0x7fe96013f5c0 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3986671174 0 0) 0x7fe94c004eb0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe96013bee0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe954003610 con 0x7fe9600a6ba0 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): dump: 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3240946316 0 0) 0x7fe9600a9de0 con 0x7fe96013f5c0 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe94c004eb0 con 0x7fe96013f5c0 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe950003cd0 con 0x7fe96013f5c0 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient(hunting): dump: 2024-08-19T10:08:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1695829348 0 0) 0x7fe96013bee0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.82:6791/0 conn(0x7fe96013f5c0 legacy=0x7fe9601d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe9600a67c0 legacy=0x7fe9601ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 monclient: found mon.i 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe960142cb0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: _finish_auth 0 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:08:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 5 monclient: authenticate success, global_id 204505 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484739728 0 0) 0x7fe96013bc00 con 0x7fe9600a67c0 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 10 monclient: _renew_subs 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe96013a6e0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe954003b90 con 0x7fe9600a6ba0 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe9540040c0 con 0x7fe9600a6ba0 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95dffb640 10 monclient: dump: 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 10 client.204505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 10 monclient: _renew_subs 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.82:6792/0 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.82:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe96013a440 con 0x7fe9600a6ba0 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe95f7fe640 4 set_mon_vals no callback set 2024-08-19T10:08:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.966+0000 7fe9675c5640 1 librados: init done 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe954004260 con 0x7fe9600a6ba0 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe9675c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe960143050 con 0x7fe96013d380 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.7 v1:172.21.15.82:6792/0 8 ==== osd_map(658..658 src has 299..658) ==== 5987+0+0 (unknown 879229092 0 0) 0x7fe95404e480 con 0x7fe9600a6ba0 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe95dffb640 10 client.204505.objecter ms_dispatch 0x7fe960000b90 osd_map(658..658 src has 299..658) 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe95dffb640 3 client.204505.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe95dffb640 3 client.204505.objecter handle_osd_map decoding full epoch 658 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe95dffb640 20 client.204505.objecter dump_active .. 0 homeless 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.967+0000 7fe965b62640 10 monclient: get_auth_request con 0x7fe94c041550 auth_method 0 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 client.204505.objecter ms_handle_connect 0x7fe96013d380 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 client.204505.objecter resend_mon_ops 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe965361640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe96013d380 legacy=0x7fe96009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1950065187 0 0) 0x7fe960143050 con 0x7fe96013d380 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient(hunting): my global_id is 184880 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe96013bc00 con 0x7fe96013d380 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 152206260 0 0) 0x7fe96013bc00 con 0x7fe96013d380 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe960143050 con 0x7fe96013d380 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe958001b00 con 0x7fe96013d380 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 10 monclient: dump: 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.993 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.969+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4051297497 0 0) 0x7fe960143050 con 0x7fe96013d380 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe96013d760 con 0x7fe96013d380 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 10 client.204505.objecter ms_handle_connect 0x7fe94c041550 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe9580068e0 con 0x7fe96013d380 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe96013d380 legacy=0x7fe96009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.970+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe96013d380 legacy=0x7fe96009da50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.976+0000 7fe9675c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.976+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe9601cef90 con 0x7fe9601cebb0 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 10 client.204505.objecter ms_handle_connect 0x7fe9601cebb0 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 10 client.204505.objecter resend_mon_ops 2024-08-19T10:08:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe965361640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe9601cebb0 legacy=0x7fe96009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095260544 0 0) 0x7fe9601cef90 con 0x7fe9601cebb0 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 10 monclient(hunting): my global_id is 184889 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.978+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe960143050 con 0x7fe9601cebb0 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2936132460 0 0) 0x7fe960143050 con 0x7fe9601cebb0 2024-08-19T10:08:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9601cef90 con 0x7fe9601cebb0 2024-08-19T10:08:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe958005170 con 0x7fe9601cebb0 2024-08-19T10:08:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:08:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 10 monclient: dump: 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1394491331 0 0) 0x7fe9601cef90 con 0x7fe9601cebb0 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe960068f60 con 0x7fe9601cebb0 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7674 (unknown 0 0 3876575260) 0x7fe9580065f0 con 0x7fe9601cebb0 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe9601cebb0 legacy=0x7fe96009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe95dffb640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.1:6790/0 conn(0x7fe9601cebb0 legacy=0x7fe96009e160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 402227, 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:04.999 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.001 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:05.002 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:05.003 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.004 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:05.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:05.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:08:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:05.020 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe9675c5640 10 librados: watch_flush enter 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe9675c5640 10 librados: watch_flush exit 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe9675c5640 20 client.204505.objecter shutdown clearing up homeless session... 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe9675c5640 10 client.204505.objecter successfully canceled tick 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.979+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe94c041550 legacy=0x7fe94c043a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 10 monclient: shutdown 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 >> v1:172.21.15.82:6792/0 conn(0x7fe9600a6ba0 legacy=0x7fe9601d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe965b62640 1 -- 172.21.15.1:0/1196138424 reap_dead start 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe965b62640 1 -- 172.21.15.1:0/1196138424 reap_dead start 2024-08-19T10:08:05.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 shutdown_connections 2024-08-19T10:08:05.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 >> 172.21.15.1:0/1196138424 conn(0x7fe9600ac4b0 msgr2=0x7fe96013ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 shutdown_connections 2024-08-19T10:08:05.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 1 -- 172.21.15.1:0/1196138424 wait complete. 2024-08-19T10:08:05.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:04.980+0000 7fe9675c5640 1 librados: shutdown 2024-08-19T10:08:05.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:05.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T10:08:05.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:08:05.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.121+0000 7ff8a1b81640 10 monclient: get_monmap_and_config 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.121+0000 7ff8a1b81640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.121+0000 7ff8a1b81640 10 monclient: monmap: 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:05.122549+0000 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:05.122549+0000 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.124 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 1 Processor -- start 2024-08-19T10:08:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 1 -- start start 2024-08-19T10:08:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 10 monclient: init 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 10 monclient: _add_conns ranks=[7,4,3,0,5,8,1,2,6] 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 10 monclient(hunting): picked mon.noname-i con 0x7ff89c0a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 10 monclient(hunting): picked mon.noname-c con 0x7ff89c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 10 monclient(hunting): picked mon.noname-g con 0x7ff89c13f5c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.122+0000 7ff8a1b81640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89c0a3b40 con 0x7ff89c13f5c0 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.123+0000 7ff8a1b81640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89c0a8130 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.123+0000 7ff8a1b81640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89c06ede0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.123+0000 7ff8a1b81640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.123+0000 7ff8a1b81640 10 monclient(hunting): authenticate will time out at 1741.199035s 2024-08-19T10:08:05.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.123+0000 7ff89affd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff89c0a6ba0 0x7ff89c0a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37694/0 (socket says 172.21.15.1:37694) 2024-08-19T10:08:05.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.123+0000 7ff89affd640 1 -- 172.21.15.1:0/3279530742 learned_addr learned my addr 172.21.15.1:0/3279530742 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89affd640 1 -- 172.21.15.1:0/3279530742 >> v1:172.21.15.1:6791/0 conn(0x7ff89c0a6ba0 legacy=0x7ff89c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499593240 0 0) 0x7ff89c0a8130 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): my global_id is 194701 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff888003520 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89b7fe640 1 -- 172.21.15.1:0/3279530742 >> v1:172.21.15.82:6792/0 conn(0x7ff89c0a67c0 legacy=0x7ff89c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89bfff640 1 -- 172.21.15.1:0/3279530742 >> v1:172.21.15.82:6790/0 conn(0x7ff89c13f5c0 legacy=0x7ff89c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306008268 0 0) 0x7ff89c06ede0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient(hunting): my global_id is 204514 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff89c0a8130 con 0x7ff89c0a67c0 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 469704659 0 0) 0x7ff888003520 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 >> v1:172.21.15.82:6792/0 conn(0x7ff89c0a67c0 legacy=0x7ff89c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 >> v1:172.21.15.82:6790/0 conn(0x7ff89c13f5c0 legacy=0x7ff89c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 monclient: found mon.noname-c 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89c0a92a0 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff884001ae0 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff8a1b81640 5 monclient: authenticate success, global_id 194701 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff8a1b81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: dump: 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.133 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff8a1b81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff8840048a0 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 1 -- 172.21.15.1:0/3279530742 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff884004e10 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff8a1b81640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff8a1b81640 4 set_mon_vals no callback set 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.125+0000 7ff89a7fc640 10 monclient: dump: 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.135 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.136 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 10 monclient: shutdown 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 1 -- 172.21.15.1:0/3279530742 >> v1:172.21.15.1:6791/0 conn(0x7ff89c0a6ba0 legacy=0x7ff89c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 1 -- 172.21.15.1:0/3279530742 shutdown_connections 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 1 -- 172.21.15.1:0/3279530742 >> 172.21.15.1:0/3279530742 conn(0x7ff89c0ac4b0 msgr2=0x7ff89c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 1 -- 172.21.15.1:0/3279530742 shutdown_connections 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 1 -- 172.21.15.1:0/3279530742 wait complete. 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.126+0000 7ff8a1b81640 10 monclient: monmap: 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 librados: starting msgr at 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 librados: starting objecter 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 Processor -- start 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 -- start start 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 librados: setting wanted keys 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 librados: calling monclient init 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient: init 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient: _add_conns ranks=[0,4,6,7,2,3,1,5,8] 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient(hunting): picked mon.noname-a con 0x7ff89c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient(hunting): picked mon.noname-c con 0x7ff89c0a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient(hunting): picked mon.noname-d con 0x7ff89c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89c0a9cc0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89c13bbc0 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89c13bea0 con 0x7ff89c13f5c0 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.127+0000 7ff8a1b81640 10 monclient(hunting): authenticate will time out at 1741.203971s 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.128+0000 7ff89affd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7ff89c0a6ba0 0x7ff89c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37710/0 (socket says 172.21.15.1:37710) 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.128+0000 7ff89b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 0x7ff89c1ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34530/0 (socket says 172.21.15.1:34530) 2024-08-19T10:08:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.128+0000 7ff89affd640 1 -- 172.21.15.1:0/1434925238 learned_addr learned my addr 172.21.15.1:0/1434925238 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff89b7fe640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 legacy=0x7ff89c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 client.?.objecter ms_handle_connect 0x7ff89c0a67c0 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff89bfff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6792/0 conn(0x7ff89c13f5c0 legacy=0x7ff89c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 client.?.objecter ms_handle_connect 0x7ff89c13f5c0 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 client.?.objecter ms_handle_connect 0x7ff89c0a6ba0 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff89affd640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c0a6ba0 legacy=0x7ff89c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814673659 0 0) 0x7ff89c0a8130 con 0x7ff89c0a67c0 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): my global_id is 204552 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff87c004ee0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2767088669 0 0) 0x7ff89c13bbc0 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): my global_id is 194710 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff89c0a8130 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 265377798 0 0) 0x7ff87c004ee0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff89c13bbc0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff88c003990 con 0x7ff89c0a67c0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): dump: 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3523761734 0 0) 0x7ff89c0a8130 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff87c004ee0 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff884004d90 con 0x7ff89c0a6ba0 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient(hunting): dump: 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 594107536 0 0) 0x7ff89c13bbc0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6792/0 conn(0x7ff89c13f5c0 legacy=0x7ff89c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c0a6ba0 legacy=0x7ff89c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 monclient: found mon.a 2024-08-19T10:08:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89c1cf020 con 0x7ff89c0a67c0 2024-08-19T10:08:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff8a1b81640 5 monclient: authenticate success, global_id 204552 2024-08-19T10:08:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff8a1b81640 10 monclient: _renew_subs 2024-08-19T10:08:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff8a1b81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.130+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff89c13a6a0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 10 client.204552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 10 monclient: _renew_subs 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff89c13e390 con 0x7ff89c0a67c0 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff88c003ec0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 1 librados: init done 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff89b7fe640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 legacy=0x7ff89c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff89b7fe640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 legacy=0x7ff89c1ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff89b7fe640 1 --1- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 0x7ff89c1ce200 :-1 s=OPENED pgs=859 cs=1 l=1).handle_message read tag failed 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff88c0043f0 con 0x7ff89c0a67c0 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff89b7fe640 1 --1- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 0x7ff89c1ce200 :-1 s=OPENED pgs=859 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: dump: 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.1:6789/0 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0a67c0 legacy=0x7ff89c1ce200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff899ffb640 4 set_mon_vals no callback set 2024-08-19T10:08:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient: _add_conns ranks=[3,5,6,2,1,8,4,0,7] 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient(hunting): picked mon.g con 0x7ff87c007870 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient(hunting): picked mon.h con 0x7ff87c00b430 addr v1:172.21.15.82:6791/0 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient(hunting): picked mon.d con 0x7ff89c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff89c13bbc0 con 0x7ff87c007870 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff89c0a8130 con 0x7ff87c00b430 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff87c015190 con 0x7ff89c13f5c0 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff893fff640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.131+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff89c142540 con 0x7ff89c09da50 2024-08-19T10:08:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.133+0000 7ff893fff640 10 client.204552.objecter ms_handle_connect 0x7ff89c13f5c0 2024-08-19T10:08:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.133+0000 7ff89bfff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6792/0 conn(0x7ff89c13f5c0 legacy=0x7ff87c011560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.133+0000 7ff893fff640 10 client.204552.objecter resend_mon_ops 2024-08-19T10:08:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.133+0000 7ff89b7fe640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c09da50 legacy=0x7ff89c09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.133+0000 7ff893fff640 10 client.204552.objecter ms_handle_connect 0x7ff89c09da50 2024-08-19T10:08:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 client.204552.objecter resend_mon_ops 2024-08-19T10:08:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 client.204552.objecter ms_handle_connect 0x7ff87c007870 2024-08-19T10:08:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 client.204552.objecter resend_mon_ops 2024-08-19T10:08:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 client.204552.objecter ms_handle_connect 0x7ff87c00b430 2024-08-19T10:08:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 client.204552.objecter resend_mon_ops 2024-08-19T10:08:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff89b7fe640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.82:6790/0 conn(0x7ff87c007870 legacy=0x7ff87c009c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff89affd640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.82:6791/0 conn(0x7ff87c00b430 legacy=0x7ff87c00d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195275388 0 0) 0x7ff89c13bbc0 con 0x7ff89c09da50 2024-08-19T10:08:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): my global_id is 194719 2024-08-19T10:08:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff87c0131a0 con 0x7ff89c09da50 2024-08-19T10:08:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2189663623 0 0) 0x7ff87c0131a0 con 0x7ff89c09da50 2024-08-19T10:08:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff89c13bbc0 con 0x7ff89c09da50 2024-08-19T10:08:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff88c006280 con 0x7ff89c09da50 2024-08-19T10:08:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): dump: 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.166 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3504484988 0 0) 0x7ff89c0a8130 con 0x7ff87c00b430 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.82:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7ff87c0131a0 con 0x7ff87c00b430 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864958582 0 0) 0x7ff89c13bbc0 con 0x7ff87c007870 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T10:08:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.82:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7ff89c0a8130 con 0x7ff87c007870 2024-08-19T10:08:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2459611798 0 0) 0x7ff89c13e390 con 0x7ff89c09da50 2024-08-19T10:08:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff89c0008d0 con 0x7ff89c09da50 2024-08-19T10:08:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1300839402 0 0) 0x7ff87c0131a0 con 0x7ff87c00b430 2024-08-19T10:08:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6792/0 conn(0x7ff89c13f5c0 legacy=0x7ff87c011560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.82:6790/0 conn(0x7ff87c007870 legacy=0x7ff87c009c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 monclient: found mon.h 2024-08-19T10:08:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:08:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7ff87c0165b0 con 0x7ff87c00b430 2024-08-19T10:08:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff884001d30 con 0x7ff87c00b430 2024-08-19T10:08:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: dump: 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.5 v1:172.21.15.82:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff884001a80 con 0x7ff87c00b430 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff88c0084a0 con 0x7ff89c09da50 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c09da50 legacy=0x7ff89c09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c09da50 legacy=0x7ff89c09de30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff89bfff640 1 -- 172.21.15.1:0/1434925238 reap_dead start 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.134+0000 7ff8997fa640 4 set_mon_vals no callback set 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.5 v1:172.21.15.82:6791/0 5 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ff884011750 con 0x7ff87c00b430 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.5 v1:172.21.15.82:6791/0 6 ==== osd_map(658..658 src has 299..658) ==== 5987+0+0 (unknown 879229092 0 0) 0x7ff88404e0d0 con 0x7ff87c00b430 2024-08-19T10:08:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff893fff640 10 client.204552.objecter ms_dispatch 0x7ff89c000b90 osd_map(658..658 src has 299..658) 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff893fff640 3 client.204552.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff893fff640 3 client.204552.objecter handle_osd_map decoding full epoch 658 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff893fff640 20 client.204552.objecter dump_active .. 0 homeless 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.135+0000 7ff89b7fe640 10 monclient: get_auth_request con 0x7ff87c046970 auth_method 0 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.138+0000 7ff893fff640 10 client.204552.objecter ms_handle_connect 0x7ff87c046970 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.141+0000 7ff8a1b81640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.141+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff89c0a2870 con 0x7ff89c13b600 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff89bfff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c13b600 legacy=0x7ff89c1d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 client.204552.objecter ms_handle_connect 0x7ff89c13b600 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 client.204552.objecter resend_mon_ops 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154053183 0 0) 0x7ff89c0a2870 con 0x7ff89c13b600 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient(hunting): my global_id is 194728 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff87c0131a0 con 0x7ff89c13b600 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 359485345 0 0) 0x7ff87c0131a0 con 0x7ff89c13b600 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff89c0a2870 con 0x7ff89c13b600 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ff8940019b0 con 0x7ff89c13b600 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.143+0000 7ff893fff640 10 monclient: dump: 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1489981561 0 0) 0x7ff89c0a2870 con 0x7ff89c13b600 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff89c1d3fe0 con 0x7ff89c13b600 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7674 (unknown 0 0 1557906324) 0x7ff8940057c0 con 0x7ff89c13b600 2024-08-19T10:08:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c13b600 legacy=0x7ff89c1d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff893fff640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.1:6791/0 conn(0x7ff89c13b600 legacy=0x7ff89c1d38d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:05.178 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 272831, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.179 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.180 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.181 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:05.182 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.183 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.184 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:05.185 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:05.186 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.187 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:05.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:05.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:05.195 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.196 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.200 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.201 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:05.202 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:05.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 10 librados: watch_flush enter 2024-08-19T10:08:05.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 10 librados: watch_flush exit 2024-08-19T10:08:05.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 20 client.204552.objecter shutdown clearing up homeless session... 2024-08-19T10:08:05.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 10 client.204552.objecter successfully canceled tick 2024-08-19T10:08:05.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.82:6812/2549104415 conn(0x7ff87c046970 legacy=0x7ff87c048e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 10 monclient: shutdown 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 >> v1:172.21.15.82:6791/0 conn(0x7ff87c00b430 legacy=0x7ff87c00d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 shutdown_connections 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 >> 172.21.15.1:0/1434925238 conn(0x7ff89c0ac4b0 msgr2=0x7ff89c141e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 shutdown_connections 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 -- 172.21.15.1:0/1434925238 wait complete. 2024-08-19T10:08:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.144+0000 7ff8a1b81640 1 librados: shutdown 2024-08-19T10:08:05.205 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:05.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-08-19T10:08:05.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.303+0000 7f620159a640 10 monclient: get_monmap_and_config 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.303+0000 7f620159a640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.303+0000 7f620159a640 10 monclient: monmap: 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:05.304814+0000 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:05.304814+0000 2024-08-19T10:08:05.305 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.306 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.304+0000 7f620159a640 1 Processor -- start 2024-08-19T10:08:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.304+0000 7f620159a640 1 -- start start 2024-08-19T10:08:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.304+0000 7f620159a640 10 monclient: init 2024-08-19T10:08:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient: _add_conns ranks=[3,2,8,0,5,7,1,6,4] 2024-08-19T10:08:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient(hunting): picked mon.noname-g con 0x7f61fc0a67c0 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient(hunting): picked mon.noname-b con 0x7f61fc0a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient(hunting): picked mon.noname-e con 0x7f61fc13f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:08:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc0a3b40 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc0a8130 con 0x7f61fc0a67c0 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc06ede0 con 0x7f61fc13f5c0 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f620159a640 10 monclient(hunting): authenticate will time out at 1741.381474s 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f61fb7fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f61fc13f5c0 0x7f61fc1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:51618/0 (socket says 172.21.15.1:51618) 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.305+0000 7f61fb7fe640 1 -- 172.21.15.1:0/2543134562 learned_addr learned my addr 172.21.15.1:0/2543134562 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61faffd640 1 -- 172.21.15.1:0/2543134562 >> v1:172.21.15.82:6790/0 conn(0x7f61fc0a67c0 legacy=0x7f61fc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61fb7fe640 1 -- 172.21.15.1:0/2543134562 >> v1:172.21.15.1:6793/0 conn(0x7f61fc13f5c0 legacy=0x7f61fc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61fa7fc640 1 -- 172.21.15.1:0/2543134562 >> v1:172.21.15.1:6790/0 conn(0x7f61fc0a6ba0 legacy=0x7f61fc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2172110423 0 0) 0x7f61fc0a8130 con 0x7f61fc0a67c0 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): my global_id is 214293 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e4003520 con 0x7f61fc0a67c0 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452306191 0 0) 0x7f61fc06ede0 con 0x7f61fc13f5c0 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.307+0000 7f61f9ffb640 10 monclient(hunting): my global_id is 204542 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc0a8130 con 0x7f61fc13f5c0 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360996455 0 0) 0x7f61fc0a3b40 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient(hunting): my global_id is 184898 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc06ede0 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1723914464 0 0) 0x7f61e4003520 con 0x7f61fc0a67c0 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 >> v1:172.21.15.1:6793/0 conn(0x7f61fc13f5c0 legacy=0x7f61fc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 >> v1:172.21.15.1:6790/0 conn(0x7f61fc0a6ba0 legacy=0x7f61fc0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 monclient: found mon.noname-g 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc0a92a0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61ec001b30 con 0x7f61fc0a67c0 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 5 monclient: authenticate success, global_id 214293 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: dump: 2024-08-19T10:08:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.319 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61ec003610 con 0x7f61fc0a67c0 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 1 -- 172.21.15.1:0/2543134562 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61ec003b80 con 0x7f61fc0a67c0 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 4 set_mon_vals no callback set 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f61f9ffb640 10 monclient: dump: 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.322 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 10 monclient: shutdown 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 1 -- 172.21.15.1:0/2543134562 >> v1:172.21.15.82:6790/0 conn(0x7f61fc0a67c0 legacy=0x7f61fc0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 1 -- 172.21.15.1:0/2543134562 shutdown_connections 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.308+0000 7f620159a640 1 -- 172.21.15.1:0/2543134562 >> 172.21.15.1:0/2543134562 conn(0x7f61fc0ac4b0 msgr2=0x7f61fc0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.309+0000 7f620159a640 1 -- 172.21.15.1:0/2543134562 shutdown_connections 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.309+0000 7f620159a640 1 -- 172.21.15.1:0/2543134562 wait complete. 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.309+0000 7f620159a640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.309+0000 7f620159a640 10 monclient: monmap: 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.309+0000 7f620159a640 1 librados: starting msgr at 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.309+0000 7f620159a640 1 librados: starting objecter 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 Processor -- start 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 -- start start 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 librados: setting wanted keys 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 librados: calling monclient init 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient: init 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient: _add_conns ranks=[4,0,1,3,7,2,5,8,6] 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient(hunting): picked mon.noname-c con 0x7f61fc0a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient(hunting): picked mon.noname-a con 0x7f61fc0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:08:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient(hunting): picked mon.noname-f con 0x7f61fc13f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc0a1870 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc142bd0 con 0x7f61fc13f5c0 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc13a9b0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f620159a640 10 monclient(hunting): authenticate will time out at 1741.386592s 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f61fb7fe640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f61fc13f5c0 legacy=0x7f61fc1d6250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f61faffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f61fc0a67c0 0x7f61fc0a01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37748/0 (socket says 172.21.15.1:37748) 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f61fa7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61fc0a6ba0 0x7f61fc0a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34544/0 (socket says 172.21.15.1:34544) 2024-08-19T10:08:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.310+0000 7f61faffd640 1 -- 172.21.15.1:0/3948188940 learned_addr learned my addr 172.21.15.1:0/3948188940 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61fa7fc640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6789/0 conn(0x7f61fc0a6ba0 legacy=0x7f61fc0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61faffd640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6791/0 conn(0x7f61fc0a67c0 legacy=0x7f61fc0a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 client.?.objecter ms_handle_connect 0x7f61fc0a6ba0 2024-08-19T10:08:05.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 client.?.objecter ms_handle_connect 0x7f61fc0a67c0 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2112971370 0 0) 0x7f61fc13a9b0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 10 monclient(hunting): my global_id is 194737 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e8004eb0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.312+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118412205 0 0) 0x7f61fc0a1870 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): my global_id is 204561 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc13a9b0 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1313809058 0 0) 0x7f61e8004eb0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61fc0a1870 con 0x7f61fc0a67c0 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61ec003b00 con 0x7f61fc0a67c0 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): dump: 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:05.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1616029880 0 0) 0x7f61fc13a9b0 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61e8004eb0 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61f0003610 con 0x7f61fc0a6ba0 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient(hunting): dump: 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.334 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 421541939 0 0) 0x7f61fc0a1870 con 0x7f61fc0a67c0 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.82:6789/0 conn(0x7f61fc13f5c0 legacy=0x7f61fc1d6250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6789/0 conn(0x7f61fc0a6ba0 legacy=0x7f61fc0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 monclient: found mon.c 2024-08-19T10:08:05.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc0a2890 con 0x7f61fc0a67c0 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 5 monclient: authenticate success, global_id 194737 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 10 monclient: _renew_subs 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc13a110 con 0x7f61fc0a67c0 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 10 client.194737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 10 monclient: _renew_subs 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f61fc13f290 con 0x7f61fc0a67c0 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f620159a640 1 librados: init done 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61ec001b50 con 0x7f61fc0a67c0 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61ec0057c0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61f97fa640 4 set_mon_vals no callback set 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.313+0000 7f61e37fe640 10 monclient: dump: 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f620159a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61dc0065c0 con 0x7f61dc002860 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f61ec011c20 con 0x7f61fc0a67c0 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(658..658 src has 299..658) ==== 5987+0+0 (unknown 879229092 0 0) 0x7f61ec04e4d0 con 0x7f61fc0a67c0 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61e37fe640 10 client.194737.objecter ms_dispatch 0x7f61fc000b90 osd_map(658..658 src has 299..658) 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61e37fe640 3 client.194737.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61e37fe640 3 client.194737.objecter handle_osd_map decoding full epoch 658 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61e37fe640 20 client.194737.objecter dump_active .. 0 homeless 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.314+0000 7f61fb7fe640 10 monclient: get_auth_request con 0x7f61e80414a0 auth_method 0 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.316+0000 7f61e37fe640 10 client.194737.objecter ms_handle_connect 0x7f61dc002860 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.316+0000 7f61e37fe640 10 client.194737.objecter resend_mon_ops 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.316+0000 7f61fa7fc640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6792/0 conn(0x7f61dc002860 legacy=0x7f61dc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.317+0000 7f61e37fe640 10 client.194737.objecter ms_handle_connect 0x7f61e80414a0 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462711291 0 0) 0x7f61dc0065c0 con 0x7f61dc002860 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient(hunting): my global_id is 204423 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc0a1870 con 0x7f61dc002860 2024-08-19T10:08:05.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3371027892 0 0) 0x7f61fc0a1870 con 0x7f61dc002860 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61dc0065c0 con 0x7f61dc002860 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61f0003d60 con 0x7f61dc002860 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 10 monclient: dump: 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3512867287 0 0) 0x7f61dc0065c0 con 0x7f61dc002860 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.566+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61dc00a160 con 0x7f61dc002860 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.567+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f61f0003ff0 con 0x7f61dc002860 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.567+0000 7f61e37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.567+0000 7f61e37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:05.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.567+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6792/0 conn(0x7f61dc002860 legacy=0x7f61dc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.567+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6792/0 conn(0x7f61dc002860 legacy=0x7f61dc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.573+0000 7f620159a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:05.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.573+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61dc009e70 con 0x7f61dc009130 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.575+0000 7f61fa7fc640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6792/0 conn(0x7f61dc009130 legacy=0x7f61dc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.575+0000 7f61e37fe640 10 client.194737.objecter ms_handle_connect 0x7f61dc009130 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.575+0000 7f61e37fe640 10 client.194737.objecter resend_mon_ops 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104912983 0 0) 0x7f61dc009e70 con 0x7f61dc009130 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient(hunting): my global_id is 204432 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61dc0065c0 con 0x7f61dc009130 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4022740150 0 0) 0x7f61dc0065c0 con 0x7f61dc009130 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61dc009e70 con 0x7f61dc009130 2024-08-19T10:08:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f61f0004470 con 0x7f61dc009130 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient: dump: 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.579 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1665909324 0 0) 0x7f61dc009e70 con 0x7f61dc009130 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61dc00c410 con 0x7f61dc009130 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7674 (unknown 0 0 3150668113) 0x7f61f0005cb0 con 0x7f61dc009130 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6792/0 conn(0x7f61dc009130 legacy=0x7f61dc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.576+0000 7f61e37fe640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6792/0 conn(0x7f61dc009130 legacy=0x7f61dc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 151447, 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:05.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.582 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:05.583 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:05.584 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:05.585 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.588 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:05.589 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:05.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.590 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.592 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:05.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:05.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.598 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.602 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:05.604 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 10 librados: watch_flush enter 2024-08-19T10:08:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 10 librados: watch_flush exit 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 20 client.194737.objecter shutdown clearing up homeless session... 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 10 client.194737.objecter successfully canceled tick 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.82:6812/2549104415 conn(0x7f61e80414a0 legacy=0x7f61e8043960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 10 monclient: shutdown 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 >> v1:172.21.15.1:6791/0 conn(0x7f61fc0a67c0 legacy=0x7f61fc0a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f61fb7fe640 1 -- 172.21.15.1:0/3948188940 reap_dead start 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f61fb7fe640 1 -- 172.21.15.1:0/3948188940 reap_dead start 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 shutdown_connections 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 >> 172.21.15.1:0/3948188940 conn(0x7f61fc0ac4b0 msgr2=0x7f61fc138750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 shutdown_connections 2024-08-19T10:08:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 -- 172.21.15.1:0/3948188940 wait complete. 2024-08-19T10:08:05.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.577+0000 7f620159a640 1 librados: shutdown 2024-08-19T10:08:05.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:05.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-08-19T10:08:05.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.707+0000 7f9f8ef0b640 10 monclient: get_monmap_and_config 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.707+0000 7f9f8ef0b640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.707+0000 7f9f8ef0b640 10 monclient: monmap: 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:05.708031+0000 2024-08-19T10:08:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:05.708031+0000 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.709 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.707+0000 7f9f8ef0b640 1 Processor -- start 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.707+0000 7f9f8ef0b640 1 -- start start 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.707+0000 7f9f8ef0b640 10 monclient: init 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient: _add_conns ranks=[2,7,8,3,0,1,5,6,4] 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f880a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient(hunting): picked mon.noname-i con 0x7f9f880a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient(hunting): picked mon.noname-e con 0x7f9f8813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f880a3b40 con 0x7f9f880a67c0 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f880a8130 con 0x7f9f880a6ba0 2024-08-19T10:08:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f8806ede0 con 0x7f9f8813f5c0 2024-08-19T10:08:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8ef0b640 10 monclient(hunting): authenticate will time out at 1741.784398s 2024-08-19T10:08:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8d4a8640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f9f8813f5c0 0x7f9f881419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:51628/0 (socket says 172.21.15.1:51628) 2024-08-19T10:08:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8cca7640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f9f880a67c0 0x7f9f880a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53744/0 (socket says 172.21.15.1:53744) 2024-08-19T10:08:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.708+0000 7f9f8d4a8640 1 -- 172.21.15.1:0/4044056446 learned_addr learned my addr 172.21.15.1:0/4044056446 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f8cca7640 1 -- 172.21.15.1:0/4044056446 >> v1:172.21.15.1:6790/0 conn(0x7f9f880a67c0 legacy=0x7f9f880a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f8d4a8640 1 -- 172.21.15.1:0/4044056446 >> v1:172.21.15.1:6793/0 conn(0x7f9f8813f5c0 legacy=0x7f9f881419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f87fff640 1 -- 172.21.15.1:0/4044056446 >> v1:172.21.15.82:6792/0 conn(0x7f9f880a6ba0 legacy=0x7f9f880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89251573 0 0) 0x7f9f880a3b40 con 0x7f9f880a67c0 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): my global_id is 184907 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f74003520 con 0x7f9f880a67c0 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565774969 0 0) 0x7f9f8806ede0 con 0x7f9f8813f5c0 2024-08-19T10:08:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 10 monclient(hunting): my global_id is 204551 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.710+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f880a3b40 con 0x7f9f8813f5c0 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 924985499 0 0) 0x7f9f880a8130 con 0x7f9f880a6ba0 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient(hunting): my global_id is 204523 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f8806ede0 con 0x7f9f880a6ba0 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3945300058 0 0) 0x7f9f880a3b40 con 0x7f9f8813f5c0 2024-08-19T10:08:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 >> v1:172.21.15.82:6792/0 conn(0x7f9f880a6ba0 legacy=0x7f9f880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 >> v1:172.21.15.1:6790/0 conn(0x7f9f880a67c0 legacy=0x7f9f880a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 monclient: found mon.noname-e 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f880a92a0 con 0x7f9f8813f5c0 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f70001b10 con 0x7f9f8813f5c0 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 5 monclient: authenticate success, global_id 204551 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: dump: 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f70001d20 con 0x7f9f8813f5c0 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 1 -- 172.21.15.1:0/4044056446 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f70004be0 con 0x7f9f8813f5c0 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 4 set_mon_vals no callback set 2024-08-19T10:08:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f877fe640 10 monclient: dump: 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 10 monclient: shutdown 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/4044056446 >> v1:172.21.15.1:6793/0 conn(0x7f9f8813f5c0 legacy=0x7f9f881419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/4044056446 shutdown_connections 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/4044056446 >> 172.21.15.1:0/4044056446 conn(0x7f9f880ac4b0 msgr2=0x7f9f880a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/4044056446 shutdown_connections 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.711+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/4044056446 wait complete. 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 10 monclient: monmap: 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 1 librados: starting msgr at 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 1 librados: starting objecter 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 1 Processor -- start 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 1 -- start start 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 1 librados: setting wanted keys 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 1 librados: calling monclient init 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.712+0000 7f9f8ef0b640 10 monclient: init 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient: _add_conns ranks=[6,4,5,1,8,0,2,3,7] 2024-08-19T10:08:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient(hunting): picked mon.noname-d con 0x7f9f880a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f880a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient(hunting): picked mon.noname-h con 0x7f9f8813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f8813be10 con 0x7f9f880a6ba0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f881cf020 con 0x7f9f8813f5c0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f88142e90 con 0x7f9f880a67c0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8ef0b640 10 monclient(hunting): authenticate will time out at 1741.789467s 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8cca7640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f9f880a67c0 0x7f9f881ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:57318/0 (socket says 172.21.15.1:57318) 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f8cca7640 1 -- 172.21.15.1:0/3838556432 learned_addr learned my addr 172.21.15.1:0/3838556432 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.713+0000 7f9f87fff640 1 --1- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6791/0 conn(0x7f9f880a6ba0 0x7f9f881ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37754/0 (socket says 172.21.15.1:37754) 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f8cca7640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6792/0 conn(0x7f9f880a67c0 legacy=0x7f9f881ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 client.?.objecter ms_handle_connect 0x7f9f880a67c0 2024-08-19T10:08:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 client.?.objecter ms_handle_connect 0x7f9f880a6ba0 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f87fff640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6791/0 conn(0x7f9f880a6ba0 legacy=0x7f9f881ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189080357 0 0) 0x7f9f88142e90 con 0x7f9f880a67c0 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): my global_id is 204441 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f8d4a8640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.82:6791/0 conn(0x7f9f8813f5c0 legacy=0x7f9f881d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f68004eb0 con 0x7f9f880a67c0 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 client.?.objecter ms_handle_connect 0x7f9f8813f5c0 2024-08-19T10:08:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3534321850 0 0) 0x7f9f8813be10 con 0x7f9f880a6ba0 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 10 monclient(hunting): my global_id is 194746 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.715+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f88142e90 con 0x7f9f880a6ba0 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 891574684 0 0) 0x7f9f68004eb0 con 0x7f9f880a67c0 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f8813be10 con 0x7f9f880a67c0 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f800047e0 con 0x7f9f880a67c0 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): dump: 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3656212296 0 0) 0x7f9f88142e90 con 0x7f9f880a6ba0 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f68004eb0 con 0x7f9f880a6ba0 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f78003990 con 0x7f9f880a6ba0 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): dump: 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3404086118 0 0) 0x7f9f881cf020 con 0x7f9f8813f5c0 2024-08-19T10:08:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient(hunting): my global_id is 214295 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f88142e90 con 0x7f9f8813f5c0 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 954643038 0 0) 0x7f9f8813be10 con 0x7f9f880a67c0 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.82:6791/0 conn(0x7f9f8813f5c0 legacy=0x7f9f881d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6791/0 conn(0x7f9f880a6ba0 legacy=0x7f9f881ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 monclient: found mon.d 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:08:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8813afa0 con 0x7f9f880a67c0 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 5 monclient: authenticate success, global_id 204441 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 10 monclient: _renew_subs 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f88139ab0 con 0x7f9f880a67c0 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 10 client.204441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 10 monclient: _renew_subs 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f8813a8e0 con 0x7f9f880a67c0 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f8ef0b640 1 librados: init done 2024-08-19T10:08:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f80004cb0 con 0x7f9f880a67c0 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f80005220 con 0x7f9f880a67c0 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f86ffd640 4 set_mon_vals no callback set 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.716+0000 7f9f857fa640 10 monclient: dump: 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f8ef0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f6c0065c0 con 0x7f9f6c002860 2024-08-19T10:08:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9f800053c0 con 0x7f9f880a67c0 2024-08-19T10:08:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(658..658 src has 299..658) ==== 5987+0+0 (unknown 879229092 0 0) 0x7f9f8004e480 con 0x7f9f880a67c0 2024-08-19T10:08:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f857fa640 10 client.204441.objecter ms_dispatch 0x7f9f88000b90 osd_map(658..658 src has 299..658) 2024-08-19T10:08:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f857fa640 3 client.204441.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T10:08:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f857fa640 3 client.204441.objecter handle_osd_map decoding full epoch 658 2024-08-19T10:08:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f857fa640 20 client.204441.objecter dump_active .. 0 homeless 2024-08-19T10:08:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.717+0000 7f9f8d4a8640 10 monclient: get_auth_request con 0x7f9f68041620 auth_method 0 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 client.204441.objecter ms_handle_connect 0x7f9f6c002860 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 client.204441.objecter resend_mon_ops 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f87fff640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6793/0 conn(0x7f9f6c002860 legacy=0x7f9f6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 913470357 0 0) 0x7f9f6c0065c0 con 0x7f9f6c002860 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient(hunting): my global_id is 204560 2024-08-19T10:08:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f8813be10 con 0x7f9f6c002860 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2267193741 0 0) 0x7f9f8813be10 con 0x7f9f6c002860 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f6c0065c0 con 0x7f9f6c002860 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f780041d0 con 0x7f9f6c002860 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.719+0000 7f9f857fa640 10 monclient: dump: 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1731074641 0 0) 0x7f9f6c0065c0 con 0x7f9f6c002860 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f6c00a160 con 0x7f9f6c002860 2024-08-19T10:08:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 10 client.204441.objecter ms_handle_connect 0x7f9f68041620 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9f78005b40 con 0x7f9f6c002860 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6793/0 conn(0x7f9f6c002860 legacy=0x7f9f6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.720+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6793/0 conn(0x7f9f6c002860 legacy=0x7f9f6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.726+0000 7f9f8ef0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.726+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f6c009e50 con 0x7f9f6c009110 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f87fff640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6793/0 conn(0x7f9f6c009110 legacy=0x7f9f6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 10 client.204441.objecter ms_handle_connect 0x7f9f6c009110 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 10 client.204441.objecter resend_mon_ops 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 764499037 0 0) 0x7f9f6c009e50 con 0x7f9f6c009110 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 10 monclient(hunting): my global_id is 204569 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.728+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f6c0065c0 con 0x7f9f6c009110 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1453753672 0 0) 0x7f9f6c0065c0 con 0x7f9f6c009110 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f6c009e50 con 0x7f9f6c009110 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9f78004720 con 0x7f9f6c009110 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 10 monclient: dump: 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1915280862 0 0) 0x7f9f6c009e50 con 0x7f9f6c009110 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f6c00c3f0 con 0x7f9f6c009110 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7507 (unknown 0 0 684323631) 0x7f9f78006030 con 0x7f9f6c009110 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6793/0 conn(0x7f9f6c009110 legacy=0x7f9f6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f857fa640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6793/0 conn(0x7f9f6c009110 legacy=0x7f9f6c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:05.748 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 151602, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:05.749 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.750 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.751 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:05.752 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.753 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:05.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.759 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.760 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.762 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.763 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:05.764 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.765 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.766 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.767 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.768 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:05.769 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:05.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 10 librados: watch_flush enter 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 10 librados: watch_flush exit 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 20 client.204441.objecter shutdown clearing up homeless session... 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 10 client.204441.objecter successfully canceled tick 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9f68041620 legacy=0x7f9f68043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 10 monclient: shutdown 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 >> v1:172.21.15.1:6792/0 conn(0x7f9f880a67c0 legacy=0x7f9f881ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8d4a8640 1 -- 172.21.15.1:0/3838556432 reap_dead start 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8d4a8640 1 -- 172.21.15.1:0/3838556432 reap_dead start 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 shutdown_connections 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.729+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 >> 172.21.15.1:0/3838556432 conn(0x7f9f880ac4b0 msgr2=0x7f9f88141f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.730+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 shutdown_connections 2024-08-19T10:08:05.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.730+0000 7f9f8ef0b640 1 -- 172.21.15.1:0/3838556432 wait complete. 2024-08-19T10:08:05.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.730+0000 7f9f8ef0b640 1 librados: shutdown 2024-08-19T10:08:05.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:05.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-08-19T10:08:05.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:08:05.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.871+0000 7f7a18e36640 10 monclient: get_monmap_and_config 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.871+0000 7f7a18e36640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.871+0000 7f7a18e36640 10 monclient: monmap: 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:05.872487+0000 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:05.872487+0000 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.874 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.874 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.874 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.874 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 1 Processor -- start 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 1 -- start start 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient: init 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient: _add_conns ranks=[1,2,8,3,5,6,4,0,7] 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient(hunting): picked mon.noname-f con 0x7f7a140a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient(hunting): picked mon.noname-b con 0x7f7a140a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient(hunting): picked mon.noname-e con 0x7f7a1413f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a140a3b40 con 0x7f7a140a67c0 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a140a8130 con 0x7f7a140a6ba0 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1406ede0 con 0x7f7a1413f5c0 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a18e36640 10 monclient(hunting): authenticate will time out at 1741.948710s 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a09d9b640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7a140a6ba0 0x7f7a140a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53756/0 (socket says 172.21.15.1:53756) 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a12d9d640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f7a1413f5c0 0x7f7a141419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:51656/0 (socket says 172.21.15.1:51656) 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a09d9b640 1 -- 172.21.15.1:0/3482430048 learned_addr learned my addr 172.21.15.1:0/3482430048 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.872+0000 7f7a1259c640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f7a140a67c0 legacy=0x7f7a140a3f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.874+0000 7f7a09d9b640 1 -- 172.21.15.1:0/3482430048 >> v1:172.21.15.1:6790/0 conn(0x7f7a140a6ba0 legacy=0x7f7a140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a12d9d640 1 -- 172.21.15.1:0/3482430048 >> v1:172.21.15.1:6793/0 conn(0x7f7a1413f5c0 legacy=0x7f7a141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382686891 0 0) 0x7f7a140a8130 con 0x7f7a140a6ba0 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): my global_id is 184916 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a00003520 con 0x7f7a140a6ba0 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778094763 0 0) 0x7f7a1406ede0 con 0x7f7a1413f5c0 2024-08-19T10:08:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient(hunting): my global_id is 204578 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a140a8130 con 0x7f7a1413f5c0 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1908770302 0 0) 0x7f7a00003520 con 0x7f7a140a6ba0 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 >> v1:172.21.15.1:6793/0 conn(0x7f7a1413f5c0 legacy=0x7f7a141419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 >> v1:172.21.15.82:6789/0 conn(0x7f7a140a67c0 legacy=0x7f7a140a3f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 monclient: found mon.noname-b 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a140a92a0 con 0x7f7a140a6ba0 2024-08-19T10:08:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f79fc001ae0 con 0x7f7a140a6ba0 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 5 monclient: authenticate success, global_id 184916 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: dump: 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79fc0048a0 con 0x7f7a140a6ba0 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 1 -- 172.21.15.1:0/3482430048 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f79fc004e10 con 0x7f7a140a6ba0 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 4 set_mon_vals no callback set 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a11d9b640 10 monclient: dump: 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 10 monclient: shutdown 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.875+0000 7f7a18e36640 1 -- 172.21.15.1:0/3482430048 >> v1:172.21.15.1:6790/0 conn(0x7f7a140a6ba0 legacy=0x7f7a140a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 1 -- 172.21.15.1:0/3482430048 shutdown_connections 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 1 -- 172.21.15.1:0/3482430048 >> 172.21.15.1:0/3482430048 conn(0x7f7a140ac4b0 msgr2=0x7f7a140a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 1 -- 172.21.15.1:0/3482430048 shutdown_connections 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 1 -- 172.21.15.1:0/3482430048 wait complete. 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 10 monclient: build_initial_monmap 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 10 monclient: monmap: 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 1 librados: starting msgr at 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.876+0000 7f7a18e36640 1 librados: starting objecter 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 Processor -- start 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 -- start start 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 librados: setting wanted keys 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 librados: calling monclient init 2024-08-19T10:08:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient: init 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient: _add_conns ranks=[4,5,6,8,1,3,0,7,2] 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient(hunting): picked mon.noname-c con 0x7f7a140a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient(hunting): picked mon.noname-h con 0x7f7a140a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient(hunting): picked mon.noname-d con 0x7f7a1413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1413adf0 con 0x7f7a140a67c0 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1413a9d0 con 0x7f7a140a6ba0 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a140a0c80 con 0x7f7a1413f5c0 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a18e36640 10 monclient(hunting): authenticate will time out at 1741.953802s 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a1259c640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f7a140a67c0 0x7f7a1409fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37770/0 (socket says 172.21.15.1:37770) 2024-08-19T10:08:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.877+0000 7f7a1259c640 1 -- 172.21.15.1:0/325274571 learned_addr learned my addr 172.21.15.1:0/325274571 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a1259c640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.1:6791/0 conn(0x7f7a140a67c0 legacy=0x7f7a1409fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 client.?.objecter ms_handle_connect 0x7f7a140a67c0 2024-08-19T10:08:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a12d9d640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.1:6792/0 conn(0x7f7a1413f5c0 legacy=0x7f7a141d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 client.?.objecter ms_handle_connect 0x7f7a1413f5c0 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230255657 0 0) 0x7f7a04003e10 con 0x7f7a140a67c0 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): my global_id is 194755 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f4004ee0 con 0x7f7a140a67c0 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2050622097 0 0) 0x7f7a140a0c80 con 0x7f7a1413f5c0 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 10 monclient(hunting): my global_id is 204450 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a09d9b640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6791/0 conn(0x7f7a140a6ba0 legacy=0x7f7a140a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a04003e10 con 0x7f7a1413f5c0 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.880+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 client.?.objecter ms_handle_connect 0x7f7a140a6ba0 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3924625341 0 0) 0x7f79f4004ee0 con 0x7f7a140a67c0 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a140a0c80 con 0x7f7a140a67c0 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2828971595 0 0) 0x7f7a1413a9d0 con 0x7f7a140a6ba0 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): my global_id is 214304 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f4004ee0 con 0x7f7a140a6ba0 2024-08-19T10:08:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1587431744 0 0) 0x7f7a04003e10 con 0x7f7a1413f5c0 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a1413a9d0 con 0x7f7a1413f5c0 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7a04004900 con 0x7f7a140a67c0 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): dump: 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7a0c002020 con 0x7f7a1413f5c0 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient(hunting): dump: 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.898 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2598062059 0 0) 0x7f7a140a0c80 con 0x7f7a140a67c0 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.1:6792/0 conn(0x7f7a1413f5c0 legacy=0x7f7a141d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6791/0 conn(0x7f7a140a6ba0 legacy=0x7f7a140a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 monclient: found mon.c 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a14142de0 con 0x7f7a140a67c0 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: _finish_auth 0 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 5 monclient: authenticate success, global_id 194755 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 10 monclient: _renew_subs 2024-08-19T10:08:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1413f340 con 0x7f7a140a67c0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 10 client.194755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 10 monclient: _renew_subs 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a141d7970 con 0x7f7a140a67c0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a18e36640 1 librados: init done 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a04004aa0 con 0x7f7a140a67c0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7a04005090 con 0x7f7a140a67c0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a1159a640 4 set_mon_vals no callback set 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.882+0000 7f7a0b7fe640 10 monclient: dump: 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a18e36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79f80065c0 con 0x7f79f8002860 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7a040052f0 con 0x7f7a140a67c0 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(658..658 src has 299..658) ==== 5987+0+0 (unknown 879229092 0 0) 0x7f7a0404e7b0 con 0x7f7a140a67c0 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a0b7fe640 10 client.194755.objecter ms_dispatch 0x7f7a14000b90 osd_map(658..658 src has 299..658) 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a0b7fe640 3 client.194755.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.883+0000 7f7a0b7fe640 3 client.194755.objecter handle_osd_map decoding full epoch 658 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.884+0000 7f7a0b7fe640 20 client.194755.objecter dump_active .. 0 homeless 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.884+0000 7f7a12d9d640 10 monclient: get_auth_request con 0x7f79f40416a0 auth_method 0 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.885+0000 7f7a09d9b640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6790/0 conn(0x7f79f8002860 legacy=0x7f79f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.885+0000 7f7a0b7fe640 10 client.194755.objecter ms_handle_connect 0x7f79f8002860 2024-08-19T10:08:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.885+0000 7f7a0b7fe640 10 client.194755.objecter resend_mon_ops 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015239456 0 0) 0x7f79f80065c0 con 0x7f79f8002860 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient(hunting): my global_id is 214302 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a140a0c80 con 0x7f79f8002860 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1311196345 0 0) 0x7f7a140a0c80 con 0x7f79f8002860 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79f80065c0 con 0x7f79f8002860 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f79fc001d30 con 0x7f79f8002860 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient: dump: 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.903 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 client.194755.objecter ms_handle_connect 0x7f79f40416a0 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 157058357 0 0) 0x7f79f80065c0 con 0x7f79f8002860 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79f800a160 con 0x7f79f8002860 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f79fc004b50 con 0x7f79f8002860 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6790/0 conn(0x7f79f8002860 legacy=0x7f79f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.886+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6790/0 conn(0x7f79f8002860 legacy=0x7f79f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.893+0000 7f7a18e36640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.893+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79f8009e70 con 0x7f79f8009130 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a09d9b640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6790/0 conn(0x7f79f8009130 legacy=0x7f79f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 client.194755.objecter ms_handle_connect 0x7f79f8009130 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 client.194755.objecter resend_mon_ops 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808652551 0 0) 0x7f79f8009e70 con 0x7f79f8009130 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient(hunting): my global_id is 214311 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f80065c0 con 0x7f79f8009130 2024-08-19T10:08:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2143684031 0 0) 0x7f79f80065c0 con 0x7f79f8009130 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79f8009e70 con 0x7f79f8009130 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f79fc005490 con 0x7f79f8009130 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.895+0000 7f7a0b7fe640 10 monclient: dump: 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:05.908 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1815806664 0 0) 0x7f79f8009e70 con 0x7f79f8009130 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f79f800c3f0 con 0x7f79f8009130 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7506 (unknown 0 0 943798355) 0x7f79fc008600 con 0x7f79f8009130 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6790/0 conn(0x7f79f8009130 legacy=0x7f79f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a0b7fe640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6790/0 conn(0x7f79f8009130 legacy=0x7f79f80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30770, 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:05.910 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:05.912 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.913 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:05.915 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:05.917 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.918 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.919 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.920 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.921 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:05.923 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.924 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.925 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:05.926 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:05.927 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:05.930 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 10 librados: watch_flush enter 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 10 librados: watch_flush exit 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 20 client.194755.objecter shutdown clearing up homeless session... 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 10 client.194755.objecter successfully canceled tick 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.82:6812/2549104415 conn(0x7f79f40416a0 legacy=0x7f79f4043b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 10 monclient: shutdown 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 >> v1:172.21.15.1:6791/0 conn(0x7f7a140a67c0 legacy=0x7f7a1409fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a12d9d640 1 -- 172.21.15.1:0/325274571 reap_dead start 2024-08-19T10:08:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a12d9d640 1 -- 172.21.15.1:0/325274571 reap_dead start 2024-08-19T10:08:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 shutdown_connections 2024-08-19T10:08:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 >> 172.21.15.1:0/325274571 conn(0x7f7a140ac4b0 msgr2=0x7f7a140aa160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 shutdown_connections 2024-08-19T10:08:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 -- 172.21.15.1:0/325274571 wait complete. 2024-08-19T10:08:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:05.896+0000 7f7a18e36640 1 librados: shutdown 2024-08-19T10:08:05.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:05.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T10:08:05.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.036+0000 7febe9514640 10 monclient: get_monmap_and_config 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.036+0000 7febe9514640 10 monclient: build_initial_monmap 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.036+0000 7febe9514640 10 monclient: monmap: 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:06.037011+0000 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:06.037011+0000 2024-08-19T10:08:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:06.038 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.036+0000 7febe9514640 1 Processor -- start 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.036+0000 7febe9514640 1 -- start start 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.036+0000 7febe9514640 10 monclient: init 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient: _add_conns ranks=[7,6,2,1,4,3,0,5,8] 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient(hunting): picked mon.noname-i con 0x7febe40a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient(hunting): picked mon.noname-d con 0x7febe40694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient(hunting): picked mon.noname-b con 0x7febe40a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe40a3c70 con 0x7febe40a46a0 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe40a33d0 con 0x7febe40694b0 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe40a36b0 con 0x7febe40a6830 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febe9514640 10 monclient(hunting): authenticate will time out at 1742.113422s 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febdbfff640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7febe40694b0 0x7febe40a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:57346/0 (socket says 172.21.15.1:57346) 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.037+0000 7febdbfff640 1 -- 172.21.15.1:0/753792980 learned_addr learned my addr 172.21.15.1:0/753792980 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe37fe640 1 -- 172.21.15.1:0/753792980 >> v1:172.21.15.1:6790/0 conn(0x7febe40a46a0 legacy=0x7febe4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febdbfff640 1 -- 172.21.15.1:0/753792980 >> v1:172.21.15.1:6792/0 conn(0x7febe40694b0 legacy=0x7febe40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe2ffd640 1 -- 172.21.15.1:0/753792980 >> v1:172.21.15.82:6792/0 conn(0x7febe40a6830 legacy=0x7febe40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 596983351 0 0) 0x7febe40a33d0 con 0x7febe40694b0 2024-08-19T10:08:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): my global_id is 204459 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febd0003520 con 0x7febe40694b0 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2537784995 0 0) 0x7febe40a3c70 con 0x7febe40a46a0 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 10 monclient(hunting): my global_id is 184925 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.039+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe40a33d0 con 0x7febe40a46a0 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229236402 0 0) 0x7febe40a36b0 con 0x7febe40a6830 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient(hunting): my global_id is 204532 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe40a3c70 con 0x7febe40a6830 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3273980555 0 0) 0x7febe40a33d0 con 0x7febe40a46a0 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 >> v1:172.21.15.82:6792/0 conn(0x7febe40a6830 legacy=0x7febe40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 >> v1:172.21.15.1:6792/0 conn(0x7febe40694b0 legacy=0x7febe40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: _finish_hunting 0 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 monclient: found mon.noname-b 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe406dbb0 con 0x7febe40a46a0 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: _finish_auth 0 2024-08-19T10:08:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: _check_auth_tickets 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febdc001e10 con 0x7febe40a46a0 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 5 monclient: authenticate success, global_id 184925 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: dump: 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febdc003990 con 0x7febe40a46a0 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 1 -- 172.21.15.1:0/753792980 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febdc003f00 con 0x7febe40a46a0 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 4 set_mon_vals no callback set 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe27fc640 10 monclient: dump: 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 10 monclient: shutdown 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 1 -- 172.21.15.1:0/753792980 >> v1:172.21.15.1:6790/0 conn(0x7febe40a46a0 legacy=0x7febe4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 1 -- 172.21.15.1:0/753792980 shutdown_connections 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 1 -- 172.21.15.1:0/753792980 >> 172.21.15.1:0/753792980 conn(0x7febe40a9020 msgr2=0x7febe4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 1 -- 172.21.15.1:0/753792980 shutdown_connections 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.040+0000 7febe9514640 1 -- 172.21.15.1:0/753792980 wait complete. 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 10 monclient: build_initial_monmap 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 10 monclient: monmap: 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 1 librados: starting msgr at 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 1 librados: starting objecter 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 1 Processor -- start 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 1 -- start start 2024-08-19T10:08:06.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 1 librados: setting wanted keys 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 1 librados: calling monclient init 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.041+0000 7febe9514640 10 monclient: init 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient: _add_conns ranks=[8,3,6,2,0,5,7,4,1] 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient(hunting): picked mon.noname-e con 0x7febe40694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient(hunting): picked mon.noname-g con 0x7febe40a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient(hunting): picked mon.noname-d con 0x7febe40a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe41cf260 con 0x7febe40a46a0 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe4137fa0 con 0x7febe40a6830 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe4138570 con 0x7febe40694b0 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe9514640 10 monclient(hunting): authenticate will time out at 1742.118412s 2024-08-19T10:08:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe2ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7febe40694b0 0x7febe41ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:51660/0 (socket says 172.21.15.1:51660) 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.042+0000 7febe2ffd640 1 -- 172.21.15.1:0/3647254126 learned_addr learned my addr 172.21.15.1:0/3647254126 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febe2ffd640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.1:6793/0 conn(0x7febe40694b0 legacy=0x7febe41ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febe37fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.1:6792/0 conn(0x7febe40a6830 legacy=0x7febe41d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 client.?.objecter ms_handle_connect 0x7febe40694b0 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 client.?.objecter ms_handle_connect 0x7febe40a6830 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 client.?.objecter ms_handle_connect 0x7febe40a46a0 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdbfff640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6790/0 conn(0x7febe40a46a0 legacy=0x7febe41ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359859818 0 0) 0x7febe4138570 con 0x7febe40694b0 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): my global_id is 204587 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc4004eb0 con 0x7febe40694b0 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 787865670 0 0) 0x7febe4137fa0 con 0x7febe40a6830 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 10 monclient(hunting): my global_id is 204468 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe4138570 con 0x7febe40a6830 2024-08-19T10:08:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.044+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2081667435 0 0) 0x7febc4004eb0 con 0x7febe40694b0 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febe4137fa0 con 0x7febe40694b0 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febd4003950 con 0x7febe40694b0 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): dump: 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964385115 0 0) 0x7febe41cf260 con 0x7febe40a46a0 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): my global_id is 214320 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc4004eb0 con 0x7febe40a46a0 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4241371548 0 0) 0x7febe4138570 con 0x7febe40a6830 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febe41cf260 con 0x7febe40a6830 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febdc003e80 con 0x7febe40a6830 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient(hunting): dump: 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1689797409 0 0) 0x7febe4137fa0 con 0x7febe40694b0 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.1:6792/0 conn(0x7febe40a6830 legacy=0x7febe41d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6790/0 conn(0x7febe40a46a0 legacy=0x7febe41ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 monclient: found mon.e 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe41378b0 con 0x7febe40694b0 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: _finish_auth 0 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 5 monclient: authenticate success, global_id 204587 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 10 monclient: _renew_subs 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febe4136af0 con 0x7febe40694b0 2024-08-19T10:08:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 10 client.204587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 10 monclient: _renew_subs 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7febe41d7dc0 con 0x7febe40694b0 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe9514640 1 librados: init done 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febd4001ce0 con 0x7febe40694b0 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febd40041a0 con 0x7febe40694b0 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febe1ffb640 4 set_mon_vals no callback set 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.045+0000 7febdb7fe640 10 monclient: dump: 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7febd4011a80 con 0x7febe40694b0 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febe9514640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7febc80065c0 con 0x7febc8002860 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(659..659 src has 299..659) ==== 5987+0+0 (unknown 2531336428 0 0) 0x7febd404e5a0 con 0x7febe40694b0 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febdb7fe640 10 client.204587.objecter ms_dispatch 0x7febe4000b90 osd_map(659..659 src has 299..659) 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febdb7fe640 3 client.204587.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febdb7fe640 3 client.204587.objecter handle_osd_map decoding full epoch 659 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febdb7fe640 20 client.204587.objecter dump_active .. 0 homeless 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.046+0000 7febe37fe640 10 monclient: get_auth_request con 0x7febc4041720 auth_method 0 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.048+0000 7febdb7fe640 10 client.204587.objecter ms_handle_connect 0x7febc8002860 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.048+0000 7febdbfff640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6791/0 conn(0x7febc8002860 legacy=0x7febc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.048+0000 7febdb7fe640 10 client.204587.objecter resend_mon_ops 2024-08-19T10:08:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.048+0000 7febdb7fe640 10 client.204587.objecter ms_handle_connect 0x7febc4041720 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187316562 0 0) 0x7febc80065c0 con 0x7febc8002860 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient(hunting): my global_id is 214313 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe4137fa0 con 0x7febc8002860 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 923881076 0 0) 0x7febe4137fa0 con 0x7febc8002860 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febc80065c0 con 0x7febc8002860 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febcc0047e0 con 0x7febc8002860 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient: dump: 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3386573179 0 0) 0x7febc80065c0 con 0x7febc8002860 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febc800a160 con 0x7febc8002860 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7febcc004870 con 0x7febc8002860 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6791/0 conn(0x7febc8002860 legacy=0x7febc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.049+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6791/0 conn(0x7febc8002860 legacy=0x7febc8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.056+0000 7febe9514640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.056+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7febc8009e70 con 0x7febc8009130 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdbfff640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6791/0 conn(0x7febc8009130 legacy=0x7febc80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 10 client.204587.objecter ms_handle_connect 0x7febc8009130 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 10 client.204587.objecter resend_mon_ops 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098341498 0 0) 0x7febc8009e70 con 0x7febc8009130 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 10 monclient(hunting): my global_id is 214322 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.058+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc80065c0 con 0x7febc8009130 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3473025011 0 0) 0x7febc80065c0 con 0x7febc8009130 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febc8009e70 con 0x7febc8009130 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7febcc005110 con 0x7febc8009130 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 10 monclient: dump: 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3806143331 0 0) 0x7febc8009e70 con 0x7febc8009130 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febc800c410 con 0x7febc8009130 2024-08-19T10:08:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7506 (unknown 0 0 3441121632) 0x7febcc0074b0 con 0x7febc8009130 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6791/0 conn(0x7febc8009130 legacy=0x7febc80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.059+0000 7febdb7fe640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6791/0 conn(0x7febc8009130 legacy=0x7febc80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30933, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.085 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.086 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:06.087 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.089 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.090 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.091 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:06.095 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.096 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.097 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:06.098 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 10 librados: watch_flush enter 2024-08-19T10:08:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 10 librados: watch_flush exit 2024-08-19T10:08:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 20 client.204587.objecter shutdown clearing up homeless session... 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 10 client.204587.objecter successfully canceled tick 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.82:6812/2549104415 conn(0x7febc4041720 legacy=0x7febc4043be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 10 monclient: shutdown 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 >> v1:172.21.15.1:6793/0 conn(0x7febe40694b0 legacy=0x7febe41ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe37fe640 1 -- 172.21.15.1:0/3647254126 reap_dead start 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe37fe640 1 -- 172.21.15.1:0/3647254126 reap_dead start 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 shutdown_connections 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 >> 172.21.15.1:0/3647254126 conn(0x7febe40a9020 msgr2=0x7febe40a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 shutdown_connections 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 -- 172.21.15.1:0/3647254126 wait complete. 2024-08-19T10:08:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.060+0000 7febe9514640 1 librados: shutdown 2024-08-19T10:08:06.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:06.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-08-19T10:08:06.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.201+0000 7f656473a640 10 monclient: get_monmap_and_config 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.201+0000 7f656473a640 10 monclient: build_initial_monmap 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.201+0000 7f656473a640 10 monclient: monmap: 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:08:06.201990+0000 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:08:06.201990+0000 2024-08-19T10:08:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:06.203 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.201+0000 7f656473a640 1 Processor -- start 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.201+0000 7f656473a640 1 -- start start 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.201+0000 7f656473a640 10 monclient: init 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient: _add_conns ranks=[1,4,3,8,7,6,5,2,0] 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient(hunting): picked mon.noname-f con 0x7f655c0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient(hunting): picked mon.noname-c con 0x7f655c0694b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient(hunting): picked mon.noname-g con 0x7f655c0a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c0a3c70 con 0x7f655c0a6830 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c0a33d0 con 0x7f655c0a46a0 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c0a36b0 con 0x7f655c0694b0 2024-08-19T10:08:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f656473a640 10 monclient(hunting): authenticate will time out at 1742.278384s 2024-08-19T10:08:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f6561cd5640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f655c0694b0 0x7f655c0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37786/0 (socket says 172.21.15.1:37786) 2024-08-19T10:08:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f6561cd5640 1 -- 172.21.15.1:0/4042589567 learned_addr learned my addr 172.21.15.1:0/4042589567 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.202+0000 7f65624d6640 1 -- 172.21.15.1:0/4042589567 >> v1:172.21.15.82:6789/0 conn(0x7f655c0a6830 legacy=0x7f655c0a6c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f6561cd5640 1 -- 172.21.15.1:0/4042589567 >> v1:172.21.15.1:6791/0 conn(0x7f655c0694b0 legacy=0x7f655c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29684344 0 0) 0x7f655c0a36b0 con 0x7f655c0694b0 2024-08-19T10:08:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 10 monclient(hunting): my global_id is 194764 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6544003520 con 0x7f655c0694b0 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f6562cd7640 1 -- 172.21.15.1:0/4042589567 >> v1:172.21.15.82:6790/0 conn(0x7f655c0a46a0 legacy=0x7f655c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2229863379 0 0) 0x7f6544003520 con 0x7f655c0694b0 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.204+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 >> v1:172.21.15.82:6790/0 conn(0x7f655c0a46a0 legacy=0x7f655c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 >> v1:172.21.15.82:6789/0 conn(0x7f655c0a6830 legacy=0x7f655c0a6c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: _finish_hunting 0 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 1 monclient: found mon.noname-c 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c06dbb0 con 0x7f655c0694b0 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: _finish_auth 0 2024-08-19T10:08:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: _check_auth_tickets 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6550001ae0 con 0x7f655c0694b0 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 5 monclient: authenticate success, global_id 194764 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: dump: 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65500048a0 con 0x7f655c0694b0 2024-08-19T10:08:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 1 -- 172.21.15.1:0/4042589567 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6550004e10 con 0x7f655c0694b0 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f65614d4640 10 monclient: dump: 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 10 monclient: get_monmap_and_config success 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 4 set_mon_vals no callback set 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 10 monclient: shutdown 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 1 -- 172.21.15.1:0/4042589567 >> v1:172.21.15.1:6791/0 conn(0x7f655c0694b0 legacy=0x7f655c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 1 -- 172.21.15.1:0/4042589567 shutdown_connections 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 1 -- 172.21.15.1:0/4042589567 >> 172.21.15.1:0/4042589567 conn(0x7f655c0a9020 msgr2=0x7f655c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 1 -- 172.21.15.1:0/4042589567 shutdown_connections 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.205+0000 7f656473a640 1 -- 172.21.15.1:0/4042589567 wait complete. 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 10 monclient: build_initial_monmap 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 10 monclient: monmap: 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:08:06.215 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 1 librados: starting msgr at 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 1 librados: starting objecter 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 1 Processor -- start 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 1 -- start start 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 1 librados: setting wanted keys 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 1 librados: calling monclient init 2024-08-19T10:08:06.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.206+0000 7f656473a640 10 monclient: init 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient: _reopen_session rank -1 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient: _add_conns ranks=[4,1,6,8,7,5,0,2,3] 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient(hunting): picked mon.noname-c con 0x7f655c0694b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient(hunting): picked mon.noname-f con 0x7f655c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient(hunting): picked mon.noname-d con 0x7f655c0a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c1378b0 con 0x7f655c0a46a0 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c1387d0 con 0x7f655c0694b0 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c138ab0 con 0x7f655c0a6830 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient(hunting): _renew_subs 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f656473a640 10 monclient(hunting): authenticate will time out at 1742.283127s 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f65624d6640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f655c0694b0 0x7f655c1353c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:37798/0 (socket says 172.21.15.1:37798) 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f65624d6640 1 -- 172.21.15.1:0/4265069188 learned_addr learned my addr 172.21.15.1:0/4265069188 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:08:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.207+0000 7f6561cd5640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6789/0 conn(0x7f655c0a46a0 legacy=0x7f655c135ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f65624d6640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.1:6791/0 conn(0x7f655c0694b0 legacy=0x7f655c1353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 client.?.objecter ms_handle_connect 0x7f655c0694b0 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 client.?.objecter ms_handle_connect 0x7f655c0a6830 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f6562cd7640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.1:6792/0 conn(0x7f655c0a6830 legacy=0x7f655c1361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28838581 0 0) 0x7f655c1387d0 con 0x7f655c0694b0 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): my global_id is 194773 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6548004ee0 con 0x7f655c0694b0 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788101790 0 0) 0x7f655c138ab0 con 0x7f655c0a6830 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): my global_id is 204477 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655c1387d0 con 0x7f655c0a6830 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 440518713 0 0) 0x7f6548004ee0 con 0x7f655c0694b0 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f655c138ab0 con 0x7f655c0694b0 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4196093938 0 0) 0x7f655c1387d0 con 0x7f655c0a6830 2024-08-19T10:08:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6548004ee0 con 0x7f655c0a6830 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6554004620 con 0x7f655c0694b0 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): dump: 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f654c003990 con 0x7f655c0a6830 2024-08-19T10:08:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient(hunting): dump: 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 63938845 0 0) 0x7f655c138ab0 con 0x7f655c0694b0 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.1:6792/0 conn(0x7f655c0a6830 legacy=0x7f655c1361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6789/0 conn(0x7f655c0a46a0 legacy=0x7f655c135ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient: _finish_hunting 0 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 monclient: found mon.c 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c138d90 con 0x7f655c0694b0 2024-08-19T10:08:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient: _finish_auth 0 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 10 monclient: _check_auth_tickets 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f655affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f656473a640 5 monclient: authenticate success, global_id 194773 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f656473a640 10 monclient: _renew_subs 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f656473a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.209+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f655c136bb0 con 0x7f655c0694b0 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 10 client.194773.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 10 monclient: _renew_subs 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f655c1d74a0 con 0x7f655c0694b0 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 1 librados: init done 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65540048f0 con 0x7f655c0694b0 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:08:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6554004e60 con 0x7f655c0694b0 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f6560cd3640 4 set_mon_vals no callback set 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 10 monclient: dump: 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f653c0065c0 con 0x7f653c002860 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.210+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6554005000 con 0x7f655c0694b0 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.211+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(659..659 src has 299..659) ==== 5987+0+0 (unknown 2531336428 0 0) 0x7f655404e430 con 0x7f655c0694b0 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.211+0000 7f655affd640 10 client.194773.objecter ms_dispatch 0x7f655c000b90 osd_map(659..659 src has 299..659) 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.211+0000 7f655affd640 3 client.194773.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T10:08:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.211+0000 7f655affd640 3 client.194773.objecter handle_osd_map decoding full epoch 659 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.211+0000 7f655affd640 20 client.194773.objecter dump_active .. 0 homeless 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.211+0000 7f6562cd7640 10 monclient: get_auth_request con 0x7f65480414d0 auth_method 0 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 client.194773.objecter ms_handle_connect 0x7f653c002860 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f6561cd5640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6792/0 conn(0x7f653c002860 legacy=0x7f653c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 client.194773.objecter resend_mon_ops 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728499500 0 0) 0x7f653c0065c0 con 0x7f653c002860 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 monclient(hunting): my global_id is 204541 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655c138ab0 con 0x7f653c002860 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.213+0000 7f655affd640 10 client.194773.objecter ms_handle_connect 0x7f65480414d0 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3758162851 0 0) 0x7f655c138ab0 con 0x7f653c002860 2024-08-19T10:08:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f653c0065c0 con 0x7f653c002860 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f65500047e0 con 0x7f653c002860 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 10 monclient: dump: 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2318886861 0 0) 0x7f653c0065c0 con 0x7f653c002860 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f653c00a160 con 0x7f653c002860 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f65500047e0 con 0x7f653c002860 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6792/0 conn(0x7f653c002860 legacy=0x7f653c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.214+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6792/0 conn(0x7f653c002860 legacy=0x7f653c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:06.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.221+0000 7f656473a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:08:06.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.221+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f653c009e50 con 0x7f653c009110 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 10 client.194773.objecter ms_handle_connect 0x7f653c009110 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f6561cd5640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6792/0 conn(0x7f653c009110 legacy=0x7f653c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 10 client.194773.objecter resend_mon_ops 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54580195 0 0) 0x7f653c009e50 con 0x7f653c009110 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 10 monclient(hunting): my global_id is 204550 2024-08-19T10:08:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.223+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f653c0065c0 con 0x7f653c009110 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3053401215 0 0) 0x7f653c0065c0 con 0x7f653c009110 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f653c009e50 con 0x7f653c009110 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f65500050b0 con 0x7f653c009110 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 10 monclient: dump: 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:08:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2028997941 0 0) 0x7f653c009e50 con 0x7f653c009110 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f653c00c3f0 con 0x7f653c009110 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7507 (unknown 0 0 3521728454) 0x7f6550007d60 con 0x7f653c009110 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:08:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6792/0 conn(0x7f653c009110 legacy=0x7f653c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.224+0000 7f655affd640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6792/0 conn(0x7f653c009110 legacy=0x7f653c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 146, 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 152094, 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:08:06.238 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:06.239 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:06.240 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:08:06.241 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:08:06.242 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:08:06.246 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:08:06.247 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.248 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.249 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:08:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.252 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.254 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.255 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:08:06.258 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:08:06.259 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:08:06.259 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:08:06.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.259 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:08:06.261 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:08:06.262 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:08:06.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:08:06.262 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:08:06.262 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:08:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 10 librados: watch_flush enter 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 10 librados: watch_flush exit 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 20 client.194773.objecter shutdown clearing up homeless session... 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 10 client.194773.objecter successfully canceled tick 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.82:6812/2549104415 conn(0x7f65480414d0 legacy=0x7f6548043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 10 monclient: shutdown 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 >> v1:172.21.15.1:6791/0 conn(0x7f655c0694b0 legacy=0x7f655c1353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f6562cd7640 1 -- 172.21.15.1:0/4265069188 reap_dead start 2024-08-19T10:08:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f6562cd7640 1 -- 172.21.15.1:0/4265069188 reap_dead start 2024-08-19T10:08:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 shutdown_connections 2024-08-19T10:08:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 >> 172.21.15.1:0/4265069188 conn(0x7f655c0a9020 msgr2=0x7f655c0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:08:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 shutdown_connections 2024-08-19T10:08:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 -- 172.21.15.1:0/4265069188 wait complete. 2024-08-19T10:08:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:08:06.225+0000 7f656473a640 1 librados: shutdown 2024-08-19T10:08:06.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T10:08:06.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T10:09:20.651 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8918 ms) 2024-08-19T10:09:20.651 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-08-19T10:09:20.651 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14498 ms) 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7482 ms) 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15886 ms) 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8107 ms) 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9580 ms) 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: hmm, no HitSet yet 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8434 ms) 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-08-19T10:09:20.652 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: first is 1724062052 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,1724062057,1724062058,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,1724062057,1724062058,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,1724062057,1724062058,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,1724062057,1724062058,1724062060,1724062061,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,1724062057,1724062058,1724062060,1724062061,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062052,1724062054,1724062055,1724062057,1724062058,1724062060,1724062061,0 2024-08-19T10:09:20.653 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: got ls 1724062055,1724062057,1724062058,1724062060,1724062061,1724062063,1724062064,0 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: first now 1724062055, trimmed 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21807 ms) 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: foo0 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (30967 ms) 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20166 ms) 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22629 ms) 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6312 ms) 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-08-19T10:09:20.654 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3004 ms) 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3018 ms) 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (5137 ms) 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (283519 ms total) 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1081223 ms total) 2024-08-19T10:09:20.655 INFO:tasks.workunit.client.0.smithi082.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-08-19T10:09:20.661 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.661 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25698 2024-08-19T10:09:20.661 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25698 2024-08-19T10:09:20.661 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25931 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25931 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26459 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26459 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26151 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26151 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26022 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26022 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.662 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25774 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25774 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26086 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26086 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26610 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26610 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26633 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26633 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25820 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25820 2024-08-19T10:09:20.663 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26471 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26471 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25672 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25672 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25734 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25734 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26386 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26386 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25663 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25663 2024-08-19T10:09:20.664 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25683 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25683 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26751 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26751 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25975 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25975 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26949 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26949 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26051 2024-08-19T10:09:20.665 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26051 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26589 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26589 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=27030 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 27030 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26007 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26007 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26408 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26408 2024-08-19T10:09:20.666 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26727 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26727 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26524 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26524 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26889 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26889 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26368 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26368 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.667 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25897 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25897 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26979 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26979 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26210 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26210 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25656 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25656 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25715 2024-08-19T10:09:20.668 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25715 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26132 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26132 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=25756 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 25756 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26253 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26253 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26294 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26294 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=26660 2024-08-19T10:09:20.669 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 26660 2024-08-19T10:09:20.670 INFO:tasks.workunit.client.0.smithi082.stderr:+ for t in "${!pids[@]}" 2024-08-19T10:09:20.670 INFO:tasks.workunit.client.0.smithi082.stderr:+ pid=27062 2024-08-19T10:09:20.670 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait 27062 2024-08-19T10:09:20.670 INFO:tasks.workunit.client.0.smithi082.stderr:+ exit 0 2024-08-19T10:09:20.670 INFO:tasks.workunit.client.0.smithi082.stderr:+ cleanup 2024-08-19T10:09:20.670 INFO:tasks.workunit.client.0.smithi082.stderr:+ pkill -P 25650 2024-08-19T10:09:20.671 INFO:tasks.workunit.client.0.smithi082.stderr:+ true 2024-08-19T10:09:20.671 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T10:09:20.672 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T10:09:20.700 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-08-19T10:09:20.701 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-19T10:09:21.030 DEBUG:teuthology.parallel:result is None 2024-08-19T10:09:21.031 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T10:09:21.056 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T10:09:21.056 DEBUG:teuthology.orchestra.run.smithi082:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-08-19T10:09:21.119 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T10:09:21.119 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-08-19T10:09:21.130 INFO:tasks.mon_thrash:joining mon_thrasher 2024-08-19T10:09:21.130 ERROR:teuthology.run_tasks:Manager failed: mon_thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 466, in task thrash_proc.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T10:09:21.132 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-19T10:09:21.208 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-19T10:09:21.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T10:09:21.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.320+0000 7f6bc6bf2640 10 monclient: get_monmap_and_config 2024-08-19T10:09:21.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.320+0000 7f6bc6bf2640 10 monclient: build_initial_monmap 2024-08-19T10:09:21.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.320+0000 7f6bc6bf2640 10 monclient: monmap: 2024-08-19T10:09:21.321 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:21.321 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.321 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:21.321038+0000 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:21.321038+0000 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:21.322 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:21.323 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:21.323 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:21.324 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:21.324 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:21.324 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.320+0000 7f6bc6bf2640 1 Processor -- start 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.320+0000 7f6bc6bf2640 1 -- start start 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.320+0000 7f6bc6bf2640 10 monclient: init 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient: _add_conns ranks=[7,2,0,6,5,8,3,4,1] 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient(hunting): picked mon.noname-i con 0x7f6bc00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient(hunting): picked mon.noname-b con 0x7f6bc00a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient(hunting): picked mon.noname-a con 0x7f6bc013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc00a3b40 con 0x7f6bc013f5c0 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc00a8130 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc006ede0 con 0x7f6bc00a67c0 2024-08-19T10:09:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc6bf2640 10 monclient(hunting): authenticate will time out at 1817.397488s 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bbffff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f6bc00a6ba0 0x7f6bc00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49102/0 (socket says 172.21.15.1:49102) 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bc518f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6bc013f5c0 0x7f6bc01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51488/0 (socket says 172.21.15.1:51488) 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.321+0000 7f6bbffff640 1 -- 172.21.15.1:0/680743985 learned_addr learned my addr 172.21.15.1:0/680743985 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.323+0000 7f6bc518f640 1 -- 172.21.15.1:0/680743985 >> v1:172.21.15.1:6789/0 conn(0x7f6bc013f5c0 legacy=0x7f6bc01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.323+0000 7f6bbffff640 1 -- 172.21.15.1:0/680743985 >> v1:172.21.15.1:6790/0 conn(0x7f6bc00a6ba0 legacy=0x7f6bc00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.323+0000 7f6bc498e640 1 -- 172.21.15.1:0/680743985 >> v1:172.21.15.82:6792/0 conn(0x7f6bc00a67c0 legacy=0x7f6bc00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872819565 0 0) 0x7f6bc00a8130 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): my global_id is 184934 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb0003520 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755026906 0 0) 0x7f6bc00a3b40 con 0x7f6bc013f5c0 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): my global_id is 204570 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc00a8130 con 0x7f6bc013f5c0 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610568264 0 0) 0x7f6bc006ede0 con 0x7f6bc00a67c0 2024-08-19T10:09:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient(hunting): my global_id is 204559 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc00a3b40 con 0x7f6bc00a67c0 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3385606968 0 0) 0x7f6bc00a8130 con 0x7f6bc013f5c0 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 >> v1:172.21.15.82:6792/0 conn(0x7f6bc00a67c0 legacy=0x7f6bc00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 >> v1:172.21.15.1:6790/0 conn(0x7f6bc00a6ba0 legacy=0x7f6bc00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 monclient: found mon.noname-a 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc00a92a0 con 0x7f6bc013f5c0 2024-08-19T10:09:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bc6bf2640 5 monclient: authenticate success, global_id 204570 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6bb8001e40 con 0x7f6bc013f5c0 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bc6bf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: dump: 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bc6bf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bb8004b60 con 0x7f6bc013f5c0 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 1 -- 172.21.15.1:0/680743985 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6bb8005050 con 0x7f6bc013f5c0 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bc6bf2640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bc6bf2640 4 set_mon_vals no callback set 2024-08-19T10:09:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.324+0000 7f6bbf7fe640 10 monclient: dump: 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.337 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 10 monclient: shutdown 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/680743985 >> v1:172.21.15.1:6789/0 conn(0x7f6bc013f5c0 legacy=0x7f6bc01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/680743985 shutdown_connections 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/680743985 >> 172.21.15.1:0/680743985 conn(0x7f6bc00ac4b0 msgr2=0x7f6bc00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/680743985 shutdown_connections 2024-08-19T10:09:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/680743985 wait complete. 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 10 monclient: build_initial_monmap 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.325+0000 7f6bc6bf2640 10 monclient: monmap: 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 1 librados: starting msgr at 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 1 librados: starting objecter 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 1 Processor -- start 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 1 -- start start 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 1 librados: setting wanted keys 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 1 librados: calling monclient init 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 10 monclient: init 2024-08-19T10:09:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 10 monclient: _add_conns ranks=[0,3,8,5,1,6,4,2,7] 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.326+0000 7f6bc6bf2640 10 monclient(hunting): picked mon.noname-a con 0x7f6bc00a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 10 monclient(hunting): picked mon.noname-g con 0x7f6bc00a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 10 monclient(hunting): picked mon.noname-e con 0x7f6bc013f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc013d690 con 0x7f6bc00a67c0 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc01cb050 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc00a9fc0 con 0x7f6bc013f5c0 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc498e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6bc00a67c0 0x7f6bc01ca230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51502/0 (socket says 172.21.15.1:51502) 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc498e640 1 -- 172.21.15.1:0/2878120412 learned_addr learned my addr 172.21.15.1:0/2878120412 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:21.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.327+0000 7f6bc6bf2640 10 monclient(hunting): authenticate will time out at 1817.403145s 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bc498e640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.1:6789/0 conn(0x7f6bc00a67c0 legacy=0x7f6bc01ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 client.?.objecter ms_handle_connect 0x7f6bc00a67c0 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bc518f640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.1:6793/0 conn(0x7f6bc013f5c0 legacy=0x7f6bc01d5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 client.?.objecter ms_handle_connect 0x7f6bc013f5c0 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 client.?.objecter ms_handle_connect 0x7f6bc00a6ba0 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbffff640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.82:6790/0 conn(0x7f6bc00a6ba0 legacy=0x7f6bc01ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310534135 0 0) 0x7f6bc013d690 con 0x7f6bc00a67c0 2024-08-19T10:09:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): my global_id is 204579 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ba4004eb0 con 0x7f6bc00a67c0 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208660785 0 0) 0x7f6bc00a9fc0 con 0x7f6bc013f5c0 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): my global_id is 204596 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc013d690 con 0x7f6bc013f5c0 2024-08-19T10:09:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26898059 0 0) 0x7f6bc01cb050 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): my global_id is 214338 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc00a9fc0 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1854690680 0 0) 0x7f6ba4004eb0 con 0x7f6bc00a67c0 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bc01cb050 con 0x7f6bc00a67c0 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6bb4003950 con 0x7f6bc00a67c0 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): dump: 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.349 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2023543602 0 0) 0x7f6bc013d690 con 0x7f6bc013f5c0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ba4004eb0 con 0x7f6bc013f5c0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1393080628 0 0) 0x7f6bc00a9fc0 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bc013d690 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6bb8002090 con 0x7f6bc013f5c0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): dump: 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.352 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6bac0047e0 con 0x7f6bc00a6ba0 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 10 monclient(hunting): dump: 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 664226202 0 0) 0x7f6bc01cb050 con 0x7f6bc00a67c0 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.1:6793/0 conn(0x7f6bc013f5c0 legacy=0x7f6bc01d5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.329+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.82:6790/0 conn(0x7f6bc00a6ba0 legacy=0x7f6bc01ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:09:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 1 monclient: found mon.a 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc013d3b0 con 0x7f6bc00a67c0 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: _finish_auth 0 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 5 monclient: authenticate success, global_id 204579 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 10 monclient: _renew_subs 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc013be90 con 0x7f6bc00a67c0 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 10 client.204579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 10 monclient: _renew_subs 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6bc01d73f0 con 0x7f6bc00a67c0 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bc6bf2640 1 librados: init done 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bb4003c20 con 0x7f6bc00a67c0 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6bb4004190 con 0x7f6bc00a67c0 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbeffd640 4 set_mon_vals no callback set 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.330+0000 7f6bbd7fa640 10 monclient: dump: 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6bb4004330 con 0x7f6bc00a67c0 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bc6bf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bc6bf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bc6bf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba0005080 con 0x7f6bc00a67c0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f6bb404df00 con 0x7f6bc00a67c0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbd7fa640 10 client.204579.objecter ms_dispatch 0x7f6bc0000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbd7fa640 3 client.204579.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbd7fa640 3 client.204579.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbd7fa640 20 client.204579.objecter dump_active .. 0 homeless 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.331+0000 7f6bbffff640 10 monclient: get_auth_request con 0x7f6ba40418f0 auth_method 0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.334+0000 7f6bbd7fa640 10 client.204579.objecter ms_handle_connect 0x7f6ba40418f0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.336+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6bb4017500 con 0x7f6bc00a67c0 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.336+0000 7f6bbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.336+0000 7f6bbd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.517+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6ba0000b90 con 0x7f6ba40418f0 2024-08-19T10:09:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.522+0000 7f6bbd7fa640 1 -- 172.21.15.1:0/2878120412 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114209 (unknown 2965378022 0 3985610741) 0x7f6ba0000b90 con 0x7f6ba40418f0 2024-08-19T10:09:21.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T10:09:21.525 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T10:09:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 10 librados: watch_flush enter 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 10 librados: watch_flush exit 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 20 client.204579.objecter shutdown clearing up homeless session... 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 10 client.204579.objecter successfully canceled tick 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6ba40418f0 legacy=0x7f6ba4043db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 10 monclient: shutdown 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 >> v1:172.21.15.1:6789/0 conn(0x7f6bc00a67c0 legacy=0x7f6bc01ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 shutdown_connections 2024-08-19T10:09:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 >> 172.21.15.1:0/2878120412 conn(0x7f6bc00ac4b0 msgr2=0x7f6bc01419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 shutdown_connections 2024-08-19T10:09:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 -- 172.21.15.1:0/2878120412 wait complete. 2024-08-19T10:09:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.524+0000 7f6bc6bf2640 1 librados: shutdown 2024-08-19T10:09:21.536 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1281,"stamp":"2024-08-19T10:09:21.060770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":442812,"kb_used_data":39084,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565788228,"statfs":{"total":579820584960,"available":579367145472,"internally_reserved":0,"allocated":40022016,"data_stored":17962702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":413260450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001024"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1492,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478370+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T10:09:18.478370+0000","last_peered":"2024-08-19T10:09:18.478370+0000","last_clean":"2024-08-19T10:09:18.478370+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T10:09:18.478370+0000","last_undegraded":"2024-08-19T10:09:18.478370+0000","last_fullsized":"2024-08-19T10:09:18.478370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1398,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219416+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T10:09:17.219416+0000","last_peered":"2024-08-19T10:09:17.219416+0000","last_clean":"2024-08-19T10:09:17.219416+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T10:09:17.219416+0000","last_undegraded":"2024-08-19T10:09:17.219416+0000","last_fullsized":"2024-08-19T10:09:17.219416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1401,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478445+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T10:09:18.478445+0000","last_peered":"2024-08-19T10:09:18.478445+0000","last_clean":"2024-08-19T10:09:18.478445+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T10:09:18.478445+0000","last_undegraded":"2024-08-19T10:09:18.478445+0000","last_fullsized":"2024-08-19T10:09:18.478445+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1397,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216502+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T10:09:17.216502+0000","last_peered":"2024-08-19T10:09:17.216502+0000","last_clean":"2024-08-19T10:09:17.216502+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T10:09:17.216502+0000","last_undegraded":"2024-08-19T10:09:17.216502+0000","last_fullsized":"2024-08-19T10:09:17.216502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1399,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216190+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T10:09:17.216190+0000","last_peered":"2024-08-19T10:09:17.216190+0000","last_clean":"2024-08-19T10:09:17.216190+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T10:09:17.216190+0000","last_undegraded":"2024-08-19T10:09:17.216190+0000","last_fullsized":"2024-08-19T10:09:17.216190+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1391,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217571+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T10:09:16.217571+0000","last_peered":"2024-08-19T10:09:16.217571+0000","last_clean":"2024-08-19T10:09:16.217571+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T10:09:16.217571+0000","last_undegraded":"2024-08-19T10:09:16.217571+0000","last_fullsized":"2024-08-19T10:09:16.217571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1401,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478316+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T10:09:18.478316+0000","last_peered":"2024-08-19T10:09:18.478316+0000","last_clean":"2024-08-19T10:09:18.478316+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T10:09:18.478316+0000","last_undegraded":"2024-08-19T10:09:18.478316+0000","last_fullsized":"2024-08-19T10:09:18.478316+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1390,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217636+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T10:09:16.217636+0000","last_peered":"2024-08-19T10:09:16.217636+0000","last_clean":"2024-08-19T10:09:16.217636+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T10:09:16.217636+0000","last_undegraded":"2024-08-19T10:09:16.217636+0000","last_fullsized":"2024-08-19T10:09:16.217636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1400,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478414+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T10:09:18.478414+0000","last_peered":"2024-08-19T10:09:18.478414+0000","last_clean":"2024-08-19T10:09:18.478414+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T10:09:18.478414+0000","last_undegraded":"2024-08-19T10:09:18.478414+0000","last_fullsized":"2024-08-19T10:09:18.478414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1","version":"23'7","reported_seq":1396,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478569+0000","last_change":"2024-08-19T09:51:22.586587+0000","last_active":"2024-08-19T10:09:18.478569+0000","last_peered":"2024-08-19T10:09:18.478569+0000","last_clean":"2024-08-19T10:09:18.478569+0000","last_became_active":"2024-08-19T09:51:22.585789+0000","last_became_peered":"2024-08-19T09:51:22.585789+0000","last_unstale":"2024-08-19T10:09:18.478569+0000","last_undegraded":"2024-08-19T10:09:18.478569+0000","last_fullsized":"2024-08-19T10:09:18.478569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:25:11.479863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217047+0000","last_change":"2024-08-19T09:51:22.586723+0000","last_active":"2024-08-19T10:09:17.217047+0000","last_peered":"2024-08-19T10:09:17.217047+0000","last_clean":"2024-08-19T10:09:17.217047+0000","last_became_active":"2024-08-19T09:51:22.583401+0000","last_became_peered":"2024-08-19T09:51:22.583401+0000","last_unstale":"2024-08-19T10:09:17.217047+0000","last_undegraded":"2024-08-19T10:09:17.217047+0000","last_fullsized":"2024-08-19T10:09:17.217047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:56:50.500489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217571+0000","last_change":"2024-08-19T09:51:22.588184+0000","last_active":"2024-08-19T10:09:17.217571+0000","last_peered":"2024-08-19T10:09:17.217571+0000","last_clean":"2024-08-19T10:09:17.217571+0000","last_became_active":"2024-08-19T09:51:22.587936+0000","last_became_peered":"2024-08-19T09:51:22.587936+0000","last_unstale":"2024-08-19T10:09:17.217571+0000","last_undegraded":"2024-08-19T10:09:17.217571+0000","last_fullsized":"2024-08-19T10:09:17.217571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:22:07.623601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217092+0000","last_change":"2024-08-19T09:51:22.640927+0000","last_active":"2024-08-19T10:09:17.217092+0000","last_peered":"2024-08-19T10:09:17.217092+0000","last_clean":"2024-08-19T10:09:17.217092+0000","last_became_active":"2024-08-19T09:51:22.640604+0000","last_became_peered":"2024-08-19T09:51:22.640604+0000","last_unstale":"2024-08-19T10:09:17.217092+0000","last_undegraded":"2024-08-19T10:09:17.217092+0000","last_fullsized":"2024-08-19T10:09:17.217092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:38:41.853584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1383,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478207+0000","last_change":"2024-08-19T09:51:22.581514+0000","last_active":"2024-08-19T10:09:18.478207+0000","last_peered":"2024-08-19T10:09:18.478207+0000","last_clean":"2024-08-19T10:09:18.478207+0000","last_became_active":"2024-08-19T09:51:22.581246+0000","last_became_peered":"2024-08-19T09:51:22.581246+0000","last_unstale":"2024-08-19T10:09:18.478207+0000","last_undegraded":"2024-08-19T10:09:18.478207+0000","last_fullsized":"2024-08-19T10:09:18.478207+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:12:50.239096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.220013+0000","last_change":"2024-08-19T09:51:22.620559+0000","last_active":"2024-08-19T10:09:17.220013+0000","last_peered":"2024-08-19T10:09:17.220013+0000","last_clean":"2024-08-19T10:09:17.220013+0000","last_became_active":"2024-08-19T09:51:22.620042+0000","last_became_peered":"2024-08-19T09:51:22.620042+0000","last_unstale":"2024-08-19T10:09:17.220013+0000","last_undegraded":"2024-08-19T10:09:17.220013+0000","last_fullsized":"2024-08-19T10:09:17.220013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:27:35.980026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218905+0000","last_change":"2024-08-19T09:51:22.606649+0000","last_active":"2024-08-19T10:09:17.218905+0000","last_peered":"2024-08-19T10:09:17.218905+0000","last_clean":"2024-08-19T10:09:17.218905+0000","last_became_active":"2024-08-19T09:51:22.606054+0000","last_became_peered":"2024-08-19T09:51:22.606054+0000","last_unstale":"2024-08-19T10:09:17.218905+0000","last_undegraded":"2024-08-19T10:09:17.218905+0000","last_fullsized":"2024-08-19T10:09:17.218905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:23:52.344010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217755+0000","last_change":"2024-08-19T09:51:22.653723+0000","last_active":"2024-08-19T10:09:17.217755+0000","last_peered":"2024-08-19T10:09:17.217755+0000","last_clean":"2024-08-19T10:09:17.217755+0000","last_became_active":"2024-08-19T09:51:22.653455+0000","last_became_peered":"2024-08-19T09:51:22.653455+0000","last_unstale":"2024-08-19T10:09:17.217755+0000","last_undegraded":"2024-08-19T10:09:17.217755+0000","last_fullsized":"2024-08-19T10:09:17.217755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:25:13.028210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.679991+0000","last_change":"2024-08-19T09:51:22.560997+0000","last_active":"2024-08-19T10:09:16.679991+0000","last_peered":"2024-08-19T10:09:16.679991+0000","last_clean":"2024-08-19T10:09:16.679991+0000","last_became_active":"2024-08-19T09:51:22.553419+0000","last_became_peered":"2024-08-19T09:51:22.553419+0000","last_unstale":"2024-08-19T10:09:16.679991+0000","last_undegraded":"2024-08-19T10:09:16.679991+0000","last_fullsized":"2024-08-19T10:09:16.679991+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:25.353142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216939+0000","last_change":"2024-08-19T09:51:22.554649+0000","last_active":"2024-08-19T10:09:17.216939+0000","last_peered":"2024-08-19T10:09:17.216939+0000","last_clean":"2024-08-19T10:09:17.216939+0000","last_became_active":"2024-08-19T09:51:22.554397+0000","last_became_peered":"2024-08-19T09:51:22.554397+0000","last_unstale":"2024-08-19T10:09:17.216939+0000","last_undegraded":"2024-08-19T10:09:17.216939+0000","last_fullsized":"2024-08-19T10:09:17.216939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:36:10.709773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1378,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217364+0000","last_change":"2024-08-19T09:55:22.703762+0000","last_active":"2024-08-19T10:09:16.217364+0000","last_peered":"2024-08-19T10:09:16.217364+0000","last_clean":"2024-08-19T10:09:16.217364+0000","last_became_active":"2024-08-19T09:55:22.703438+0000","last_became_peered":"2024-08-19T09:55:22.703438+0000","last_unstale":"2024-08-19T10:09:16.217364+0000","last_undegraded":"2024-08-19T10:09:16.217364+0000","last_fullsized":"2024-08-19T10:09:16.217364+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:29:17.373388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217274+0000","last_change":"2024-08-19T09:51:22.529738+0000","last_active":"2024-08-19T10:09:16.217274+0000","last_peered":"2024-08-19T10:09:16.217274+0000","last_clean":"2024-08-19T10:09:16.217274+0000","last_became_active":"2024-08-19T09:51:22.528219+0000","last_became_peered":"2024-08-19T09:51:22.528219+0000","last_unstale":"2024-08-19T10:09:16.217274+0000","last_undegraded":"2024-08-19T10:09:16.217274+0000","last_fullsized":"2024-08-19T10:09:16.217274+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:44.262012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219294+0000","last_change":"2024-08-19T09:51:22.614370+0000","last_active":"2024-08-19T10:09:17.219294+0000","last_peered":"2024-08-19T10:09:17.219294+0000","last_clean":"2024-08-19T10:09:17.219294+0000","last_became_active":"2024-08-19T09:51:22.614136+0000","last_became_peered":"2024-08-19T09:51:22.614136+0000","last_unstale":"2024-08-19T10:09:17.219294+0000","last_undegraded":"2024-08-19T10:09:17.219294+0000","last_fullsized":"2024-08-19T10:09:17.219294+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:14:54.869367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218815+0000","last_change":"2024-08-19T09:51:22.587447+0000","last_active":"2024-08-19T10:09:17.218815+0000","last_peered":"2024-08-19T10:09:17.218815+0000","last_clean":"2024-08-19T10:09:17.218815+0000","last_became_active":"2024-08-19T09:51:22.587149+0000","last_became_peered":"2024-08-19T09:51:22.587149+0000","last_unstale":"2024-08-19T10:09:17.218815+0000","last_undegraded":"2024-08-19T10:09:17.218815+0000","last_fullsized":"2024-08-19T10:09:17.218815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:08:36.757587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479995+0000","last_change":"2024-08-19T09:51:22.638208+0000","last_active":"2024-08-19T10:09:18.479995+0000","last_peered":"2024-08-19T10:09:18.479995+0000","last_clean":"2024-08-19T10:09:18.479995+0000","last_became_active":"2024-08-19T09:51:22.634477+0000","last_became_peered":"2024-08-19T09:51:22.634477+0000","last_unstale":"2024-08-19T10:09:18.479995+0000","last_undegraded":"2024-08-19T10:09:18.479995+0000","last_fullsized":"2024-08-19T10:09:18.479995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:01:43.314792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217657+0000","last_change":"2024-08-19T09:51:22.594096+0000","last_active":"2024-08-19T10:09:17.217657+0000","last_peered":"2024-08-19T10:09:17.217657+0000","last_clean":"2024-08-19T10:09:17.217657+0000","last_became_active":"2024-08-19T09:51:22.590688+0000","last_became_peered":"2024-08-19T09:51:22.590688+0000","last_unstale":"2024-08-19T10:09:17.217657+0000","last_undegraded":"2024-08-19T10:09:17.217657+0000","last_fullsized":"2024-08-19T10:09:17.217657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:51.718926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217017+0000","last_change":"2024-08-19T09:51:22.537238+0000","last_active":"2024-08-19T10:09:16.217017+0000","last_peered":"2024-08-19T10:09:16.217017+0000","last_clean":"2024-08-19T10:09:16.217017+0000","last_became_active":"2024-08-19T09:51:22.536584+0000","last_became_peered":"2024-08-19T09:51:22.536584+0000","last_unstale":"2024-08-19T10:09:16.217017+0000","last_undegraded":"2024-08-19T10:09:16.217017+0000","last_fullsized":"2024-08-19T10:09:16.217017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:42:52.494448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1369,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.679764+0000","last_change":"2024-08-19T09:51:22.556447+0000","last_active":"2024-08-19T10:09:16.679764+0000","last_peered":"2024-08-19T10:09:16.679764+0000","last_clean":"2024-08-19T10:09:16.679764+0000","last_became_active":"2024-08-19T09:51:22.556182+0000","last_became_peered":"2024-08-19T09:51:22.556182+0000","last_unstale":"2024-08-19T10:09:16.679764+0000","last_undegraded":"2024-08-19T10:09:16.679764+0000","last_fullsized":"2024-08-19T10:09:16.679764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:04:06.155397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217049+0000","last_change":"2024-08-19T09:51:22.672202+0000","last_active":"2024-08-19T10:09:16.217049+0000","last_peered":"2024-08-19T10:09:16.217049+0000","last_clean":"2024-08-19T10:09:16.217049+0000","last_became_active":"2024-08-19T09:51:22.671918+0000","last_became_peered":"2024-08-19T09:51:22.671918+0000","last_unstale":"2024-08-19T10:09:16.217049+0000","last_undegraded":"2024-08-19T10:09:16.217049+0000","last_fullsized":"2024-08-19T10:09:16.217049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:56:23.219267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217181+0000","last_change":"2024-08-19T09:51:22.605834+0000","last_active":"2024-08-19T10:09:16.217181+0000","last_peered":"2024-08-19T10:09:16.217181+0000","last_clean":"2024-08-19T10:09:16.217181+0000","last_became_active":"2024-08-19T09:51:22.605586+0000","last_became_peered":"2024-08-19T09:51:22.605586+0000","last_unstale":"2024-08-19T10:09:16.217181+0000","last_undegraded":"2024-08-19T10:09:16.217181+0000","last_fullsized":"2024-08-19T10:09:16.217181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:53:29.756052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217163+0000","last_change":"2024-08-19T09:51:22.551390+0000","last_active":"2024-08-19T10:09:16.217163+0000","last_peered":"2024-08-19T10:09:16.217163+0000","last_clean":"2024-08-19T10:09:16.217163+0000","last_became_active":"2024-08-19T09:51:22.551119+0000","last_became_peered":"2024-08-19T09:51:22.551119+0000","last_unstale":"2024-08-19T10:09:16.217163+0000","last_undegraded":"2024-08-19T10:09:16.217163+0000","last_fullsized":"2024-08-19T10:09:16.217163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:43:26.828213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218719+0000","last_change":"2024-08-19T09:51:22.631476+0000","last_active":"2024-08-19T10:09:17.218719+0000","last_peered":"2024-08-19T10:09:17.218719+0000","last_clean":"2024-08-19T10:09:17.218719+0000","last_became_active":"2024-08-19T09:51:22.630612+0000","last_became_peered":"2024-08-19T09:51:22.630612+0000","last_unstale":"2024-08-19T10:09:17.218719+0000","last_undegraded":"2024-08-19T10:09:17.218719+0000","last_fullsized":"2024-08-19T10:09:17.218719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:15.371670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1369,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.680505+0000","last_change":"2024-08-19T09:51:22.490285+0000","last_active":"2024-08-19T10:09:16.680505+0000","last_peered":"2024-08-19T10:09:16.680505+0000","last_clean":"2024-08-19T10:09:16.680505+0000","last_became_active":"2024-08-19T09:51:22.490018+0000","last_became_peered":"2024-08-19T09:51:22.490018+0000","last_unstale":"2024-08-19T10:09:16.680505+0000","last_undegraded":"2024-08-19T10:09:16.680505+0000","last_fullsized":"2024-08-19T10:09:16.680505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:57:54.192511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479361+0000","last_change":"2024-08-19T09:51:22.646201+0000","last_active":"2024-08-19T10:09:18.479361+0000","last_peered":"2024-08-19T10:09:18.479361+0000","last_clean":"2024-08-19T10:09:18.479361+0000","last_became_active":"2024-08-19T09:51:22.644785+0000","last_became_peered":"2024-08-19T09:51:22.644785+0000","last_unstale":"2024-08-19T10:09:18.479361+0000","last_undegraded":"2024-08-19T10:09:18.479361+0000","last_fullsized":"2024-08-19T10:09:18.479361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:30:51.586745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478788+0000","last_change":"2024-08-19T09:51:22.633554+0000","last_active":"2024-08-19T10:09:18.478788+0000","last_peered":"2024-08-19T10:09:18.478788+0000","last_clean":"2024-08-19T10:09:18.478788+0000","last_became_active":"2024-08-19T09:51:22.618248+0000","last_became_peered":"2024-08-19T09:51:22.618248+0000","last_unstale":"2024-08-19T10:09:18.478788+0000","last_undegraded":"2024-08-19T10:09:18.478788+0000","last_fullsized":"2024-08-19T10:09:18.478788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:04:45.586830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217017+0000","last_change":"2024-08-19T09:51:22.559064+0000","last_active":"2024-08-19T10:09:16.217017+0000","last_peered":"2024-08-19T10:09:16.217017+0000","last_clean":"2024-08-19T10:09:16.217017+0000","last_became_active":"2024-08-19T09:51:22.558815+0000","last_became_peered":"2024-08-19T09:51:22.558815+0000","last_unstale":"2024-08-19T10:09:16.217017+0000","last_undegraded":"2024-08-19T10:09:16.217017+0000","last_fullsized":"2024-08-19T10:09:16.217017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:06:02.138369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217465+0000","last_change":"2024-08-19T09:51:22.582602+0000","last_active":"2024-08-19T10:09:16.217465+0000","last_peered":"2024-08-19T10:09:16.217465+0000","last_clean":"2024-08-19T10:09:16.217465+0000","last_became_active":"2024-08-19T09:51:22.581840+0000","last_became_peered":"2024-08-19T09:51:22.581840+0000","last_unstale":"2024-08-19T10:09:16.217465+0000","last_undegraded":"2024-08-19T10:09:16.217465+0000","last_fullsized":"2024-08-19T10:09:16.217465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:24:46.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.680151+0000","last_change":"2024-08-19T09:55:22.701986+0000","last_active":"2024-08-19T10:09:16.680151+0000","last_peered":"2024-08-19T10:09:16.680151+0000","last_clean":"2024-08-19T10:09:16.680151+0000","last_became_active":"2024-08-19T09:55:22.701660+0000","last_became_peered":"2024-08-19T09:55:22.701660+0000","last_unstale":"2024-08-19T10:09:16.680151+0000","last_undegraded":"2024-08-19T10:09:16.680151+0000","last_fullsized":"2024-08-19T10:09:16.680151+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:56:19.752844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478819+0000","last_change":"2024-08-19T09:51:22.565151+0000","last_active":"2024-08-19T10:09:18.478819+0000","last_peered":"2024-08-19T10:09:18.478819+0000","last_clean":"2024-08-19T10:09:18.478819+0000","last_became_active":"2024-08-19T09:51:22.564919+0000","last_became_peered":"2024-08-19T09:51:22.564919+0000","last_unstale":"2024-08-19T10:09:18.478819+0000","last_undegraded":"2024-08-19T10:09:18.478819+0000","last_fullsized":"2024-08-19T10:09:18.478819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:05.826534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219907+0000","last_change":"2024-08-19T09:51:22.612829+0000","last_active":"2024-08-19T10:09:17.219907+0000","last_peered":"2024-08-19T10:09:17.219907+0000","last_clean":"2024-08-19T10:09:17.219907+0000","last_became_active":"2024-08-19T09:51:22.612552+0000","last_became_peered":"2024-08-19T09:51:22.612552+0000","last_unstale":"2024-08-19T10:09:17.219907+0000","last_undegraded":"2024-08-19T10:09:17.219907+0000","last_fullsized":"2024-08-19T10:09:17.219907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:58:05.134880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218803+0000","last_change":"2024-08-19T09:51:22.609929+0000","last_active":"2024-08-19T10:09:17.218803+0000","last_peered":"2024-08-19T10:09:17.218803+0000","last_clean":"2024-08-19T10:09:17.218803+0000","last_became_active":"2024-08-19T09:51:22.609674+0000","last_became_peered":"2024-08-19T09:51:22.609674+0000","last_unstale":"2024-08-19T10:09:17.218803+0000","last_undegraded":"2024-08-19T10:09:17.218803+0000","last_fullsized":"2024-08-19T10:09:17.218803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:41:23.025542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1373,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479455+0000","last_change":"2024-08-19T09:51:22.652416+0000","last_active":"2024-08-19T10:09:18.479455+0000","last_peered":"2024-08-19T10:09:18.479455+0000","last_clean":"2024-08-19T10:09:18.479455+0000","last_became_active":"2024-08-19T09:51:22.652083+0000","last_became_peered":"2024-08-19T09:51:22.652083+0000","last_unstale":"2024-08-19T10:09:18.479455+0000","last_undegraded":"2024-08-19T10:09:18.479455+0000","last_fullsized":"2024-08-19T10:09:18.479455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:47.783611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575086,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73720,"kb_used_data":6436,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298120,"statfs":{"total":96636764160,"available":96561274880,"internally_reserved":0,"allocated":6590464,"data_stored":2911767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74128,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297712,"statfs":{"total":96636764160,"available":96560857088,"internally_reserved":0,"allocated":7000064,"data_stored":3313724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69424,"kb_used_data":6292,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302416,"statfs":{"total":96636764160,"available":96565673984,"internally_reserved":0,"allocated":6443008,"data_stored":2760234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607785,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81992,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289848,"statfs":{"total":96636764160,"available":96552804352,"internally_reserved":0,"allocated":6537216,"data_stored":2864612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69960,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301880,"statfs":{"total":96636764160,"available":96565125120,"internally_reserved":0,"allocated":6995968,"data_stored":3323903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73588,"kb_used_data":6304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298252,"statfs":{"total":96636764160,"available":96561410048,"internally_reserved":0,"allocated":6455296,"data_stored":2788462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T10:09:21.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T10:09:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.635+0000 7f3d90f14640 10 monclient: get_monmap_and_config 2024-08-19T10:09:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.635+0000 7f3d90f14640 10 monclient: build_initial_monmap 2024-08-19T10:09:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.635+0000 7f3d90f14640 10 monclient: monmap: 2024-08-19T10:09:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:21.636167+0000 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:21.636167+0000 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:21.637 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.635+0000 7f3d90f14640 1 Processor -- start 2024-08-19T10:09:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 1 -- start start 2024-08-19T10:09:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient: init 2024-08-19T10:09:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient: _add_conns ranks=[3,2,1,4,8,6,0,5,7] 2024-08-19T10:09:21.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient(hunting): picked mon.noname-g con 0x7f3d8c0a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8c0694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient(hunting): picked mon.noname-f con 0x7f3d8c0a46a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a3c70 con 0x7f3d8c0a46a0 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a33d0 con 0x7f3d8c0694b0 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a36b0 con 0x7f3d8c0a6830 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d90f14640 10 monclient(hunting): authenticate will time out at 1817.712601s 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d8ad9d640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f3d8c0a46a0 legacy=0x7f3d8c141850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d83fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0694b0 0x7f3d8c0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49104/0 (socket says 172.21.15.1:49104) 2024-08-19T10:09:21.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.636+0000 7f3d83fff640 1 -- 172.21.15.1:0/1214537484 learned_addr learned my addr 172.21.15.1:0/1214537484 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.638+0000 7f3d83fff640 1 -- 172.21.15.1:0/1214537484 >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0694b0 legacy=0x7f3d8c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.638+0000 7f3d8a59c640 1 -- 172.21.15.1:0/1214537484 >> v1:172.21.15.82:6790/0 conn(0x7f3d8c0a6830 legacy=0x7f3d8c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4291182597 0 0) 0x7f3d8c0a33d0 con 0x7f3d8c0694b0 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): my global_id is 184943 2024-08-19T10:09:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d78003520 con 0x7f3d8c0694b0 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727136437 0 0) 0x7f3d8c0a36b0 con 0x7f3d8c0a6830 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient(hunting): my global_id is 214347 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c0a33d0 con 0x7f3d8c0a6830 2024-08-19T10:09:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 734476237 0 0) 0x7f3d78003520 con 0x7f3d8c0694b0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 >> v1:172.21.15.82:6790/0 conn(0x7f3d8c0a6830 legacy=0x7f3d8c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 >> v1:172.21.15.82:6789/0 conn(0x7f3d8c0a46a0 legacy=0x7f3d8c141850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 monclient: found mon.noname-b 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c06dbb0 con 0x7f3d8c0694b0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: _finish_auth 0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d74001ae0 con 0x7f3d8c0694b0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: dump: 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d90f14640 5 monclient: authenticate success, global_id 184943 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d90f14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d740048a0 con 0x7f3d8c0694b0 2024-08-19T10:09:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 1 -- 172.21.15.1:0/1214537484 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d74004e10 con 0x7f3d8c0694b0 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d89d9b640 10 monclient: dump: 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d90f14640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.639+0000 7f3d90f14640 4 set_mon_vals no callback set 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 10 monclient: shutdown 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 1 -- 172.21.15.1:0/1214537484 >> v1:172.21.15.1:6790/0 conn(0x7f3d8c0694b0 legacy=0x7f3d8c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 1 -- 172.21.15.1:0/1214537484 shutdown_connections 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 1 -- 172.21.15.1:0/1214537484 >> 172.21.15.1:0/1214537484 conn(0x7f3d8c0a9020 msgr2=0x7f3d8c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 1 -- 172.21.15.1:0/1214537484 shutdown_connections 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.640+0000 7f3d90f14640 1 -- 172.21.15.1:0/1214537484 wait complete. 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 10 monclient: build_initial_monmap 2024-08-19T10:09:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 10 monclient: monmap: 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 1 librados: starting msgr at 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 1 librados: starting objecter 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 1 Processor -- start 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 1 -- start start 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 1 librados: setting wanted keys 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 1 librados: calling monclient init 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.641+0000 7f3d90f14640 10 monclient: init 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient: _add_conns ranks=[8,5,1,0,3,7,4,6,2] 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient(hunting): picked mon.noname-e con 0x7f3d8c0694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient(hunting): picked mon.noname-h con 0x7f3d8c0a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient(hunting): picked mon.noname-f con 0x7f3d8c0a6830 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c1380a0 con 0x7f3d8c0a6830 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c138c30 con 0x7f3d8c0a46a0 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c1cf280 con 0x7f3d8c0694b0 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d90f14640 10 monclient(hunting): authenticate will time out at 1817.718249s 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d8ad9d640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f3d8c0a6830 legacy=0x7f3d8c1d66c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d8a59c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f3d8c0694b0 0x7f3d8c1ce460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48944/0 (socket says 172.21.15.1:48944) 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.642+0000 7f3d8a59c640 1 -- 172.21.15.1:0/1345458129 learned_addr learned my addr 172.21.15.1:0/1345458129 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.644+0000 7f3d8a59c640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.1:6793/0 conn(0x7f3d8c0694b0 legacy=0x7f3d8c1ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.644+0000 7f3d82ffd640 10 client.?.objecter ms_handle_connect 0x7f3d8c0694b0 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.644+0000 7f3d82ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.644+0000 7f3d82ffd640 10 client.?.objecter ms_handle_connect 0x7f3d8c0a46a0 2024-08-19T10:09:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.644+0000 7f3d82ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.644+0000 7f3d83fff640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.82:6791/0 conn(0x7f3d8c0a46a0 legacy=0x7f3d8c1ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:21.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:21.843+0000 7f3d8ad9d640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.82:6789/0 conn(0x7f3d8c0a6830 legacy=0x7f3d8c1d66c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.168+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215107972 0 0) 0x7f3d8c138c30 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): my global_id is 214331 2024-08-19T10:09:22.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d6c004eb0 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2811641437 0 0) 0x7f3d6c004eb0 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d8c138c30 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271461775 0 0) 0x7f3d8c1cf280 con 0x7f3d8c0694b0 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): my global_id is 204605 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d6c004eb0 con 0x7f3d8c0694b0 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d74001d30 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:22.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.176 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.169+0000 7f3d82ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1112362069 0 0) 0x7f3d8c138c30 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.1:6793/0 conn(0x7f3d8c0694b0 legacy=0x7f3d8c1ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.82:6789/0 conn(0x7f3d8c0a6830 legacy=0x7f3d8c1d66c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 monclient: found mon.h 2024-08-19T10:09:22.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c138f10 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 5 monclient: authenticate success, global_id 214331 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 10 monclient: _renew_subs 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8c068b30 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 10 client.214331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:22.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 10 monclient: _renew_subs 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d74001320 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d8c138950 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d90f14640 1 librados: init done 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d74005380 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d82ffd640 10 monclient: dump: 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.180 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.170+0000 7f3d8959a640 4 set_mon_vals no callback set 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.172+0000 7f3d90f14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.172+0000 7f3d90f14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.172+0000 7f3d90f14640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.82:6791/0 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.172+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d8c0ab530 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3d74011860 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f3d7404de20 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d82ffd640 10 client.214331.objecter ms_dispatch 0x7f3d8c000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:22.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d82ffd640 3 client.214331.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:22.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d82ffd640 3 client.214331.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:22.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d82ffd640 20 client.214331.objecter dump_active .. 0 homeless 2024-08-19T10:09:22.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.173+0000 7f3d8a59c640 10 monclient: get_auth_request con 0x7f3d6c040e00 auth_method 0 2024-08-19T10:09:22.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.176+0000 7f3d82ffd640 10 client.214331.objecter ms_handle_connect 0x7f3d6c040e00 2024-08-19T10:09:22.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.181+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mon.5 v1:172.21.15.82:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f3d740809c0 con 0x7f3d8c0a46a0 2024-08-19T10:09:22.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.181+0000 7f3d82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.181+0000 7f3d82ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:22.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.366+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d8c0a33d0 con 0x7f3d6c040e00 2024-08-19T10:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.369+0000 7f3d82ffd640 1 -- 172.21.15.1:0/1345458129 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114209 (unknown 2965378022 0 3985610741) 0x7f3d8c0a33d0 con 0x7f3d6c040e00 2024-08-19T10:09:22.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T10:09:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 10 librados: watch_flush enter 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 10 librados: watch_flush exit 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 20 client.214331.objecter shutdown clearing up homeless session... 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 10 client.214331.objecter successfully canceled tick 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3d6c040e00 legacy=0x7f3d6c0432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 10 monclient: shutdown 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 >> v1:172.21.15.82:6791/0 conn(0x7f3d8c0a46a0 legacy=0x7f3d8c1ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 shutdown_connections 2024-08-19T10:09:22.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 >> 172.21.15.1:0/1345458129 conn(0x7f3d8c0a9020 msgr2=0x7f3d8c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:22.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 shutdown_connections 2024-08-19T10:09:22.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 -- 172.21.15.1:0/1345458129 wait complete. 2024-08-19T10:09:22.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.371+0000 7f3d90f14640 1 librados: shutdown 2024-08-19T10:09:22.382 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1281,"stamp":"2024-08-19T10:09:21.060770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":442812,"kb_used_data":39084,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565788228,"statfs":{"total":579820584960,"available":579367145472,"internally_reserved":0,"allocated":40022016,"data_stored":17962702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":413260450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001024"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1492,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478370+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T10:09:18.478370+0000","last_peered":"2024-08-19T10:09:18.478370+0000","last_clean":"2024-08-19T10:09:18.478370+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T10:09:18.478370+0000","last_undegraded":"2024-08-19T10:09:18.478370+0000","last_fullsized":"2024-08-19T10:09:18.478370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1398,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219416+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T10:09:17.219416+0000","last_peered":"2024-08-19T10:09:17.219416+0000","last_clean":"2024-08-19T10:09:17.219416+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T10:09:17.219416+0000","last_undegraded":"2024-08-19T10:09:17.219416+0000","last_fullsized":"2024-08-19T10:09:17.219416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1401,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478445+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T10:09:18.478445+0000","last_peered":"2024-08-19T10:09:18.478445+0000","last_clean":"2024-08-19T10:09:18.478445+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T10:09:18.478445+0000","last_undegraded":"2024-08-19T10:09:18.478445+0000","last_fullsized":"2024-08-19T10:09:18.478445+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1397,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216502+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T10:09:17.216502+0000","last_peered":"2024-08-19T10:09:17.216502+0000","last_clean":"2024-08-19T10:09:17.216502+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T10:09:17.216502+0000","last_undegraded":"2024-08-19T10:09:17.216502+0000","last_fullsized":"2024-08-19T10:09:17.216502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1399,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216190+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T10:09:17.216190+0000","last_peered":"2024-08-19T10:09:17.216190+0000","last_clean":"2024-08-19T10:09:17.216190+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T10:09:17.216190+0000","last_undegraded":"2024-08-19T10:09:17.216190+0000","last_fullsized":"2024-08-19T10:09:17.216190+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1391,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217571+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T10:09:16.217571+0000","last_peered":"2024-08-19T10:09:16.217571+0000","last_clean":"2024-08-19T10:09:16.217571+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T10:09:16.217571+0000","last_undegraded":"2024-08-19T10:09:16.217571+0000","last_fullsized":"2024-08-19T10:09:16.217571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1401,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478316+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T10:09:18.478316+0000","last_peered":"2024-08-19T10:09:18.478316+0000","last_clean":"2024-08-19T10:09:18.478316+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T10:09:18.478316+0000","last_undegraded":"2024-08-19T10:09:18.478316+0000","last_fullsized":"2024-08-19T10:09:18.478316+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1390,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217636+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T10:09:16.217636+0000","last_peered":"2024-08-19T10:09:16.217636+0000","last_clean":"2024-08-19T10:09:16.217636+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T10:09:16.217636+0000","last_undegraded":"2024-08-19T10:09:16.217636+0000","last_fullsized":"2024-08-19T10:09:16.217636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1400,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478414+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T10:09:18.478414+0000","last_peered":"2024-08-19T10:09:18.478414+0000","last_clean":"2024-08-19T10:09:18.478414+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T10:09:18.478414+0000","last_undegraded":"2024-08-19T10:09:18.478414+0000","last_fullsized":"2024-08-19T10:09:18.478414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1","version":"23'7","reported_seq":1396,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478569+0000","last_change":"2024-08-19T09:51:22.586587+0000","last_active":"2024-08-19T10:09:18.478569+0000","last_peered":"2024-08-19T10:09:18.478569+0000","last_clean":"2024-08-19T10:09:18.478569+0000","last_became_active":"2024-08-19T09:51:22.585789+0000","last_became_peered":"2024-08-19T09:51:22.585789+0000","last_unstale":"2024-08-19T10:09:18.478569+0000","last_undegraded":"2024-08-19T10:09:18.478569+0000","last_fullsized":"2024-08-19T10:09:18.478569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:25:11.479863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217047+0000","last_change":"2024-08-19T09:51:22.586723+0000","last_active":"2024-08-19T10:09:17.217047+0000","last_peered":"2024-08-19T10:09:17.217047+0000","last_clean":"2024-08-19T10:09:17.217047+0000","last_became_active":"2024-08-19T09:51:22.583401+0000","last_became_peered":"2024-08-19T09:51:22.583401+0000","last_unstale":"2024-08-19T10:09:17.217047+0000","last_undegraded":"2024-08-19T10:09:17.217047+0000","last_fullsized":"2024-08-19T10:09:17.217047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:56:50.500489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217571+0000","last_change":"2024-08-19T09:51:22.588184+0000","last_active":"2024-08-19T10:09:17.217571+0000","last_peered":"2024-08-19T10:09:17.217571+0000","last_clean":"2024-08-19T10:09:17.217571+0000","last_became_active":"2024-08-19T09:51:22.587936+0000","last_became_peered":"2024-08-19T09:51:22.587936+0000","last_unstale":"2024-08-19T10:09:17.217571+0000","last_undegraded":"2024-08-19T10:09:17.217571+0000","last_fullsized":"2024-08-19T10:09:17.217571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:22:07.623601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217092+0000","last_change":"2024-08-19T09:51:22.640927+0000","last_active":"2024-08-19T10:09:17.217092+0000","last_peered":"2024-08-19T10:09:17.217092+0000","last_clean":"2024-08-19T10:09:17.217092+0000","last_became_active":"2024-08-19T09:51:22.640604+0000","last_became_peered":"2024-08-19T09:51:22.640604+0000","last_unstale":"2024-08-19T10:09:17.217092+0000","last_undegraded":"2024-08-19T10:09:17.217092+0000","last_fullsized":"2024-08-19T10:09:17.217092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:38:41.853584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1383,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478207+0000","last_change":"2024-08-19T09:51:22.581514+0000","last_active":"2024-08-19T10:09:18.478207+0000","last_peered":"2024-08-19T10:09:18.478207+0000","last_clean":"2024-08-19T10:09:18.478207+0000","last_became_active":"2024-08-19T09:51:22.581246+0000","last_became_peered":"2024-08-19T09:51:22.581246+0000","last_unstale":"2024-08-19T10:09:18.478207+0000","last_undegraded":"2024-08-19T10:09:18.478207+0000","last_fullsized":"2024-08-19T10:09:18.478207+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:12:50.239096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.220013+0000","last_change":"2024-08-19T09:51:22.620559+0000","last_active":"2024-08-19T10:09:17.220013+0000","last_peered":"2024-08-19T10:09:17.220013+0000","last_clean":"2024-08-19T10:09:17.220013+0000","last_became_active":"2024-08-19T09:51:22.620042+0000","last_became_peered":"2024-08-19T09:51:22.620042+0000","last_unstale":"2024-08-19T10:09:17.220013+0000","last_undegraded":"2024-08-19T10:09:17.220013+0000","last_fullsized":"2024-08-19T10:09:17.220013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:27:35.980026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218905+0000","last_change":"2024-08-19T09:51:22.606649+0000","last_active":"2024-08-19T10:09:17.218905+0000","last_peered":"2024-08-19T10:09:17.218905+0000","last_clean":"2024-08-19T10:09:17.218905+0000","last_became_active":"2024-08-19T09:51:22.606054+0000","last_became_peered":"2024-08-19T09:51:22.606054+0000","last_unstale":"2024-08-19T10:09:17.218905+0000","last_undegraded":"2024-08-19T10:09:17.218905+0000","last_fullsized":"2024-08-19T10:09:17.218905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:23:52.344010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217755+0000","last_change":"2024-08-19T09:51:22.653723+0000","last_active":"2024-08-19T10:09:17.217755+0000","last_peered":"2024-08-19T10:09:17.217755+0000","last_clean":"2024-08-19T10:09:17.217755+0000","last_became_active":"2024-08-19T09:51:22.653455+0000","last_became_peered":"2024-08-19T09:51:22.653455+0000","last_unstale":"2024-08-19T10:09:17.217755+0000","last_undegraded":"2024-08-19T10:09:17.217755+0000","last_fullsized":"2024-08-19T10:09:17.217755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:25:13.028210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.679991+0000","last_change":"2024-08-19T09:51:22.560997+0000","last_active":"2024-08-19T10:09:16.679991+0000","last_peered":"2024-08-19T10:09:16.679991+0000","last_clean":"2024-08-19T10:09:16.679991+0000","last_became_active":"2024-08-19T09:51:22.553419+0000","last_became_peered":"2024-08-19T09:51:22.553419+0000","last_unstale":"2024-08-19T10:09:16.679991+0000","last_undegraded":"2024-08-19T10:09:16.679991+0000","last_fullsized":"2024-08-19T10:09:16.679991+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:25.353142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216939+0000","last_change":"2024-08-19T09:51:22.554649+0000","last_active":"2024-08-19T10:09:17.216939+0000","last_peered":"2024-08-19T10:09:17.216939+0000","last_clean":"2024-08-19T10:09:17.216939+0000","last_became_active":"2024-08-19T09:51:22.554397+0000","last_became_peered":"2024-08-19T09:51:22.554397+0000","last_unstale":"2024-08-19T10:09:17.216939+0000","last_undegraded":"2024-08-19T10:09:17.216939+0000","last_fullsized":"2024-08-19T10:09:17.216939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:36:10.709773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1378,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217364+0000","last_change":"2024-08-19T09:55:22.703762+0000","last_active":"2024-08-19T10:09:16.217364+0000","last_peered":"2024-08-19T10:09:16.217364+0000","last_clean":"2024-08-19T10:09:16.217364+0000","last_became_active":"2024-08-19T09:55:22.703438+0000","last_became_peered":"2024-08-19T09:55:22.703438+0000","last_unstale":"2024-08-19T10:09:16.217364+0000","last_undegraded":"2024-08-19T10:09:16.217364+0000","last_fullsized":"2024-08-19T10:09:16.217364+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:29:17.373388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217274+0000","last_change":"2024-08-19T09:51:22.529738+0000","last_active":"2024-08-19T10:09:16.217274+0000","last_peered":"2024-08-19T10:09:16.217274+0000","last_clean":"2024-08-19T10:09:16.217274+0000","last_became_active":"2024-08-19T09:51:22.528219+0000","last_became_peered":"2024-08-19T09:51:22.528219+0000","last_unstale":"2024-08-19T10:09:16.217274+0000","last_undegraded":"2024-08-19T10:09:16.217274+0000","last_fullsized":"2024-08-19T10:09:16.217274+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:44.262012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219294+0000","last_change":"2024-08-19T09:51:22.614370+0000","last_active":"2024-08-19T10:09:17.219294+0000","last_peered":"2024-08-19T10:09:17.219294+0000","last_clean":"2024-08-19T10:09:17.219294+0000","last_became_active":"2024-08-19T09:51:22.614136+0000","last_became_peered":"2024-08-19T09:51:22.614136+0000","last_unstale":"2024-08-19T10:09:17.219294+0000","last_undegraded":"2024-08-19T10:09:17.219294+0000","last_fullsized":"2024-08-19T10:09:17.219294+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:14:54.869367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218815+0000","last_change":"2024-08-19T09:51:22.587447+0000","last_active":"2024-08-19T10:09:17.218815+0000","last_peered":"2024-08-19T10:09:17.218815+0000","last_clean":"2024-08-19T10:09:17.218815+0000","last_became_active":"2024-08-19T09:51:22.587149+0000","last_became_peered":"2024-08-19T09:51:22.587149+0000","last_unstale":"2024-08-19T10:09:17.218815+0000","last_undegraded":"2024-08-19T10:09:17.218815+0000","last_fullsized":"2024-08-19T10:09:17.218815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:08:36.757587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479995+0000","last_change":"2024-08-19T09:51:22.638208+0000","last_active":"2024-08-19T10:09:18.479995+0000","last_peered":"2024-08-19T10:09:18.479995+0000","last_clean":"2024-08-19T10:09:18.479995+0000","last_became_active":"2024-08-19T09:51:22.634477+0000","last_became_peered":"2024-08-19T09:51:22.634477+0000","last_unstale":"2024-08-19T10:09:18.479995+0000","last_undegraded":"2024-08-19T10:09:18.479995+0000","last_fullsized":"2024-08-19T10:09:18.479995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:01:43.314792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217657+0000","last_change":"2024-08-19T09:51:22.594096+0000","last_active":"2024-08-19T10:09:17.217657+0000","last_peered":"2024-08-19T10:09:17.217657+0000","last_clean":"2024-08-19T10:09:17.217657+0000","last_became_active":"2024-08-19T09:51:22.590688+0000","last_became_peered":"2024-08-19T09:51:22.590688+0000","last_unstale":"2024-08-19T10:09:17.217657+0000","last_undegraded":"2024-08-19T10:09:17.217657+0000","last_fullsized":"2024-08-19T10:09:17.217657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:51.718926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217017+0000","last_change":"2024-08-19T09:51:22.537238+0000","last_active":"2024-08-19T10:09:16.217017+0000","last_peered":"2024-08-19T10:09:16.217017+0000","last_clean":"2024-08-19T10:09:16.217017+0000","last_became_active":"2024-08-19T09:51:22.536584+0000","last_became_peered":"2024-08-19T09:51:22.536584+0000","last_unstale":"2024-08-19T10:09:16.217017+0000","last_undegraded":"2024-08-19T10:09:16.217017+0000","last_fullsized":"2024-08-19T10:09:16.217017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:42:52.494448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1369,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.679764+0000","last_change":"2024-08-19T09:51:22.556447+0000","last_active":"2024-08-19T10:09:16.679764+0000","last_peered":"2024-08-19T10:09:16.679764+0000","last_clean":"2024-08-19T10:09:16.679764+0000","last_became_active":"2024-08-19T09:51:22.556182+0000","last_became_peered":"2024-08-19T09:51:22.556182+0000","last_unstale":"2024-08-19T10:09:16.679764+0000","last_undegraded":"2024-08-19T10:09:16.679764+0000","last_fullsized":"2024-08-19T10:09:16.679764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:04:06.155397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217049+0000","last_change":"2024-08-19T09:51:22.672202+0000","last_active":"2024-08-19T10:09:16.217049+0000","last_peered":"2024-08-19T10:09:16.217049+0000","last_clean":"2024-08-19T10:09:16.217049+0000","last_became_active":"2024-08-19T09:51:22.671918+0000","last_became_peered":"2024-08-19T09:51:22.671918+0000","last_unstale":"2024-08-19T10:09:16.217049+0000","last_undegraded":"2024-08-19T10:09:16.217049+0000","last_fullsized":"2024-08-19T10:09:16.217049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:56:23.219267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217181+0000","last_change":"2024-08-19T09:51:22.605834+0000","last_active":"2024-08-19T10:09:16.217181+0000","last_peered":"2024-08-19T10:09:16.217181+0000","last_clean":"2024-08-19T10:09:16.217181+0000","last_became_active":"2024-08-19T09:51:22.605586+0000","last_became_peered":"2024-08-19T09:51:22.605586+0000","last_unstale":"2024-08-19T10:09:16.217181+0000","last_undegraded":"2024-08-19T10:09:16.217181+0000","last_fullsized":"2024-08-19T10:09:16.217181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:53:29.756052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217163+0000","last_change":"2024-08-19T09:51:22.551390+0000","last_active":"2024-08-19T10:09:16.217163+0000","last_peered":"2024-08-19T10:09:16.217163+0000","last_clean":"2024-08-19T10:09:16.217163+0000","last_became_active":"2024-08-19T09:51:22.551119+0000","last_became_peered":"2024-08-19T09:51:22.551119+0000","last_unstale":"2024-08-19T10:09:16.217163+0000","last_undegraded":"2024-08-19T10:09:16.217163+0000","last_fullsized":"2024-08-19T10:09:16.217163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:43:26.828213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218719+0000","last_change":"2024-08-19T09:51:22.631476+0000","last_active":"2024-08-19T10:09:17.218719+0000","last_peered":"2024-08-19T10:09:17.218719+0000","last_clean":"2024-08-19T10:09:17.218719+0000","last_became_active":"2024-08-19T09:51:22.630612+0000","last_became_peered":"2024-08-19T09:51:22.630612+0000","last_unstale":"2024-08-19T10:09:17.218719+0000","last_undegraded":"2024-08-19T10:09:17.218719+0000","last_fullsized":"2024-08-19T10:09:17.218719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:15.371670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1369,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.680505+0000","last_change":"2024-08-19T09:51:22.490285+0000","last_active":"2024-08-19T10:09:16.680505+0000","last_peered":"2024-08-19T10:09:16.680505+0000","last_clean":"2024-08-19T10:09:16.680505+0000","last_became_active":"2024-08-19T09:51:22.490018+0000","last_became_peered":"2024-08-19T09:51:22.490018+0000","last_unstale":"2024-08-19T10:09:16.680505+0000","last_undegraded":"2024-08-19T10:09:16.680505+0000","last_fullsized":"2024-08-19T10:09:16.680505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:57:54.192511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479361+0000","last_change":"2024-08-19T09:51:22.646201+0000","last_active":"2024-08-19T10:09:18.479361+0000","last_peered":"2024-08-19T10:09:18.479361+0000","last_clean":"2024-08-19T10:09:18.479361+0000","last_became_active":"2024-08-19T09:51:22.644785+0000","last_became_peered":"2024-08-19T09:51:22.644785+0000","last_unstale":"2024-08-19T10:09:18.479361+0000","last_undegraded":"2024-08-19T10:09:18.479361+0000","last_fullsized":"2024-08-19T10:09:18.479361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:30:51.586745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478788+0000","last_change":"2024-08-19T09:51:22.633554+0000","last_active":"2024-08-19T10:09:18.478788+0000","last_peered":"2024-08-19T10:09:18.478788+0000","last_clean":"2024-08-19T10:09:18.478788+0000","last_became_active":"2024-08-19T09:51:22.618248+0000","last_became_peered":"2024-08-19T09:51:22.618248+0000","last_unstale":"2024-08-19T10:09:18.478788+0000","last_undegraded":"2024-08-19T10:09:18.478788+0000","last_fullsized":"2024-08-19T10:09:18.478788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:04:45.586830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217017+0000","last_change":"2024-08-19T09:51:22.559064+0000","last_active":"2024-08-19T10:09:16.217017+0000","last_peered":"2024-08-19T10:09:16.217017+0000","last_clean":"2024-08-19T10:09:16.217017+0000","last_became_active":"2024-08-19T09:51:22.558815+0000","last_became_peered":"2024-08-19T09:51:22.558815+0000","last_unstale":"2024-08-19T10:09:16.217017+0000","last_undegraded":"2024-08-19T10:09:16.217017+0000","last_fullsized":"2024-08-19T10:09:16.217017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:06:02.138369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217465+0000","last_change":"2024-08-19T09:51:22.582602+0000","last_active":"2024-08-19T10:09:16.217465+0000","last_peered":"2024-08-19T10:09:16.217465+0000","last_clean":"2024-08-19T10:09:16.217465+0000","last_became_active":"2024-08-19T09:51:22.581840+0000","last_became_peered":"2024-08-19T09:51:22.581840+0000","last_unstale":"2024-08-19T10:09:16.217465+0000","last_undegraded":"2024-08-19T10:09:16.217465+0000","last_fullsized":"2024-08-19T10:09:16.217465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:24:46.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.680151+0000","last_change":"2024-08-19T09:55:22.701986+0000","last_active":"2024-08-19T10:09:16.680151+0000","last_peered":"2024-08-19T10:09:16.680151+0000","last_clean":"2024-08-19T10:09:16.680151+0000","last_became_active":"2024-08-19T09:55:22.701660+0000","last_became_peered":"2024-08-19T09:55:22.701660+0000","last_unstale":"2024-08-19T10:09:16.680151+0000","last_undegraded":"2024-08-19T10:09:16.680151+0000","last_fullsized":"2024-08-19T10:09:16.680151+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:56:19.752844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478819+0000","last_change":"2024-08-19T09:51:22.565151+0000","last_active":"2024-08-19T10:09:18.478819+0000","last_peered":"2024-08-19T10:09:18.478819+0000","last_clean":"2024-08-19T10:09:18.478819+0000","last_became_active":"2024-08-19T09:51:22.564919+0000","last_became_peered":"2024-08-19T09:51:22.564919+0000","last_unstale":"2024-08-19T10:09:18.478819+0000","last_undegraded":"2024-08-19T10:09:18.478819+0000","last_fullsized":"2024-08-19T10:09:18.478819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:05.826534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219907+0000","last_change":"2024-08-19T09:51:22.612829+0000","last_active":"2024-08-19T10:09:17.219907+0000","last_peered":"2024-08-19T10:09:17.219907+0000","last_clean":"2024-08-19T10:09:17.219907+0000","last_became_active":"2024-08-19T09:51:22.612552+0000","last_became_peered":"2024-08-19T09:51:22.612552+0000","last_unstale":"2024-08-19T10:09:17.219907+0000","last_undegraded":"2024-08-19T10:09:17.219907+0000","last_fullsized":"2024-08-19T10:09:17.219907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:58:05.134880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218803+0000","last_change":"2024-08-19T09:51:22.609929+0000","last_active":"2024-08-19T10:09:17.218803+0000","last_peered":"2024-08-19T10:09:17.218803+0000","last_clean":"2024-08-19T10:09:17.218803+0000","last_became_active":"2024-08-19T09:51:22.609674+0000","last_became_peered":"2024-08-19T09:51:22.609674+0000","last_unstale":"2024-08-19T10:09:17.218803+0000","last_undegraded":"2024-08-19T10:09:17.218803+0000","last_fullsized":"2024-08-19T10:09:17.218803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:41:23.025542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1373,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479455+0000","last_change":"2024-08-19T09:51:22.652416+0000","last_active":"2024-08-19T10:09:18.479455+0000","last_peered":"2024-08-19T10:09:18.479455+0000","last_clean":"2024-08-19T10:09:18.479455+0000","last_became_active":"2024-08-19T09:51:22.652083+0000","last_became_peered":"2024-08-19T09:51:22.652083+0000","last_unstale":"2024-08-19T10:09:18.479455+0000","last_undegraded":"2024-08-19T10:09:18.479455+0000","last_fullsized":"2024-08-19T10:09:18.479455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:47.783611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575086,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73720,"kb_used_data":6436,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298120,"statfs":{"total":96636764160,"available":96561274880,"internally_reserved":0,"allocated":6590464,"data_stored":2911767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74128,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297712,"statfs":{"total":96636764160,"available":96560857088,"internally_reserved":0,"allocated":7000064,"data_stored":3313724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69424,"kb_used_data":6292,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302416,"statfs":{"total":96636764160,"available":96565673984,"internally_reserved":0,"allocated":6443008,"data_stored":2760234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607785,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81992,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289848,"statfs":{"total":96636764160,"available":96552804352,"internally_reserved":0,"allocated":6537216,"data_stored":2864612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69960,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301880,"statfs":{"total":96636764160,"available":96565125120,"internally_reserved":0,"allocated":6995968,"data_stored":3323903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73588,"kb_used_data":6304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298252,"statfs":{"total":96636764160,"available":96561410048,"internally_reserved":0,"allocated":6455296,"data_stored":2788462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T10:09:22.384 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-19T10:09:22.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.481+0000 7fd8a466e640 10 monclient: get_monmap_and_config 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.481+0000 7fd8a466e640 10 monclient: build_initial_monmap 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.481+0000 7fd8a466e640 10 monclient: monmap: 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:22.482799+0000 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:22.482799+0000 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:22.483 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:22.484 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.482+0000 7fd8a466e640 1 Processor -- start 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.482+0000 7fd8a466e640 1 -- start start 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.482+0000 7fd8a466e640 10 monclient: init 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient: _add_conns ranks=[8,3,6,0,4,5,2,7,1] 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient(hunting): picked mon.noname-e con 0x7fd89c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient(hunting): picked mon.noname-g con 0x7fd89c0a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient(hunting): picked mon.noname-d con 0x7fd89c13f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c0a3b40 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c0a8130 con 0x7fd89c13f5c0 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c06ede0 con 0x7fd89c0a67c0 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a466e640 10 monclient(hunting): authenticate will time out at 1818.559118s 2024-08-19T10:09:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a240a640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd89c0a67c0 0x7fd89c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48960/0 (socket says 172.21.15.1:48960) 2024-08-19T10:09:22.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.483+0000 7fd8a240a640 1 -- 172.21.15.1:0/1675480149 learned_addr learned my addr 172.21.15.1:0/1675480149 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:22.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a2c0b640 1 -- 172.21.15.1:0/1675480149 >> v1:172.21.15.1:6792/0 conn(0x7fd89c13f5c0 legacy=0x7fd89c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a240a640 1 -- 172.21.15.1:0/1675480149 >> v1:172.21.15.1:6793/0 conn(0x7fd89c0a67c0 legacy=0x7fd89c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1c09640 1 -- 172.21.15.1:0/1675480149 >> v1:172.21.15.82:6790/0 conn(0x7fd89c0a6ba0 legacy=0x7fd89c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630252839 0 0) 0x7fd89c0a8130 con 0x7fd89c13f5c0 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): my global_id is 204486 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd888003520 con 0x7fd89c13f5c0 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3582382418 0 0) 0x7fd89c06ede0 con 0x7fd89c0a67c0 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): my global_id is 204614 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c0a8130 con 0x7fd89c0a67c0 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183535531 0 0) 0x7fd89c0a3b40 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient(hunting): my global_id is 214356 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c06ede0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3201894800 0 0) 0x7fd888003520 con 0x7fd89c13f5c0 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 >> v1:172.21.15.1:6793/0 conn(0x7fd89c0a67c0 legacy=0x7fd89c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 >> v1:172.21.15.82:6790/0 conn(0x7fd89c0a6ba0 legacy=0x7fd89c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: _finish_hunting 0 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 monclient: found mon.noname-d 2024-08-19T10:09:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c0a92a0 con 0x7fd89c13f5c0 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: _finish_auth 0 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: _check_auth_tickets 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd898001eb0 con 0x7fd89c13f5c0 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a466e640 5 monclient: authenticate success, global_id 204486 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a466e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 10 monclient: dump: 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a466e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.485+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd898003990 con 0x7fd89c13f5c0 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a1408640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a1408640 1 -- 172.21.15.1:0/1675480149 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd898003f00 con 0x7fd89c13f5c0 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a1408640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 4 set_mon_vals no callback set 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a1408640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a1408640 10 monclient: dump: 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.495 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 10 monclient: shutdown 2024-08-19T10:09:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 1 -- 172.21.15.1:0/1675480149 >> v1:172.21.15.1:6792/0 conn(0x7fd89c13f5c0 legacy=0x7fd89c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 1 -- 172.21.15.1:0/1675480149 shutdown_connections 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 1 -- 172.21.15.1:0/1675480149 >> 172.21.15.1:0/1675480149 conn(0x7fd89c0ac4b0 msgr2=0x7fd89c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 1 -- 172.21.15.1:0/1675480149 shutdown_connections 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.486+0000 7fd8a466e640 1 -- 172.21.15.1:0/1675480149 wait complete. 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 10 monclient: build_initial_monmap 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 10 monclient: monmap: 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 1 librados: starting msgr at 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 1 librados: starting objecter 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 1 Processor -- start 2024-08-19T10:09:22.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 1 -- start start 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 1 librados: setting wanted keys 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 1 librados: calling monclient init 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 10 monclient: init 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 10 monclient: _add_conns ranks=[1,0,4,3,6,8,7,2,5] 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.487+0000 7fd8a466e640 10 monclient(hunting): picked mon.noname-f con 0x7fd89c0a67c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 10 monclient(hunting): picked mon.noname-a con 0x7fd89c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 10 monclient(hunting): picked mon.noname-c con 0x7fd89c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c13add0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c142cb0 con 0x7fd89c0a67c0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c13bc40 con 0x7fd89c13f5c0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a240a640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7fd89c0a67c0 legacy=0x7fd89c1ce200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:22.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a466e640 10 monclient(hunting): authenticate will time out at 1818.564079s 2024-08-19T10:09:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a1c09640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd89c0a6ba0 0x7fd89c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51504/0 (socket says 172.21.15.1:51504) 2024-08-19T10:09:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.488+0000 7fd8a1c09640 1 -- 172.21.15.1:0/2133864509 learned_addr learned my addr 172.21.15.1:0/2133864509 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd8a1c09640 1 -- 172.21.15.1:0/2133864509 >> v1:172.21.15.1:6789/0 conn(0x7fd89c0a6ba0 legacy=0x7fd89c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 client.?.objecter ms_handle_connect 0x7fd89c0a6ba0 2024-08-19T10:09:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 client.?.objecter ms_handle_connect 0x7fd89c13f5c0 2024-08-19T10:09:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd8a2c0b640 1 -- 172.21.15.1:0/2133864509 >> v1:172.21.15.1:6791/0 conn(0x7fd89c13f5c0 legacy=0x7fd89c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539899354 0 0) 0x7fd89c13add0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): my global_id is 204588 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd880004eb0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2977214552 0 0) 0x7fd89c13bc40 con 0x7fd89c13f5c0 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): my global_id is 194782 2024-08-19T10:09:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c13add0 con 0x7fd89c13f5c0 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3709534315 0 0) 0x7fd880004eb0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd89c13bc40 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd88c003610 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.505 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 733550680 0 0) 0x7fd89c13add0 con 0x7fd89c13f5c0 2024-08-19T10:09:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd880004eb0 con 0x7fd89c13f5c0 2024-08-19T10:09:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd898003e80 con 0x7fd89c13f5c0 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2068368669 0 0) 0x7fd89c13bc40 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 >> v1:172.21.15.1:6791/0 conn(0x7fd89c13f5c0 legacy=0x7fd89c1d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 >> v1:172.21.15.82:6789/0 conn(0x7fd89c0a67c0 legacy=0x7fd89c1ce200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 monclient: found mon.a 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c1cf020 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd892ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd8a466e640 5 monclient: authenticate success, global_id 204588 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd8a466e640 10 monclient: _renew_subs 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd8a466e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.490+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c13b3a0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 10 client.204588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 10 monclient: _renew_subs 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd89c13e4b0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 1 librados: init done 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd88c003bd0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd88c004140 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a0c07640 4 set_mon_vals no callback set 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 10 monclient: dump: 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.515 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd88c0042e0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.491+0000 7fd8a466e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd884005080 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fd88c04e050 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd892ffd640 10 client.204588.objecter ms_dispatch 0x7fd89c000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd892ffd640 3 client.204588.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd892ffd640 3 client.204588.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd892ffd640 20 client.204588.objecter dump_active .. 0 homeless 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.492+0000 7fd8a240a640 10 monclient: get_auth_request con 0x7fd880041490 auth_method 0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.494+0000 7fd892ffd640 10 client.204588.objecter ms_handle_connect 0x7fd880041490 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.496+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fd88c0176b0 con 0x7fd89c0a6ba0 2024-08-19T10:09:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.496+0000 7fd892ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.496+0000 7fd892ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:22.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.677+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd884000b90 con 0x7fd880041490 2024-08-19T10:09:22.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.680+0000 7fd892ffd640 1 -- 172.21.15.1:0/2133864509 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114209 (unknown 2965378022 0 3985610741) 0x7fd884000b90 con 0x7fd880041490 2024-08-19T10:09:22.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T10:09:22.682 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 10 librados: watch_flush enter 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 10 librados: watch_flush exit 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 20 client.204588.objecter shutdown clearing up homeless session... 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 10 client.204588.objecter successfully canceled tick 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd880041490 legacy=0x7fd880043950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 10 monclient: shutdown 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 >> v1:172.21.15.1:6789/0 conn(0x7fd89c0a6ba0 legacy=0x7fd89c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 shutdown_connections 2024-08-19T10:09:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 >> 172.21.15.1:0/2133864509 conn(0x7fd89c0ac4b0 msgr2=0x7fd89c0aa380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:22.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 shutdown_connections 2024-08-19T10:09:22.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 -- 172.21.15.1:0/2133864509 wait complete. 2024-08-19T10:09:22.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.682+0000 7fd8a466e640 1 librados: shutdown 2024-08-19T10:09:22.694 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1281,"stamp":"2024-08-19T10:09:21.060770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":442812,"kb_used_data":39084,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565788228,"statfs":{"total":579820584960,"available":579367145472,"internally_reserved":0,"allocated":40022016,"data_stored":17962702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":413260450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001024"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1492,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478370+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T10:09:18.478370+0000","last_peered":"2024-08-19T10:09:18.478370+0000","last_clean":"2024-08-19T10:09:18.478370+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T10:09:18.478370+0000","last_undegraded":"2024-08-19T10:09:18.478370+0000","last_fullsized":"2024-08-19T10:09:18.478370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1398,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219416+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T10:09:17.219416+0000","last_peered":"2024-08-19T10:09:17.219416+0000","last_clean":"2024-08-19T10:09:17.219416+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T10:09:17.219416+0000","last_undegraded":"2024-08-19T10:09:17.219416+0000","last_fullsized":"2024-08-19T10:09:17.219416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1401,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478445+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T10:09:18.478445+0000","last_peered":"2024-08-19T10:09:18.478445+0000","last_clean":"2024-08-19T10:09:18.478445+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T10:09:18.478445+0000","last_undegraded":"2024-08-19T10:09:18.478445+0000","last_fullsized":"2024-08-19T10:09:18.478445+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1397,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216502+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T10:09:17.216502+0000","last_peered":"2024-08-19T10:09:17.216502+0000","last_clean":"2024-08-19T10:09:17.216502+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T10:09:17.216502+0000","last_undegraded":"2024-08-19T10:09:17.216502+0000","last_fullsized":"2024-08-19T10:09:17.216502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1399,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216190+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T10:09:17.216190+0000","last_peered":"2024-08-19T10:09:17.216190+0000","last_clean":"2024-08-19T10:09:17.216190+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T10:09:17.216190+0000","last_undegraded":"2024-08-19T10:09:17.216190+0000","last_fullsized":"2024-08-19T10:09:17.216190+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1391,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217571+0000","last_change":"2024-08-19T09:50:17.769859+0000","last_active":"2024-08-19T10:09:16.217571+0000","last_peered":"2024-08-19T10:09:16.217571+0000","last_clean":"2024-08-19T10:09:16.217571+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T10:09:16.217571+0000","last_undegraded":"2024-08-19T10:09:16.217571+0000","last_fullsized":"2024-08-19T10:09:16.217571+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:29:47.820898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1401,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478316+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T10:09:18.478316+0000","last_peered":"2024-08-19T10:09:18.478316+0000","last_clean":"2024-08-19T10:09:18.478316+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T10:09:18.478316+0000","last_undegraded":"2024-08-19T10:09:18.478316+0000","last_fullsized":"2024-08-19T10:09:18.478316+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1390,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217636+0000","last_change":"2024-08-19T09:50:17.769969+0000","last_active":"2024-08-19T10:09:16.217636+0000","last_peered":"2024-08-19T10:09:16.217636+0000","last_clean":"2024-08-19T10:09:16.217636+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T10:09:16.217636+0000","last_undegraded":"2024-08-19T10:09:16.217636+0000","last_fullsized":"2024-08-19T10:09:16.217636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:15.287274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1400,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478414+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T10:09:18.478414+0000","last_peered":"2024-08-19T10:09:18.478414+0000","last_clean":"2024-08-19T10:09:18.478414+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T10:09:18.478414+0000","last_undegraded":"2024-08-19T10:09:18.478414+0000","last_fullsized":"2024-08-19T10:09:18.478414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1","version":"23'7","reported_seq":1396,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478569+0000","last_change":"2024-08-19T09:51:22.586587+0000","last_active":"2024-08-19T10:09:18.478569+0000","last_peered":"2024-08-19T10:09:18.478569+0000","last_clean":"2024-08-19T10:09:18.478569+0000","last_became_active":"2024-08-19T09:51:22.585789+0000","last_became_peered":"2024-08-19T09:51:22.585789+0000","last_unstale":"2024-08-19T10:09:18.478569+0000","last_undegraded":"2024-08-19T10:09:18.478569+0000","last_fullsized":"2024-08-19T10:09:18.478569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:25:11.479863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217047+0000","last_change":"2024-08-19T09:51:22.586723+0000","last_active":"2024-08-19T10:09:17.217047+0000","last_peered":"2024-08-19T10:09:17.217047+0000","last_clean":"2024-08-19T10:09:17.217047+0000","last_became_active":"2024-08-19T09:51:22.583401+0000","last_became_peered":"2024-08-19T09:51:22.583401+0000","last_unstale":"2024-08-19T10:09:17.217047+0000","last_undegraded":"2024-08-19T10:09:17.217047+0000","last_fullsized":"2024-08-19T10:09:17.217047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:56:50.500489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217571+0000","last_change":"2024-08-19T09:51:22.588184+0000","last_active":"2024-08-19T10:09:17.217571+0000","last_peered":"2024-08-19T10:09:17.217571+0000","last_clean":"2024-08-19T10:09:17.217571+0000","last_became_active":"2024-08-19T09:51:22.587936+0000","last_became_peered":"2024-08-19T09:51:22.587936+0000","last_unstale":"2024-08-19T10:09:17.217571+0000","last_undegraded":"2024-08-19T10:09:17.217571+0000","last_fullsized":"2024-08-19T10:09:17.217571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:22:07.623601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217092+0000","last_change":"2024-08-19T09:51:22.640927+0000","last_active":"2024-08-19T10:09:17.217092+0000","last_peered":"2024-08-19T10:09:17.217092+0000","last_clean":"2024-08-19T10:09:17.217092+0000","last_became_active":"2024-08-19T09:51:22.640604+0000","last_became_peered":"2024-08-19T09:51:22.640604+0000","last_unstale":"2024-08-19T10:09:17.217092+0000","last_undegraded":"2024-08-19T10:09:17.217092+0000","last_fullsized":"2024-08-19T10:09:17.217092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:38:41.853584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1383,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478207+0000","last_change":"2024-08-19T09:51:22.581514+0000","last_active":"2024-08-19T10:09:18.478207+0000","last_peered":"2024-08-19T10:09:18.478207+0000","last_clean":"2024-08-19T10:09:18.478207+0000","last_became_active":"2024-08-19T09:51:22.581246+0000","last_became_peered":"2024-08-19T09:51:22.581246+0000","last_unstale":"2024-08-19T10:09:18.478207+0000","last_undegraded":"2024-08-19T10:09:18.478207+0000","last_fullsized":"2024-08-19T10:09:18.478207+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:12:50.239096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.220013+0000","last_change":"2024-08-19T09:51:22.620559+0000","last_active":"2024-08-19T10:09:17.220013+0000","last_peered":"2024-08-19T10:09:17.220013+0000","last_clean":"2024-08-19T10:09:17.220013+0000","last_became_active":"2024-08-19T09:51:22.620042+0000","last_became_peered":"2024-08-19T09:51:22.620042+0000","last_unstale":"2024-08-19T10:09:17.220013+0000","last_undegraded":"2024-08-19T10:09:17.220013+0000","last_fullsized":"2024-08-19T10:09:17.220013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:27:35.980026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218905+0000","last_change":"2024-08-19T09:51:22.606649+0000","last_active":"2024-08-19T10:09:17.218905+0000","last_peered":"2024-08-19T10:09:17.218905+0000","last_clean":"2024-08-19T10:09:17.218905+0000","last_became_active":"2024-08-19T09:51:22.606054+0000","last_became_peered":"2024-08-19T09:51:22.606054+0000","last_unstale":"2024-08-19T10:09:17.218905+0000","last_undegraded":"2024-08-19T10:09:17.218905+0000","last_fullsized":"2024-08-19T10:09:17.218905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:23:52.344010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217755+0000","last_change":"2024-08-19T09:51:22.653723+0000","last_active":"2024-08-19T10:09:17.217755+0000","last_peered":"2024-08-19T10:09:17.217755+0000","last_clean":"2024-08-19T10:09:17.217755+0000","last_became_active":"2024-08-19T09:51:22.653455+0000","last_became_peered":"2024-08-19T09:51:22.653455+0000","last_unstale":"2024-08-19T10:09:17.217755+0000","last_undegraded":"2024-08-19T10:09:17.217755+0000","last_fullsized":"2024-08-19T10:09:17.217755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:25:13.028210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.679991+0000","last_change":"2024-08-19T09:51:22.560997+0000","last_active":"2024-08-19T10:09:16.679991+0000","last_peered":"2024-08-19T10:09:16.679991+0000","last_clean":"2024-08-19T10:09:16.679991+0000","last_became_active":"2024-08-19T09:51:22.553419+0000","last_became_peered":"2024-08-19T09:51:22.553419+0000","last_unstale":"2024-08-19T10:09:16.679991+0000","last_undegraded":"2024-08-19T10:09:16.679991+0000","last_fullsized":"2024-08-19T10:09:16.679991+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:25.353142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1379,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.216939+0000","last_change":"2024-08-19T09:51:22.554649+0000","last_active":"2024-08-19T10:09:17.216939+0000","last_peered":"2024-08-19T10:09:17.216939+0000","last_clean":"2024-08-19T10:09:17.216939+0000","last_became_active":"2024-08-19T09:51:22.554397+0000","last_became_peered":"2024-08-19T09:51:22.554397+0000","last_unstale":"2024-08-19T10:09:17.216939+0000","last_undegraded":"2024-08-19T10:09:17.216939+0000","last_fullsized":"2024-08-19T10:09:17.216939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:36:10.709773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1378,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217364+0000","last_change":"2024-08-19T09:55:22.703762+0000","last_active":"2024-08-19T10:09:16.217364+0000","last_peered":"2024-08-19T10:09:16.217364+0000","last_clean":"2024-08-19T10:09:16.217364+0000","last_became_active":"2024-08-19T09:55:22.703438+0000","last_became_peered":"2024-08-19T09:55:22.703438+0000","last_unstale":"2024-08-19T10:09:16.217364+0000","last_undegraded":"2024-08-19T10:09:16.217364+0000","last_fullsized":"2024-08-19T10:09:16.217364+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:29:17.373388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217274+0000","last_change":"2024-08-19T09:51:22.529738+0000","last_active":"2024-08-19T10:09:16.217274+0000","last_peered":"2024-08-19T10:09:16.217274+0000","last_clean":"2024-08-19T10:09:16.217274+0000","last_became_active":"2024-08-19T09:51:22.528219+0000","last_became_peered":"2024-08-19T09:51:22.528219+0000","last_unstale":"2024-08-19T10:09:16.217274+0000","last_undegraded":"2024-08-19T10:09:16.217274+0000","last_fullsized":"2024-08-19T10:09:16.217274+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:44.262012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219294+0000","last_change":"2024-08-19T09:51:22.614370+0000","last_active":"2024-08-19T10:09:17.219294+0000","last_peered":"2024-08-19T10:09:17.219294+0000","last_clean":"2024-08-19T10:09:17.219294+0000","last_became_active":"2024-08-19T09:51:22.614136+0000","last_became_peered":"2024-08-19T09:51:22.614136+0000","last_unstale":"2024-08-19T10:09:17.219294+0000","last_undegraded":"2024-08-19T10:09:17.219294+0000","last_fullsized":"2024-08-19T10:09:17.219294+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:14:54.869367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218815+0000","last_change":"2024-08-19T09:51:22.587447+0000","last_active":"2024-08-19T10:09:17.218815+0000","last_peered":"2024-08-19T10:09:17.218815+0000","last_clean":"2024-08-19T10:09:17.218815+0000","last_became_active":"2024-08-19T09:51:22.587149+0000","last_became_peered":"2024-08-19T09:51:22.587149+0000","last_unstale":"2024-08-19T10:09:17.218815+0000","last_undegraded":"2024-08-19T10:09:17.218815+0000","last_fullsized":"2024-08-19T10:09:17.218815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:08:36.757587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479995+0000","last_change":"2024-08-19T09:51:22.638208+0000","last_active":"2024-08-19T10:09:18.479995+0000","last_peered":"2024-08-19T10:09:18.479995+0000","last_clean":"2024-08-19T10:09:18.479995+0000","last_became_active":"2024-08-19T09:51:22.634477+0000","last_became_peered":"2024-08-19T09:51:22.634477+0000","last_unstale":"2024-08-19T10:09:18.479995+0000","last_undegraded":"2024-08-19T10:09:18.479995+0000","last_fullsized":"2024-08-19T10:09:18.479995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:01:43.314792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.217657+0000","last_change":"2024-08-19T09:51:22.594096+0000","last_active":"2024-08-19T10:09:17.217657+0000","last_peered":"2024-08-19T10:09:17.217657+0000","last_clean":"2024-08-19T10:09:17.217657+0000","last_became_active":"2024-08-19T09:51:22.590688+0000","last_became_peered":"2024-08-19T09:51:22.590688+0000","last_unstale":"2024-08-19T10:09:17.217657+0000","last_undegraded":"2024-08-19T10:09:17.217657+0000","last_fullsized":"2024-08-19T10:09:17.217657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:51.718926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217017+0000","last_change":"2024-08-19T09:51:22.537238+0000","last_active":"2024-08-19T10:09:16.217017+0000","last_peered":"2024-08-19T10:09:16.217017+0000","last_clean":"2024-08-19T10:09:16.217017+0000","last_became_active":"2024-08-19T09:51:22.536584+0000","last_became_peered":"2024-08-19T09:51:22.536584+0000","last_unstale":"2024-08-19T10:09:16.217017+0000","last_undegraded":"2024-08-19T10:09:16.217017+0000","last_fullsized":"2024-08-19T10:09:16.217017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:42:52.494448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1369,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.679764+0000","last_change":"2024-08-19T09:51:22.556447+0000","last_active":"2024-08-19T10:09:16.679764+0000","last_peered":"2024-08-19T10:09:16.679764+0000","last_clean":"2024-08-19T10:09:16.679764+0000","last_became_active":"2024-08-19T09:51:22.556182+0000","last_became_peered":"2024-08-19T09:51:22.556182+0000","last_unstale":"2024-08-19T10:09:16.679764+0000","last_undegraded":"2024-08-19T10:09:16.679764+0000","last_fullsized":"2024-08-19T10:09:16.679764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:04:06.155397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217049+0000","last_change":"2024-08-19T09:51:22.672202+0000","last_active":"2024-08-19T10:09:16.217049+0000","last_peered":"2024-08-19T10:09:16.217049+0000","last_clean":"2024-08-19T10:09:16.217049+0000","last_became_active":"2024-08-19T09:51:22.671918+0000","last_became_peered":"2024-08-19T09:51:22.671918+0000","last_unstale":"2024-08-19T10:09:16.217049+0000","last_undegraded":"2024-08-19T10:09:16.217049+0000","last_fullsized":"2024-08-19T10:09:16.217049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:56:23.219267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217181+0000","last_change":"2024-08-19T09:51:22.605834+0000","last_active":"2024-08-19T10:09:16.217181+0000","last_peered":"2024-08-19T10:09:16.217181+0000","last_clean":"2024-08-19T10:09:16.217181+0000","last_became_active":"2024-08-19T09:51:22.605586+0000","last_became_peered":"2024-08-19T09:51:22.605586+0000","last_unstale":"2024-08-19T10:09:16.217181+0000","last_undegraded":"2024-08-19T10:09:16.217181+0000","last_fullsized":"2024-08-19T10:09:16.217181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:53:29.756052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217163+0000","last_change":"2024-08-19T09:51:22.551390+0000","last_active":"2024-08-19T10:09:16.217163+0000","last_peered":"2024-08-19T10:09:16.217163+0000","last_clean":"2024-08-19T10:09:16.217163+0000","last_became_active":"2024-08-19T09:51:22.551119+0000","last_became_peered":"2024-08-19T09:51:22.551119+0000","last_unstale":"2024-08-19T10:09:16.217163+0000","last_undegraded":"2024-08-19T10:09:16.217163+0000","last_fullsized":"2024-08-19T10:09:16.217163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:43:26.828213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218719+0000","last_change":"2024-08-19T09:51:22.631476+0000","last_active":"2024-08-19T10:09:17.218719+0000","last_peered":"2024-08-19T10:09:17.218719+0000","last_clean":"2024-08-19T10:09:17.218719+0000","last_became_active":"2024-08-19T09:51:22.630612+0000","last_became_peered":"2024-08-19T09:51:22.630612+0000","last_unstale":"2024-08-19T10:09:17.218719+0000","last_undegraded":"2024-08-19T10:09:17.218719+0000","last_fullsized":"2024-08-19T10:09:17.218719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:15.371670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1369,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.680505+0000","last_change":"2024-08-19T09:51:22.490285+0000","last_active":"2024-08-19T10:09:16.680505+0000","last_peered":"2024-08-19T10:09:16.680505+0000","last_clean":"2024-08-19T10:09:16.680505+0000","last_became_active":"2024-08-19T09:51:22.490018+0000","last_became_peered":"2024-08-19T09:51:22.490018+0000","last_unstale":"2024-08-19T10:09:16.680505+0000","last_undegraded":"2024-08-19T10:09:16.680505+0000","last_fullsized":"2024-08-19T10:09:16.680505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:57:54.192511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479361+0000","last_change":"2024-08-19T09:51:22.646201+0000","last_active":"2024-08-19T10:09:18.479361+0000","last_peered":"2024-08-19T10:09:18.479361+0000","last_clean":"2024-08-19T10:09:18.479361+0000","last_became_active":"2024-08-19T09:51:22.644785+0000","last_became_peered":"2024-08-19T09:51:22.644785+0000","last_unstale":"2024-08-19T10:09:18.479361+0000","last_undegraded":"2024-08-19T10:09:18.479361+0000","last_fullsized":"2024-08-19T10:09:18.479361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:30:51.586745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478788+0000","last_change":"2024-08-19T09:51:22.633554+0000","last_active":"2024-08-19T10:09:18.478788+0000","last_peered":"2024-08-19T10:09:18.478788+0000","last_clean":"2024-08-19T10:09:18.478788+0000","last_became_active":"2024-08-19T09:51:22.618248+0000","last_became_peered":"2024-08-19T09:51:22.618248+0000","last_unstale":"2024-08-19T10:09:18.478788+0000","last_undegraded":"2024-08-19T10:09:18.478788+0000","last_fullsized":"2024-08-19T10:09:18.478788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:04:45.586830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217017+0000","last_change":"2024-08-19T09:51:22.559064+0000","last_active":"2024-08-19T10:09:16.217017+0000","last_peered":"2024-08-19T10:09:16.217017+0000","last_clean":"2024-08-19T10:09:16.217017+0000","last_became_active":"2024-08-19T09:51:22.558815+0000","last_became_peered":"2024-08-19T09:51:22.558815+0000","last_unstale":"2024-08-19T10:09:16.217017+0000","last_undegraded":"2024-08-19T10:09:16.217017+0000","last_fullsized":"2024-08-19T10:09:16.217017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:06:02.138369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.217465+0000","last_change":"2024-08-19T09:51:22.582602+0000","last_active":"2024-08-19T10:09:16.217465+0000","last_peered":"2024-08-19T10:09:16.217465+0000","last_clean":"2024-08-19T10:09:16.217465+0000","last_became_active":"2024-08-19T09:51:22.581840+0000","last_became_peered":"2024-08-19T09:51:22.581840+0000","last_unstale":"2024-08-19T10:09:16.217465+0000","last_undegraded":"2024-08-19T10:09:16.217465+0000","last_fullsized":"2024-08-19T10:09:16.217465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:24:46.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T10:09:16.680151+0000","last_change":"2024-08-19T09:55:22.701986+0000","last_active":"2024-08-19T10:09:16.680151+0000","last_peered":"2024-08-19T10:09:16.680151+0000","last_clean":"2024-08-19T10:09:16.680151+0000","last_became_active":"2024-08-19T09:55:22.701660+0000","last_became_peered":"2024-08-19T09:55:22.701660+0000","last_unstale":"2024-08-19T10:09:16.680151+0000","last_undegraded":"2024-08-19T10:09:16.680151+0000","last_fullsized":"2024-08-19T10:09:16.680151+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:56:19.752844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1372,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.478819+0000","last_change":"2024-08-19T09:51:22.565151+0000","last_active":"2024-08-19T10:09:18.478819+0000","last_peered":"2024-08-19T10:09:18.478819+0000","last_clean":"2024-08-19T10:09:18.478819+0000","last_became_active":"2024-08-19T09:51:22.564919+0000","last_became_peered":"2024-08-19T09:51:22.564919+0000","last_unstale":"2024-08-19T10:09:18.478819+0000","last_undegraded":"2024-08-19T10:09:18.478819+0000","last_fullsized":"2024-08-19T10:09:18.478819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:05.826534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.219907+0000","last_change":"2024-08-19T09:51:22.612829+0000","last_active":"2024-08-19T10:09:17.219907+0000","last_peered":"2024-08-19T10:09:17.219907+0000","last_clean":"2024-08-19T10:09:17.219907+0000","last_became_active":"2024-08-19T09:51:22.612552+0000","last_became_peered":"2024-08-19T09:51:22.612552+0000","last_unstale":"2024-08-19T10:09:17.219907+0000","last_undegraded":"2024-08-19T10:09:17.219907+0000","last_fullsized":"2024-08-19T10:09:17.219907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:58:05.134880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1378,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-08-19T10:09:17.218803+0000","last_change":"2024-08-19T09:51:22.609929+0000","last_active":"2024-08-19T10:09:17.218803+0000","last_peered":"2024-08-19T10:09:17.218803+0000","last_clean":"2024-08-19T10:09:17.218803+0000","last_became_active":"2024-08-19T09:51:22.609674+0000","last_became_peered":"2024-08-19T09:51:22.609674+0000","last_unstale":"2024-08-19T10:09:17.218803+0000","last_undegraded":"2024-08-19T10:09:17.218803+0000","last_fullsized":"2024-08-19T10:09:17.218803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:41:23.025542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1373,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-08-19T10:09:18.479455+0000","last_change":"2024-08-19T09:51:22.652416+0000","last_active":"2024-08-19T10:09:18.479455+0000","last_peered":"2024-08-19T10:09:18.479455+0000","last_clean":"2024-08-19T10:09:18.479455+0000","last_became_active":"2024-08-19T09:51:22.652083+0000","last_became_peered":"2024-08-19T09:51:22.652083+0000","last_unstale":"2024-08-19T10:09:18.479455+0000","last_undegraded":"2024-08-19T10:09:18.479455+0000","last_fullsized":"2024-08-19T10:09:18.479455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:47.783611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575086,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73720,"kb_used_data":6436,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298120,"statfs":{"total":96636764160,"available":96561274880,"internally_reserved":0,"allocated":6590464,"data_stored":2911767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74128,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297712,"statfs":{"total":96636764160,"available":96560857088,"internally_reserved":0,"allocated":7000064,"data_stored":3313724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69424,"kb_used_data":6292,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302416,"statfs":{"total":96636764160,"available":96565673984,"internally_reserved":0,"allocated":6443008,"data_stored":2760234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607785,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81992,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289848,"statfs":{"total":96636764160,"available":96552804352,"internally_reserved":0,"allocated":6537216,"data_stored":2864612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69960,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301880,"statfs":{"total":96636764160,"available":96565125120,"internally_reserved":0,"allocated":6995968,"data_stored":3323903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73588,"kb_used_data":6304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298252,"statfs":{"total":96636764160,"available":96561410048,"internally_reserved":0,"allocated":6455296,"data_stored":2788462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T10:09:22.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.793+0000 7fef10a0e640 10 monclient: get_monmap_and_config 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.793+0000 7fef10a0e640 10 monclient: build_initial_monmap 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.793+0000 7fef10a0e640 10 monclient: monmap: 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:22.794825+0000 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:22.794825+0000 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.795 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.794+0000 7fef10a0e640 1 Processor -- start 2024-08-19T10:09:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.794+0000 7fef10a0e640 1 -- start start 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.794+0000 7fef10a0e640 10 monclient: init 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient: _add_conns ranks=[0,5,4,1,2,7,3,8,6] 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fef0c0a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient(hunting): picked mon.noname-h con 0x7fef0c0a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fef0c13f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0c0a3b40 con 0x7fef0c0a67c0 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0c0a8130 con 0x7fef0c13f5c0 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0c06ede0 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef10a0e640 10 monclient(hunting): authenticate will time out at 1818.871172s 2024-08-19T10:09:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef0ad9d640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fef0c13f5c0 0x7fef0c1419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48406/0 (socket says 172.21.15.1:48406) 2024-08-19T10:09:22.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef0a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fef0c0a67c0 0x7fef0c0a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51508/0 (socket says 172.21.15.1:51508) 2024-08-19T10:09:22.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.795+0000 7fef0ad9d640 1 -- 172.21.15.1:0/1672936398 learned_addr learned my addr 172.21.15.1:0/1672936398 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0a59c640 1 -- 172.21.15.1:0/1672936398 >> v1:172.21.15.1:6789/0 conn(0x7fef0c0a67c0 legacy=0x7fef0c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0ad9d640 1 -- 172.21.15.1:0/1672936398 >> v1:172.21.15.1:6791/0 conn(0x7fef0c13f5c0 legacy=0x7fef0c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef09d9b640 1 -- 172.21.15.1:0/1672936398 >> v1:172.21.15.82:6791/0 conn(0x7fef0c0a6ba0 legacy=0x7fef0c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704155898 0 0) 0x7fef0c0a3b40 con 0x7fef0c0a67c0 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): my global_id is 204597 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feef8003520 con 0x7fef0c0a67c0 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962375533 0 0) 0x7fef0c0a8130 con 0x7fef0c13f5c0 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): my global_id is 194791 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef0c0a3b40 con 0x7fef0c13f5c0 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2049597971 0 0) 0x7fef0c06ede0 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient(hunting): my global_id is 214340 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef0c0a8130 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3333097552 0 0) 0x7fef0c0a3b40 con 0x7fef0c13f5c0 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 >> v1:172.21.15.82:6791/0 conn(0x7fef0c0a6ba0 legacy=0x7fef0c0a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 >> v1:172.21.15.1:6789/0 conn(0x7fef0c0a67c0 legacy=0x7fef0c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient: _finish_hunting 0 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 monclient: found mon.noname-c 2024-08-19T10:09:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c0a92a0 con 0x7fef0c13f5c0 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient: _finish_auth 0 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient: _check_auth_tickets 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef10a0e640 5 monclient: authenticate success, global_id 194791 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef04001ac0 con 0x7fef0c13f5c0 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef10a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.797+0000 7fef0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 10 monclient: dump: 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.806 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef040048a0 con 0x7fef0c13f5c0 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 1 -- 172.21.15.1:0/1672936398 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef04004e10 con 0x7fef0c13f5c0 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 4 set_mon_vals no callback set 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef0959a640 10 monclient: dump: 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.808 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 10 monclient: shutdown 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 1 -- 172.21.15.1:0/1672936398 >> v1:172.21.15.1:6791/0 conn(0x7fef0c13f5c0 legacy=0x7fef0c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 1 -- 172.21.15.1:0/1672936398 shutdown_connections 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 1 -- 172.21.15.1:0/1672936398 >> 172.21.15.1:0/1672936398 conn(0x7fef0c0ac4b0 msgr2=0x7fef0c0a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 1 -- 172.21.15.1:0/1672936398 shutdown_connections 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.798+0000 7fef10a0e640 1 -- 172.21.15.1:0/1672936398 wait complete. 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 10 monclient: build_initial_monmap 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 10 monclient: monmap: 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:22.810 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 1 librados: starting msgr at 2024-08-19T10:09:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 1 librados: starting objecter 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 1 Processor -- start 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 1 -- start start 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 1 librados: setting wanted keys 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 1 librados: calling monclient init 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.799+0000 7fef10a0e640 10 monclient: init 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient: _add_conns ranks=[8,0,7,3,6,4,5,2,1] 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient(hunting): picked mon.noname-e con 0x7fef0c0a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fef0c0a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient(hunting): picked mon.noname-i con 0x7fef0c13f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0c0a2890 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0c139be0 con 0x7fef0c13f5c0 2024-08-19T10:09:22.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0c0a1630 con 0x7fef0c0a67c0 2024-08-19T10:09:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef10a0e640 10 monclient(hunting): authenticate will time out at 1818.876467s 2024-08-19T10:09:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef09d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fef0c0a6ba0 0x7fef0c0a05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51520/0 (socket says 172.21.15.1:51520) 2024-08-19T10:09:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef09d9b640 1 -- 172.21.15.1:0/1574859466 learned_addr learned my addr 172.21.15.1:0/1574859466 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.800+0000 7fef0a59c640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fef0c0a67c0 0x7fef0c09fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48968/0 (socket says 172.21.15.1:48968) 2024-08-19T10:09:22.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef0a59c640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.1:6793/0 conn(0x7fef0c0a67c0 legacy=0x7fef0c09fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef02ffd640 10 client.?.objecter ms_handle_connect 0x7fef0c0a67c0 2024-08-19T10:09:22.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef09d9b640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.1:6789/0 conn(0x7fef0c0a6ba0 legacy=0x7fef0c0a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef02ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef02ffd640 10 client.?.objecter ms_handle_connect 0x7fef0c0a6ba0 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef02ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef0ad9d640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.82:6792/0 conn(0x7fef0c13f5c0 legacy=0x7fef0c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef02ffd640 10 client.?.objecter ms_handle_connect 0x7fef0c13f5c0 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.802+0000 7fef02ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2576848981 0 0) 0x7fef0c0a1630 con 0x7fef0c0a67c0 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): my global_id is 204623 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feeec004eb0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68403060 0 0) 0x7fef0c0a2890 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): my global_id is 204606 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef0c0a1630 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368665054 0 0) 0x7fef0c139be0 con 0x7fef0c13f5c0 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): my global_id is 204568 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef0c0a2890 con 0x7fef0c13f5c0 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2728705819 0 0) 0x7feeec004eb0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef0c139be0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1635450296 0 0) 0x7fef0c0a1630 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feeec004eb0 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feefc004b60 con 0x7fef0c0a67c0 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feef4003610 con 0x7fef0c0a6ba0 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2130913741 0 0) 0x7fef0c0a2890 con 0x7fef0c13f5c0 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef0c0a1630 con 0x7fef0c13f5c0 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fef04001d10 con 0x7fef0c13f5c0 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3675539581 0 0) 0x7fef0c139be0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.82:6792/0 conn(0x7fef0c13f5c0 legacy=0x7fef0c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.1:6789/0 conn(0x7fef0c0a6ba0 legacy=0x7fef0c0a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 monclient: found mon.e 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c13a8d0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 5 monclient: authenticate success, global_id 204623 2024-08-19T10:09:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 10 monclient: _renew_subs 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef0c13a030 con 0x7fef0c0a67c0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 10 client.204623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 10 monclient: _renew_subs 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fef0c0a0cf0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef10a0e640 1 librados: init done 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feefc004fb0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7feefc0054e0 con 0x7fef0c0a67c0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef08d99640 4 set_mon_vals no callback set 2024-08-19T10:09:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.803+0000 7fef02ffd640 10 monclient: dump: 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef10a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7feefc005680 con 0x7fef0c0a67c0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef10a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef10a0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef0005080 con 0x7fef0c0a67c0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7feefc04e350 con 0x7fef0c0a67c0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef02ffd640 10 client.204623.objecter ms_dispatch 0x7fef0c000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef02ffd640 3 client.204623.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef02ffd640 3 client.204623.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef02ffd640 20 client.204623.objecter dump_active .. 0 homeless 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.804+0000 7fef09d9b640 10 monclient: get_auth_request con 0x7feeec041a20 auth_method 0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.807+0000 7fef02ffd640 10 client.204623.objecter ms_handle_connect 0x7feeec041a20 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.809+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7feefc017970 con 0x7fef0c0a67c0 2024-08-19T10:09:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.809+0000 7fef02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:22.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.809+0000 7fef02ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef10a0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T10:09:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef10a0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T10:09:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef10a0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feef0005370 con 0x7fef0c0a67c0 2024-08-19T10:09:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef02ffd640 1 -- 172.21.15.1:0/1574859466 <== mon.8 v1:172.21.15.1:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v711) ==== 74+0+14074 (unknown 3425890110 0 326276410) 0x7feefc01b870 con 0x7fef0c0a67c0 2024-08-19T10:09:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T10:09:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.992+0000 7fef02ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:22.994 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T10:09:22.995 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":711,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","created":"2024-08-19T09:49:54.777161+0000","modified":"2024-08-19T10:09:20.623915+0000","last_up_change":"2024-08-19T09:50:07.419178+0000","last_in_change":"2024-08-19T09:49:58.291381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":106,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T09:50:07.171793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T09:50:13.629390+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-08-19T09:51:20.253923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"70148d02-f1a0-47ce-987a-5539fce9ac64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":705,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":3862255507}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":3862255507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":3862255507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":3862255507}]},"public_addr":"172.21.15.1:6804/3862255507","cluster_addr":"172.21.15.1:6805/3862255507","heartbeat_back_addr":"172.21.15.1:6807/3862255507","heartbeat_front_addr":"172.21.15.1:6806/3862255507","state":["exists","up"]},{"osd":1,"uuid":"066ec263-41d5-4d6c-a306-7cb11e1834ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":705,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1668590231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1668590231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1668590231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1668590231}]},"public_addr":"172.21.15.1:6800/1668590231","cluster_addr":"172.21.15.1:6801/1668590231","heartbeat_back_addr":"172.21.15.1:6803/1668590231","heartbeat_front_addr":"172.21.15.1:6802/1668590231","state":["exists","up"]},{"osd":2,"uuid":"500b3e1a-274e-4e36-b96c-ff8299d89a03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":705,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":207870648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":207870648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":207870648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":207870648}]},"public_addr":"172.21.15.1:6808/207870648","cluster_addr":"172.21.15.1:6809/207870648","heartbeat_back_addr":"172.21.15.1:6811/207870648","heartbeat_front_addr":"172.21.15.1:6810/207870648","state":["exists","up"]},{"osd":3,"uuid":"cd3f5bbd-3c03-43be-b6da-7d6f2dc8f799","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":705,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6804","nonce":394826682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6805","nonce":394826682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6807","nonce":394826682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6806","nonce":394826682}]},"public_addr":"172.21.15.82:6804/394826682","cluster_addr":"172.21.15.82:6805/394826682","heartbeat_back_addr":"172.21.15.82:6807/394826682","heartbeat_front_addr":"172.21.15.82:6806/394826682","state":["exists","up"]},{"osd":4,"uuid":"411cb062-e88f-4611-9cb9-bd03b212b768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":705,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6800","nonce":2865329823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6801","nonce":2865329823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6803","nonce":2865329823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6802","nonce":2865329823}]},"public_addr":"172.21.15.82:6800/2865329823","cluster_addr":"172.21.15.82:6801/2865329823","heartbeat_back_addr":"172.21.15.82:6803/2865329823","heartbeat_front_addr":"172.21.15.82:6802/2865329823","state":["exists","up"]},{"osd":5,"uuid":"fce7c24e-33f7-45d2-8fc9-8ead3f6daa0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":705,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6808","nonce":1813220032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6809","nonce":1813220032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6811","nonce":1813220032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6810","nonce":1813220032}]},"public_addr":"172.21.15.82:6808/1813220032","cluster_addr":"172.21.15.82:6809/1813220032","heartbeat_back_addr":"172.21.15.82:6811/1813220032","heartbeat_front_addr":"172.21.15.82:6810/1813220032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.564842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.315230+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.809229+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:05.004322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.803207+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T09:50:04.616501+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"33.b","mappings":[{"from":2,"to":0}]},{"pgid":"33.1a","mappings":[{"from":2,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":1,"length":4}]},{"pool":12,"snaps":[{"begin":4,"length":3}]},{"pool":31,"snaps":[{"begin":1,"length":1}]},{"pool":65,"snaps":[{"begin":1,"length":1}]},{"pool":89,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":102,"snaps":[{"begin":1,"length":1}]},{"pool":120,"snaps":[{"begin":1,"length":2}]},{"pool":138,"snaps":[{"begin":3,"length":1}]},{"pool":143,"snaps":[{"begin":1,"length":4}]},{"pool":149,"snaps":[{"begin":2,"length":1}]},{"pool":151,"snaps":[{"begin":2,"length":2}]},{"pool":163,"snaps":[{"begin":2,"length":3}]},{"pool":169,"snaps":[{"begin":3,"length":4}]},{"pool":171,"snaps":[{"begin":2,"length":1},{"begin":4,"length":2}]},{"pool":174,"snaps":[{"begin":2,"length":2}]},{"pool":183,"snaps":[{"begin":2,"length":1}]},{"pool":190,"snaps":[{"begin":1,"length":4}]},{"pool":201,"snaps":[{"begin":7,"length":1}]},{"pool":206,"snaps":[{"begin":4,"length":4}]},{"pool":225,"snaps":[{"begin":8,"length":1}]},{"pool":237,"snaps":[{"begin":10,"length":1}]},{"pool":243,"snaps":[{"begin":11,"length":2}]},{"pool":256,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T10:09:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 10 librados: watch_flush enter 2024-08-19T10:09:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 10 librados: watch_flush exit 2024-08-19T10:09:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 20 client.204623.objecter shutdown clearing up homeless session... 2024-08-19T10:09:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 10 client.204623.objecter successfully canceled tick 2024-08-19T10:09:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.82:6812/2549104415 conn(0x7feeec041a20 legacy=0x7feeec043ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 10 monclient: shutdown 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 >> v1:172.21.15.1:6793/0 conn(0x7fef0c0a67c0 legacy=0x7fef0c09fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.994+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 shutdown_connections 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.995+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 >> 172.21.15.1:0/1574859466 conn(0x7fef0c0ac4b0 msgr2=0x7fef0c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.995+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 shutdown_connections 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.995+0000 7fef10a0e640 1 -- 172.21.15.1:0/1574859466 wait complete. 2024-08-19T10:09:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:22.995+0000 7fef10a0e640 1 librados: shutdown 2024-08-19T10:09:24.007 INFO:tasks.ceph:Scrubbing osd.0 2024-08-19T10:09:24.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.123+0000 7f4b0242e640 10 monclient: get_monmap_and_config 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.123+0000 7f4b0242e640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.123+0000 7f4b0242e640 10 monclient: monmap: 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:24.124030+0000 2024-08-19T10:09:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:24.124030+0000 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.123+0000 7f4b0242e640 1 Processor -- start 2024-08-19T10:09:24.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 1 -- start start 2024-08-19T10:09:24.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient: init 2024-08-19T10:09:24.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient: _add_conns ranks=[2,1,5,8,7,3,4,0,6] 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient(hunting): picked mon.noname-b con 0x7f4afc0a66b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient(hunting): picked mon.noname-f con 0x7f4afc0a71a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient(hunting): picked mon.noname-h con 0x7f4afc0a3f10 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc0a2e80 con 0x7f4afc0a71a0 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc06eea0 con 0x7f4afc0a66b0 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc0a7f90 con 0x7f4afc0a3f10 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4b0242e640 10 monclient(hunting): authenticate will time out at 1820.200600s 2024-08-19T10:09:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4afbfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f4afc0a66b0 0x7f4afc0a6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49118/0 (socket says 172.21.15.1:49118) 2024-08-19T10:09:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4afbfff640 1 -- 172.21.15.1:0/2197873928 learned_addr learned my addr 172.21.15.1:0/2197873928 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.124+0000 7f4af3fff640 1 -- 172.21.15.1:0/2197873928 >> v1:172.21.15.82:6789/0 conn(0x7f4afc0a71a0 legacy=0x7f4afc13e6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:24.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.126+0000 7f4afbfff640 1 -- 172.21.15.1:0/2197873928 >> v1:172.21.15.1:6790/0 conn(0x7f4afc0a66b0 legacy=0x7f4afc0a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688891299 0 0) 0x7f4afc06eea0 con 0x7f4afc0a66b0 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient(hunting): my global_id is 184952 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae8003520 con 0x7f4afc0a66b0 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4b009cb640 1 -- 172.21.15.1:0/2197873928 >> v1:172.21.15.82:6791/0 conn(0x7f4afc0a3f10 legacy=0x7f4afc141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4004416773 0 0) 0x7f4ae8003520 con 0x7f4afc0a66b0 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 >> v1:172.21.15.82:6791/0 conn(0x7f4afc0a3f10 legacy=0x7f4afc141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 >> v1:172.21.15.82:6789/0 conn(0x7f4afc0a71a0 legacy=0x7f4afc13e6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:09:24.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 monclient: found mon.noname-b 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc0a3a10 con 0x7f4afc0a66b0 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4aec001ae0 con 0x7f4afc0a66b0 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: dump: 2024-08-19T10:09:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4b0242e640 5 monclient: authenticate success, global_id 184952 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4b0242e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4aec0048a0 con 0x7f4afc0a66b0 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 1 -- 172.21.15.1:0/2197873928 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4aec004e10 con 0x7f4afc0a66b0 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4afb7fe640 10 monclient: dump: 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.136 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.137 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4b0242e640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.127+0000 7f4b0242e640 4 set_mon_vals no callback set 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 10 monclient: shutdown 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 -- 172.21.15.1:0/2197873928 >> v1:172.21.15.1:6790/0 conn(0x7f4afc0a66b0 legacy=0x7f4afc0a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 -- 172.21.15.1:0/2197873928 shutdown_connections 2024-08-19T10:09:24.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 -- 172.21.15.1:0/2197873928 >> 172.21.15.1:0/2197873928 conn(0x7f4afc0ac340 msgr2=0x7f4afc0a87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 -- 172.21.15.1:0/2197873928 shutdown_connections 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 -- 172.21.15.1:0/2197873928 wait complete. 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 10 monclient: monmap: 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 librados: starting msgr at 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.128+0000 7f4b0242e640 1 librados: starting objecter 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 Processor -- start 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 -- start start 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 librados: setting wanted keys 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 librados: calling monclient init 2024-08-19T10:09:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient: init 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient: _add_conns ranks=[5,1,4,2,7,8,0,3,6] 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient(hunting): picked mon.noname-h con 0x7f4afc0a3f10 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient(hunting): picked mon.noname-f con 0x7f4afc0a66b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient(hunting): picked mon.noname-c con 0x7f4afc0a71a0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc126fa0 con 0x7f4afc0a66b0 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc127280 con 0x7f4afc0a71a0 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc0aa740 con 0x7f4afc0a3f10 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b0242e640 10 monclient(hunting): authenticate will time out at 1820.205615s 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4af3fff640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f4afc0a66b0 legacy=0x7f4afc1d2ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b009cb640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4afc0a71a0 0x7f4afc1d63d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48414/0 (socket says 172.21.15.1:48414) 2024-08-19T10:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.129+0000 7f4b009cb640 1 -- 172.21.15.1:0/2097815204 learned_addr learned my addr 172.21.15.1:0/2097815204 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.131+0000 7f4b009cb640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.1:6791/0 conn(0x7f4afc0a71a0 legacy=0x7f4afc1d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.131+0000 7f4af97fa640 10 client.?.objecter ms_handle_connect 0x7f4afc0a71a0 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.131+0000 7f4af97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4afbfff640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.82:6791/0 conn(0x7f4afc0a3f10 legacy=0x7f4afc1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 client.?.objecter ms_handle_connect 0x7f4afc0a3f10 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781673032 0 0) 0x7f4afc127280 con 0x7f4afc0a71a0 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): my global_id is 194800 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4adc004ee0 con 0x7f4afc0a71a0 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979152618 0 0) 0x7f4afc0aa740 con 0x7f4afc0a3f10 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): my global_id is 214358 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4afc127280 con 0x7f4afc0a3f10 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3836572923 0 0) 0x7f4adc004ee0 con 0x7f4afc0a71a0 2024-08-19T10:09:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4afc0aa740 con 0x7f4afc0a71a0 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4af4003990 con 0x7f4afc0a71a0 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): dump: 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1101944937 0 0) 0x7f4afc0aa740 con 0x7f4afc0a71a0 2024-08-19T10:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.82:6791/0 conn(0x7f4afc0a3f10 legacy=0x7f4afc1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.82:6789/0 conn(0x7f4afc0a66b0 legacy=0x7f4afc1d2ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 monclient: found mon.c 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc125f30 con 0x7f4afc0a71a0 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 5 monclient: authenticate success, global_id 194800 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 10 monclient: _renew_subs 2024-08-19T10:09:24.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc1269e0 con 0x7f4afc0a71a0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 10 client.194800.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 10 monclient: _renew_subs 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f4afc1cdbf0 con 0x7f4afc0a71a0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4b0242e640 1 librados: init done 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4af4003ee0 con 0x7f4afc0a71a0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4af4004410 con 0x7f4afc0a71a0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4afaffd640 4 set_mon_vals no callback set 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.132+0000 7f4af97fa640 10 monclient: dump: 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4b0242e640 10 client.194800.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4b0242e640 15 client.194800.objecter _session_command_op_assign -1 1 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4b0242e640 20 client.194800.objecter _get_session osd=-1 returning homeless 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4b0242e640 10 client.194800.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4b0242e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4ae0002910 con 0x7f4afc0a71a0 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.133+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4af4011e00 con 0x7f4afc0a71a0 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f4af404e330 con 0x7f4afc0a71a0 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 10 client.194800.objecter ms_dispatch 0x7f4afc000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 3 client.194800.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 3 client.194800.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 10 client.194800.objecter checking command 1 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 20 client.194800.objecter _get_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 15 client.194800.objecter _session_command_op_remove -1 1 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 20 client.194800.objecter _get_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 20 client.194800.objecter get_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 15 client.194800.objecter _session_command_op_assign 0 1 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 4 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 10 client.194800.objecter _send_command 1 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6804/3862255507 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4adc04d150 con 0x7f4adc049440 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 20 client.194800.objecter dump_active .. 0 homeless 2024-08-19T10:09:24.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_get_version_reply(handle=1 version=711) ==== 24+0+0 (unknown 3044792690 0 0) 0x7f4af404e7a0 con 0x7f4afc0a71a0 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af97fa640 10 monclient: handle_get_version_reply finishing 1 version 711 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4af3fff640 10 monclient: get_auth_request con 0x7f4adc049440 auth_method 0 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.134+0000 7f4afbfff640 10 monclient: get_auth_request con 0x7f4adc040da0 auth_method 0 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.136+0000 7f4af97fa640 10 client.194800.objecter ms_handle_connect 0x7f4adc040da0 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.138+0000 7f4af97fa640 10 client.194800.objecter ms_handle_connect 0x7f4adc049440 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.139+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== osd.0 v1:172.21.15.1:6804/3862255507 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4adc04d150 con 0x7f4adc049440 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.139+0000 7f4af97fa640 10 client.194800.objecter ms_dispatch 0x7f4afc000b90 command_reply(tid 1: 0 ) 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.139+0000 7f4af97fa640 10 client.194800.objecter _finish_command 1 = system:0 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.139+0000 7f4af97fa640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 4 2024-08-19T10:09:24.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.139+0000 7f4af97fa640 15 client.194800.objecter _session_command_op_remove 0 1 2024-08-19T10:09:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 client.194800.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T10:09:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 15 client.194800.objecter _session_command_op_assign -1 2 2024-08-19T10:09:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter _get_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter _get_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 15 client.194800.objecter _session_command_op_remove -1 2 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter get_session s=0x7f4adc048d90 osd=0 3 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 15 client.194800.objecter _session_command_op_assign 0 2 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 4 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 client.194800.objecter _send_command 2 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 --> v1:172.21.15.1:6804/3862255507 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4ae00012b0 con 0x7f4adc049440 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4af97fa640 1 -- 172.21.15.1:0/2097815204 <== osd.0 v1:172.21.15.1:6804/3862255507 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4ae00012b0 con 0x7f4adc049440 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4af97fa640 10 client.194800.objecter ms_dispatch 0x7f4afc000b90 command_reply(tid 2: 0 ) 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4af97fa640 10 client.194800.objecter _finish_command 2 = system:0 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4af97fa640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 4 2024-08-19T10:09:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4af97fa640 15 client.194800.objecter _session_command_op_remove 0 2 2024-08-19T10:09:24.158 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T10:09:24.158 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 librados: watch_flush enter 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 librados: watch_flush exit 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 client.194800.objecter close_session for osd.0 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.1:6804/3862255507 conn(0x7f4adc049440 legacy=0x7f4adc04b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter put_session s=0x7f4adc048d90 osd=0 1 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 client.194800.objecter shutdown clearing up homeless session... 2024-08-19T10:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 client.194800.objecter successfully canceled tick 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4adc040da0 legacy=0x7f4adc043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 10 monclient: shutdown 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 >> v1:172.21.15.1:6791/0 conn(0x7f4afc0a71a0 legacy=0x7f4afc1d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b009cb640 1 -- 172.21.15.1:0/2097815204 reap_dead start 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 shutdown_connections 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 >> 172.21.15.1:0/2097815204 conn(0x7f4afc0ac340 msgr2=0x7f4afc1c13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 shutdown_connections 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 -- 172.21.15.1:0/2097815204 wait complete. 2024-08-19T10:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.152+0000 7f4b0242e640 1 librados: shutdown 2024-08-19T10:09:24.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.278+0000 7f246dcd3640 10 monclient: get_monmap_and_config 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.278+0000 7f246dcd3640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.278+0000 7f246dcd3640 10 monclient: monmap: 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:24.278961+0000 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:24.278961+0000 2024-08-19T10:09:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.280 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.278+0000 7f246dcd3640 1 Processor -- start 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.278+0000 7f246dcd3640 1 -- start start 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.278+0000 7f246dcd3640 10 monclient: init 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient: _add_conns ranks=[8,7,1,0,3,5,4,6,2] 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient(hunting): picked mon.noname-e con 0x7f24680a6720 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient(hunting): picked mon.noname-i con 0x7f24680a7210 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient(hunting): picked mon.noname-f con 0x7f24680a4500 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f246806ed50 con 0x7f24680a4500 2024-08-19T10:09:24.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24680a3b20 con 0x7f24680a7210 2024-08-19T10:09:24.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f246806f190 con 0x7f24680a6720 2024-08-19T10:09:24.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f246dcd3640 10 monclient(hunting): authenticate will time out at 1820.355400s 2024-08-19T10:09:24.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f24677fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f24680a6720 0x7f24680a6b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48980/0 (socket says 172.21.15.1:48980) 2024-08-19T10:09:24.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f24677fe640 1 -- 172.21.15.1:0/1049074923 learned_addr learned my addr 172.21.15.1:0/1049074923 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.279+0000 7f2467fff640 1 -- 172.21.15.1:0/1049074923 >> v1:172.21.15.82:6789/0 conn(0x7f24680a4500 legacy=0x7f2468141ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:24.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.281+0000 7f24677fe640 1 -- 172.21.15.1:0/1049074923 >> v1:172.21.15.1:6793/0 conn(0x7f24680a6720 legacy=0x7f24680a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.281+0000 7f2466ffd640 1 -- 172.21.15.1:0/1049074923 >> v1:172.21.15.82:6792/0 conn(0x7f24680a7210 legacy=0x7f246813e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1238229881 0 0) 0x7f246806f190 con 0x7f24680a6720 2024-08-19T10:09:24.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient(hunting): my global_id is 204632 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2450003520 con 0x7f24680a6720 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 982216717 0 0) 0x7f2450003520 con 0x7f24680a6720 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 >> v1:172.21.15.82:6792/0 conn(0x7f24680a7210 legacy=0x7f246813e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 >> v1:172.21.15.82:6789/0 conn(0x7f24680a4500 legacy=0x7f2468141ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 monclient: found mon.noname-e 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24680a9260 con 0x7f24680a6720 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2458001ae0 con 0x7f24680a6720 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 5 monclient: authenticate success, global_id 204632 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: dump: 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.288 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24580048a0 con 0x7f24680a6720 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 1 -- 172.21.15.1:0/1049074923 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2458004e50 con 0x7f24680a6720 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 4 set_mon_vals no callback set 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f24667fc640 10 monclient: dump: 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 10 monclient: shutdown 2024-08-19T10:09:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 1 -- 172.21.15.1:0/1049074923 >> v1:172.21.15.1:6793/0 conn(0x7f24680a6720 legacy=0x7f24680a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 1 -- 172.21.15.1:0/1049074923 shutdown_connections 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 1 -- 172.21.15.1:0/1049074923 >> 172.21.15.1:0/1049074923 conn(0x7f24680ac440 msgr2=0x7f24680a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 1 -- 172.21.15.1:0/1049074923 shutdown_connections 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.282+0000 7f246dcd3640 1 -- 172.21.15.1:0/1049074923 wait complete. 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.283+0000 7f246dcd3640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.283+0000 7f246dcd3640 10 monclient: monmap: 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.283+0000 7f246dcd3640 1 librados: starting msgr at 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.283+0000 7f246dcd3640 1 librados: starting objecter 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.283+0000 7f246dcd3640 1 Processor -- start 2024-08-19T10:09:24.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 1 -- start start 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 1 librados: setting wanted keys 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 1 librados: calling monclient init 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient: init 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient: _add_conns ranks=[8,2,4,0,7,6,5,1,3] 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient(hunting): picked mon.noname-e con 0x7f24680a4500 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient(hunting): picked mon.noname-b con 0x7f24680a6720 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient(hunting): picked mon.noname-c con 0x7f24680a7210 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2468138850 con 0x7f24680a6720 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24681364f0 con 0x7f24680a7210 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2468137fb0 con 0x7f24680a4500 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f246dcd3640 10 monclient(hunting): authenticate will time out at 1820.360449s 2024-08-19T10:09:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f24677fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f24680a4500 0x7f24681cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:48996/0 (socket says 172.21.15.1:48996) 2024-08-19T10:09:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.284+0000 7f24677fe640 1 -- 172.21.15.1:0/1431325218 learned_addr learned my addr 172.21.15.1:0/1431325218 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f24677fe640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.1:6793/0 conn(0x7f24680a4500 legacy=0x7f24681cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 client.?.objecter ms_handle_connect 0x7f24680a4500 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f2466ffd640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.1:6790/0 conn(0x7f24680a6720 legacy=0x7f2468141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 client.?.objecter ms_handle_connect 0x7f24680a6720 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 client.?.objecter ms_handle_connect 0x7f24680a7210 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f2467fff640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.1:6791/0 conn(0x7f24680a7210 legacy=0x7f24681d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469882243 0 0) 0x7f2468138850 con 0x7f24680a6720 2024-08-19T10:09:24.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): my global_id is 184961 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2454004eb0 con 0x7f24680a6720 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543181984 0 0) 0x7f2468137fb0 con 0x7f24680a4500 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): my global_id is 204641 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2468138850 con 0x7f24680a4500 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2487582464 0 0) 0x7f24681364f0 con 0x7f24680a7210 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 10 monclient(hunting): my global_id is 194809 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.286+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2468137fb0 con 0x7f24680a7210 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4127684366 0 0) 0x7f2454004eb0 con 0x7f24680a6720 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24681364f0 con 0x7f24680a6720 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4273238636 0 0) 0x7f2468137fb0 con 0x7f24680a7210 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2454004eb0 con 0x7f24680a7210 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3659017514 0 0) 0x7f2468138850 con 0x7f24680a4500 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2468137fb0 con 0x7f24680a4500 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f245c001d30 con 0x7f24680a6720 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:24.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): dump: 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2460004780 con 0x7f24680a7210 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): dump: 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.303 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2458004d90 con 0x7f24680a4500 2024-08-19T10:09:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient(hunting): dump: 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3943431310 0 0) 0x7f24681364f0 con 0x7f24680a6720 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.1:6793/0 conn(0x7f24680a4500 legacy=0x7f24681cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.1:6791/0 conn(0x7f24680a7210 legacy=0x7f24681d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 monclient: found mon.b 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2468142510 con 0x7f24680a6720 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 5 monclient: authenticate success, global_id 184961 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 10 monclient: _renew_subs 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2468137050 con 0x7f24680a6720 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 10 client.184961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 10 monclient: _renew_subs 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f24681d7880 con 0x7f24680a6720 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f246dcd3640 1 librados: init done 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f245c0038c0 con 0x7f24680a6720 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f245c003df0 con 0x7f24680a6720 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f24657fa640 4 set_mon_vals no callback set 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.287+0000 7f244ffff640 10 monclient: dump: 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f246dcd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f246dcd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f246dcd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f245c004090 con 0x7f24680a6720 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2448005080 con 0x7f24680a6720 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f245c04def0 con 0x7f24680a6720 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f244ffff640 10 client.184961.objecter ms_dispatch 0x7f2468000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:24.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f244ffff640 3 client.184961.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f244ffff640 3 client.184961.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.288+0000 7f244ffff640 20 client.184961.objecter dump_active .. 0 homeless 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.289+0000 7f24677fe640 10 monclient: get_auth_request con 0x7f2454041980 auth_method 0 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.291+0000 7f244ffff640 10 client.184961.objecter ms_handle_connect 0x7f2454041980 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.293+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f245c017550 con 0x7f24680a6720 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.293+0000 7f244ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.293+0000 7f244ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:24.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.475+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f2448000b90 con 0x7f2454041980 2024-08-19T10:09:24.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.476+0000 7f244ffff640 1 -- 172.21.15.1:0/1431325218 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f2448000b90 con 0x7f2454041980 2024-08-19T10:09:24.477 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 10 librados: watch_flush enter 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 10 librados: watch_flush exit 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 20 client.184961.objecter shutdown clearing up homeless session... 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 10 client.184961.objecter successfully canceled tick 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2454041980 legacy=0x7f2454043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 10 monclient: shutdown 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 >> v1:172.21.15.1:6790/0 conn(0x7f24680a6720 legacy=0x7f2468141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 shutdown_connections 2024-08-19T10:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 >> 172.21.15.1:0/1431325218 conn(0x7f24680ac440 msgr2=0x7f2468133660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 shutdown_connections 2024-08-19T10:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 -- 172.21.15.1:0/1431325218 wait complete. 2024-08-19T10:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.478+0000 7f246dcd3640 1 librados: shutdown 2024-08-19T10:09:24.490 INFO:tasks.ceph:Scrubbing osd.1 2024-08-19T10:09:24.490 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.591+0000 7f407f514640 10 monclient: get_monmap_and_config 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.591+0000 7f407f514640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.591+0000 7f407f514640 10 monclient: monmap: 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:24.592849+0000 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:24.592849+0000 2024-08-19T10:09:24.593 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.594 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.592+0000 7f407f514640 1 Processor -- start 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.592+0000 7f407f514640 1 -- start start 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.592+0000 7f407f514640 10 monclient: init 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient: _add_conns ranks=[7,4,5,6,3,1,0,2,8] 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient(hunting): picked mon.noname-i con 0x7f40780a66b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient(hunting): picked mon.noname-c con 0x7f40780a71a0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:24.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient(hunting): picked mon.noname-h con 0x7f40780a3f10 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40780a2e80 con 0x7f40780a71a0 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407806eea0 con 0x7f40780a3f10 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40780a7f90 con 0x7f40780a66b0 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407f514640 10 monclient(hunting): authenticate will time out at 1820.669258s 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407caaf640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f40780a71a0 0x7f407813e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48434/0 (socket says 172.21.15.1:48434) 2024-08-19T10:09:24.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.593+0000 7f407caaf640 1 -- 172.21.15.1:0/3070668373 learned_addr learned my addr 172.21.15.1:0/3070668373 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f407caaf640 1 -- 172.21.15.1:0/3070668373 >> v1:172.21.15.1:6791/0 conn(0x7f40780a71a0 legacy=0x7f407813e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f407dab1640 1 -- 172.21.15.1:0/3070668373 >> v1:172.21.15.82:6791/0 conn(0x7f40780a3f10 legacy=0x7f4078141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2571437815 0 0) 0x7f40780a2e80 con 0x7f40780a71a0 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f407d2b0640 1 -- 172.21.15.1:0/3070668373 >> v1:172.21.15.82:6792/0 conn(0x7f40780a66b0 legacy=0x7f40780a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): my global_id is 194818 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4064003520 con 0x7f40780a71a0 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3542854127 0 0) 0x7f407806eea0 con 0x7f40780a3f10 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): my global_id is 214367 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40780a2e80 con 0x7f40780a3f10 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64735758 0 0) 0x7f40740014a0 con 0x7f40780a66b0 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient(hunting): my global_id is 204586 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f407806eea0 con 0x7f40780a66b0 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2284667998 0 0) 0x7f4064003520 con 0x7f40780a71a0 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 >> v1:172.21.15.82:6792/0 conn(0x7f40780a66b0 legacy=0x7f40780a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 >> v1:172.21.15.82:6791/0 conn(0x7f40780a3f10 legacy=0x7f4078141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 monclient: found mon.noname-c 2024-08-19T10:09:24.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40780a3a10 con 0x7f40780a71a0 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4068001ae0 con 0x7f40780a71a0 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.595+0000 7f406ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 10 monclient: dump: 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.611 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 5 monclient: authenticate success, global_id 194818 2024-08-19T10:09:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40680048a0 con 0x7f40780a71a0 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 1 -- 172.21.15.1:0/3070668373 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4068004e10 con 0x7f40780a71a0 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 4 set_mon_vals no callback set 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f406ffff640 10 monclient: dump: 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 10 monclient: shutdown 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 1 -- 172.21.15.1:0/3070668373 >> v1:172.21.15.1:6791/0 conn(0x7f40780a71a0 legacy=0x7f407813e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 1 -- 172.21.15.1:0/3070668373 shutdown_connections 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 1 -- 172.21.15.1:0/3070668373 >> 172.21.15.1:0/3070668373 conn(0x7f40780ac340 msgr2=0x7f40780a87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 1 -- 172.21.15.1:0/3070668373 shutdown_connections 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.596+0000 7f407f514640 1 -- 172.21.15.1:0/3070668373 wait complete. 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 10 monclient: monmap: 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 1 librados: starting msgr at 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 1 librados: starting objecter 2024-08-19T10:09:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 1 Processor -- start 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 1 -- start start 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 1 librados: setting wanted keys 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 1 librados: calling monclient init 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.597+0000 7f407f514640 10 monclient: init 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient: _add_conns ranks=[6,8,0,1,2,3,7,4,5] 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient(hunting): picked mon.noname-d con 0x7f40780a3f10 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient(hunting): picked mon.noname-e con 0x7f40780a66b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient(hunting): picked mon.noname-a con 0x7f40780a71a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40780aa2d0 con 0x7f40780a71a0 2024-08-19T10:09:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40781271b0 con 0x7f40780a3f10 2024-08-19T10:09:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4078125c90 con 0x7f40780a66b0 2024-08-19T10:09:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407f514640 10 monclient(hunting): authenticate will time out at 1820.674124s 2024-08-19T10:09:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407d2b0640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f40780a3f10 0x7f40781cddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:37444/0 (socket says 172.21.15.1:37444) 2024-08-19T10:09:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407d2b0640 1 -- 172.21.15.1:0/2721934675 learned_addr learned my addr 172.21.15.1:0/2721934675 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.598+0000 7f407caaf640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f40780a66b0 0x7f4078139ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:49012/0 (socket says 172.21.15.1:49012) 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f407caaf640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6793/0 conn(0x7f40780a66b0 legacy=0x7f4078139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 client.?.objecter ms_handle_connect 0x7f40780a66b0 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 client.?.objecter ms_handle_connect 0x7f40780a3f10 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 client.?.objecter ms_handle_connect 0x7f40780a71a0 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f407d2b0640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6792/0 conn(0x7f40780a3f10 legacy=0x7f40781cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f407dab1640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6789/0 conn(0x7f40780a71a0 legacy=0x7f40781d6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280338653 0 0) 0x7f4078125c90 con 0x7f40780a66b0 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): my global_id is 204650 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f405c004eb0 con 0x7f40780a66b0 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325851333 0 0) 0x7f40781271b0 con 0x7f40780a3f10 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): my global_id is 204495 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4078125c90 con 0x7f40780a3f10 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992405924 0 0) 0x7f40780aa2d0 con 0x7f40780a71a0 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): my global_id is 204615 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40781271b0 con 0x7f40780a71a0 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1683860338 0 0) 0x7f405c004eb0 con 0x7f40780a66b0 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40780aa2d0 con 0x7f40780a66b0 2024-08-19T10:09:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4068001d30 con 0x7f40780a66b0 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): dump: 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2791403639 0 0) 0x7f40781271b0 con 0x7f40780a71a0 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f405c004eb0 con 0x7f40780a71a0 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1473867704 0 0) 0x7f4078125c90 con 0x7f40780a3f10 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40781271b0 con 0x7f40780a3f10 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4070003990 con 0x7f40780a71a0 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.600+0000 7f406dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient(hunting): dump: 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.625 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4074003610 con 0x7f40780a3f10 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient(hunting): dump: 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.627 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1726307476 0 0) 0x7f40780aa2d0 con 0x7f40780a66b0 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6792/0 conn(0x7f40780a3f10 legacy=0x7f40781cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6789/0 conn(0x7f40780a71a0 legacy=0x7f40781d6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 monclient: found mon.e 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4078125e60 con 0x7f40780a66b0 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 5 monclient: authenticate success, global_id 204650 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 10 monclient: _renew_subs 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f407813a1e0 con 0x7f40780a66b0 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4068005040 con 0x7f40780a66b0 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4068005530 con 0x7f40780a66b0 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406dffb640 10 monclient: dump: 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 10 client.204650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f406f7fe640 4 set_mon_vals no callback set 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 10 monclient: _renew_subs 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f407813a470 con 0x7f40780a66b0 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.601+0000 7f407f514640 1 librados: init done 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4068011990 con 0x7f40780a66b0 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407f514640 10 client.204650.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407f514640 15 client.204650.objecter _session_command_op_assign -1 1 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407f514640 20 client.204650.objecter _get_session osd=-1 returning homeless 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407f514640 10 client.204650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407f514640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f40780aa670 con 0x7f40780a66b0 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f406804dd90 con 0x7f40780a66b0 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 10 client.204650.objecter ms_dispatch 0x7f4078000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 3 client.204650.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 3 client.204650.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 10 client.204650.objecter checking command 1 2024-08-19T10:09:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 20 client.204650.objecter _get_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 15 client.204650.objecter _session_command_op_remove -1 1 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 20 client.204650.objecter _get_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 20 client.204650.objecter get_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 15 client.204650.objecter _session_command_op_assign 1 1 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 4 2024-08-19T10:09:24.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 10 client.204650.objecter _send_command 1 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6800/1668590231 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40680048a0 con 0x7f405c04a440 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 20 client.204650.objecter dump_active .. 0 homeless 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_get_version_reply(handle=1 version=711) ==== 24+0+0 (unknown 3044792690 0 0) 0x7f406804e250 con 0x7f40780a66b0 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f406dffb640 10 monclient: handle_get_version_reply finishing 1 version 711 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407dab1640 10 monclient: get_auth_request con 0x7f405c04a440 auth_method 0 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.602+0000 7f407d2b0640 10 monclient: get_auth_request con 0x7f405c041d10 auth_method 0 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.605+0000 7f406dffb640 10 client.204650.objecter ms_handle_connect 0x7f405c041d10 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.606+0000 7f406dffb640 10 client.204650.objecter ms_handle_connect 0x7f405c04a440 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.607+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== osd.1 v1:172.21.15.1:6800/1668590231 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f40680048a0 con 0x7f405c04a440 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.607+0000 7f406dffb640 10 client.204650.objecter ms_dispatch 0x7f4078000b90 command_reply(tid 1: 0 ) 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.607+0000 7f406dffb640 10 client.204650.objecter _finish_command 1 = system:0 2024-08-19T10:09:24.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.607+0000 7f406dffb640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 4 2024-08-19T10:09:24.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.607+0000 7f406dffb640 15 client.204650.objecter _session_command_op_remove 1 1 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 10 client.204650.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 15 client.204650.objecter _session_command_op_assign -1 2 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 20 client.204650.objecter _get_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 20 client.204650.objecter _get_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 15 client.204650.objecter _session_command_op_remove -1 2 2024-08-19T10:09:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 20 client.204650.objecter get_session s=0x7f405c049d90 osd=1 3 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 15 client.204650.objecter _session_command_op_assign 1 2 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 4 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 10 client.204650.objecter _send_command 2 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 --> v1:172.21.15.1:6800/1668590231 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4078051b00 con 0x7f405c04a440 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f406dffb640 1 -- 172.21.15.1:0/2721934675 <== osd.1 v1:172.21.15.1:6800/1668590231 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4078051b00 con 0x7f405c04a440 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f406dffb640 10 client.204650.objecter ms_dispatch 0x7f4078000b90 command_reply(tid 2: 0 ) 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f406dffb640 10 client.204650.objecter _finish_command 2 = system:0 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f406dffb640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 4 2024-08-19T10:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.620+0000 7f406dffb640 15 client.204650.objecter _session_command_op_remove 1 2 2024-08-19T10:09:24.638 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T10:09:24.638 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 10 librados: watch_flush enter 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 10 librados: watch_flush exit 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 10 client.204650.objecter close_session for osd.1 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6800/1668590231 conn(0x7f405c04a440 legacy=0x7f405c04c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 20 client.204650.objecter put_session s=0x7f405c049d90 osd=1 1 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 20 client.204650.objecter shutdown clearing up homeless session... 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 10 client.204650.objecter successfully canceled tick 2024-08-19T10:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.82:6812/2549104415 conn(0x7f405c041d10 legacy=0x7f405c0441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 10 monclient: shutdown 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 >> v1:172.21.15.1:6793/0 conn(0x7f40780a66b0 legacy=0x7f4078139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407dab1640 1 -- 172.21.15.1:0/2721934675 reap_dead start 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 shutdown_connections 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 >> 172.21.15.1:0/2721934675 conn(0x7f40780ac340 msgr2=0x7f40780aad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 shutdown_connections 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 -- 172.21.15.1:0/2721934675 wait complete. 2024-08-19T10:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.621+0000 7f407f514640 1 librados: shutdown 2024-08-19T10:09:24.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-19T10:09:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.740+0000 7fc8482d5640 10 monclient: get_monmap_and_config 2024-08-19T10:09:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.740+0000 7fc8482d5640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.740+0000 7fc8482d5640 10 monclient: monmap: 2024-08-19T10:09:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:24.741549+0000 2024-08-19T10:09:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:24.741549+0000 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.743 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 1 Processor -- start 2024-08-19T10:09:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 1 -- start start 2024-08-19T10:09:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient: init 2024-08-19T10:09:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient: _add_conns ranks=[8,5,2,0,4,7,3,1,6] 2024-08-19T10:09:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient(hunting): picked mon.noname-e con 0x7fc8400a6720 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient(hunting): picked mon.noname-h con 0x7fc8400a7210 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient(hunting): picked mon.noname-b con 0x7fc8400a4500 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84006ed50 con 0x7fc8400a4500 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8400a3b20 con 0x7fc8400a7210 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84006f190 con 0x7fc8400a6720 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc8482d5640 10 monclient(hunting): authenticate will time out at 1820.817898s 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.741+0000 7fc846071640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc8400a6720 0x7fc8400a6b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:49016/0 (socket says 172.21.15.1:49016) 2024-08-19T10:09:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.742+0000 7fc846071640 1 -- 172.21.15.1:0/1713382156 learned_addr learned my addr 172.21.15.1:0/1713382156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc846071640 1 -- 172.21.15.1:0/1713382156 >> v1:172.21.15.1:6793/0 conn(0x7fc8400a6720 legacy=0x7fc8400a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc846872640 1 -- 172.21.15.1:0/1713382156 >> v1:172.21.15.1:6790/0 conn(0x7fc8400a4500 legacy=0x7fc840141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119444659 0 0) 0x7fc84006ed50 con 0x7fc8400a4500 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): my global_id is 184970 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc82c003520 con 0x7fc8400a4500 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29766088 0 0) 0x7fc84006f190 con 0x7fc8400a6720 2024-08-19T10:09:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient(hunting): my global_id is 204659 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84006ed50 con 0x7fc8400a6720 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc845870640 1 -- 172.21.15.1:0/1713382156 >> v1:172.21.15.82:6791/0 conn(0x7fc8400a7210 legacy=0x7fc84013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1913669928 0 0) 0x7fc84006ed50 con 0x7fc8400a6720 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 >> v1:172.21.15.82:6791/0 conn(0x7fc8400a7210 legacy=0x7fc84013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 >> v1:172.21.15.1:6790/0 conn(0x7fc8400a4500 legacy=0x7fc840141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 monclient: found mon.noname-e 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:09:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8400a9260 con 0x7fc8400a6720 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc83c001ac0 con 0x7fc8400a6720 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: dump: 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc8482d5640 5 monclient: authenticate success, global_id 204659 2024-08-19T10:09:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc8482d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc846071640 1 --1- 172.21.15.1:0/1713382156 >> v1:172.21.15.1:6793/0 conn(0x7fc8400a6720 0x7fc8400a6b00 :-1 s=OPENED pgs=3767 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fc83c004e10 mon_map magic: 0 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc83c0048a0 con 0x7fc8400a6720 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc84506f640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc8482d5640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.744+0000 7fc8482d5640 4 set_mon_vals no callback set 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 10 monclient: shutdown 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 1 -- 172.21.15.1:0/1713382156 >> v1:172.21.15.1:6793/0 conn(0x7fc8400a6720 legacy=0x7fc8400a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc84506f640 1 -- 172.21.15.1:0/1713382156 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc83c004e10 con 0x7fc8400a6720 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc84506f640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 1 -- 172.21.15.1:0/1713382156 shutdown_connections 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 1 -- 172.21.15.1:0/1713382156 >> 172.21.15.1:0/1713382156 conn(0x7fc8400ac440 msgr2=0x7fc8400a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 1 -- 172.21.15.1:0/1713382156 shutdown_connections 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 1 -- 172.21.15.1:0/1713382156 wait complete. 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 10 monclient: build_initial_monmap 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.745+0000 7fc8482d5640 10 monclient: monmap: 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:24.756 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 librados: starting msgr at 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 librados: starting objecter 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 Processor -- start 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 -- start start 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 librados: setting wanted keys 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 librados: calling monclient init 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient: init 2024-08-19T10:09:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient: _add_conns ranks=[3,2,7,0,1,5,4,8,6] 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient(hunting): picked mon.noname-g con 0x7fc8400a4500 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient(hunting): picked mon.noname-b con 0x7fc8400a6720 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient(hunting): picked mon.noname-i con 0x7fc8400a7210 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8401425e0 con 0x7fc8400a6720 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8400aa760 con 0x7fc8400a4500 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8401274d0 con 0x7fc8400a7210 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc8482d5640 10 monclient(hunting): authenticate will time out at 1820.822684s 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc845870640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc8400a6720 0x7fc8401d3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49150/0 (socket says 172.21.15.1:49150) 2024-08-19T10:09:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.746+0000 7fc845870640 1 -- 172.21.15.1:0/1108939175 learned_addr learned my addr 172.21.15.1:0/1108939175 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc845870640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.1:6790/0 conn(0x7fc8400a6720 legacy=0x7fc8401d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 client.?.objecter ms_handle_connect 0x7fc8400a6720 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 client.?.objecter ms_handle_connect 0x7fc8400a7210 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc846872640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.82:6792/0 conn(0x7fc8400a7210 legacy=0x7fc8401d6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 client.?.objecter ms_handle_connect 0x7fc8400a4500 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc846071640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.82:6790/0 conn(0x7fc8400a4500 legacy=0x7fc840141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:24.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164206398 0 0) 0x7fc8401425e0 con 0x7fc8400a6720 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): my global_id is 184979 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc824004eb0 con 0x7fc8400a6720 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518269999 0 0) 0x7fc8400aa760 con 0x7fc8400a4500 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): my global_id is 214365 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8401425e0 con 0x7fc8400a4500 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826135979 0 0) 0x7fc8401274d0 con 0x7fc8400a7210 2024-08-19T10:09:24.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): my global_id is 204595 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8400aa760 con 0x7fc8400a7210 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3801905830 0 0) 0x7fc824004eb0 con 0x7fc8400a6720 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8401274d0 con 0x7fc8400a6720 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc830003610 con 0x7fc8400a6720 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.766 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 523366068 0 0) 0x7fc8401425e0 con 0x7fc8400a4500 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc824004eb0 con 0x7fc8400a4500 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc83c004d90 con 0x7fc8400a4500 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1742908468 0 0) 0x7fc8400aa760 con 0x7fc8400a7210 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8401425e0 con 0x7fc8400a7210 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 225302062 0 0) 0x7fc8401274d0 con 0x7fc8400a6720 2024-08-19T10:09:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.82:6792/0 conn(0x7fc8400a7210 legacy=0x7fc8401d6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.82:6790/0 conn(0x7fc8400a4500 legacy=0x7fc840141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 monclient: found mon.b 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc840128f90 con 0x7fc8400a6720 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 5 monclient: authenticate success, global_id 184979 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 10 monclient: _renew_subs 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc840128030 con 0x7fc8400a6720 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 10 client.184979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 10 monclient: _renew_subs 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc8401ce260 con 0x7fc8400a6720 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.749+0000 7fc8482d5640 1 librados: init done 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc830003b20 con 0x7fc8400a6720 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc836ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc830004010 con 0x7fc8400a6720 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc836ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc836ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc84486e640 4 set_mon_vals no callback set 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.750+0000 7fc836ffd640 10 monclient: dump: 2024-08-19T10:09:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:24.771 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc8482d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc8482d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc8482d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc828005080 con 0x7fc8400a6720 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc8300041b0 con 0x7fc8400a6720 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fc83004dfb0 con 0x7fc8400a6720 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc836ffd640 10 client.184979.objecter ms_dispatch 0x7fc840000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc836ffd640 3 client.184979.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc836ffd640 3 client.184979.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc836ffd640 20 client.184979.objecter dump_active .. 0 homeless 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.751+0000 7fc846071640 10 monclient: get_auth_request con 0x7fc824041850 auth_method 0 2024-08-19T10:09:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.753+0000 7fc836ffd640 10 client.184979.objecter ms_handle_connect 0x7fc824041850 2024-08-19T10:09:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.755+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fc8300175b0 con 0x7fc8400a6720 2024-08-19T10:09:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.755+0000 7fc836ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.755+0000 7fc836ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:24.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.938+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fc828000b90 con 0x7fc824041850 2024-08-19T10:09:24.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.938+0000 7fc836ffd640 1 -- 172.21.15.1:0/1108939175 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7fc828000b90 con 0x7fc824041850 2024-08-19T10:09:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 10 librados: watch_flush enter 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 10 librados: watch_flush exit 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 20 client.184979.objecter shutdown clearing up homeless session... 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 10 client.184979.objecter successfully canceled tick 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc824041850 legacy=0x7fc824043d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 10 monclient: shutdown 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:24.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 >> v1:172.21.15.1:6790/0 conn(0x7fc8400a6720 legacy=0x7fc8401d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:24.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 shutdown_connections 2024-08-19T10:09:24.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 >> 172.21.15.1:0/1108939175 conn(0x7fc8400ac440 msgr2=0x7fc84006df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:24.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 shutdown_connections 2024-08-19T10:09:24.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 -- 172.21.15.1:0/1108939175 wait complete. 2024-08-19T10:09:24.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:24.940+0000 7fc8482d5640 1 librados: shutdown 2024-08-19T10:09:24.952 INFO:tasks.ceph:Scrubbing osd.2 2024-08-19T10:09:24.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-19T10:09:25.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.054+0000 7fd7d5867640 10 monclient: get_monmap_and_config 2024-08-19T10:09:25.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.054+0000 7fd7d5867640 10 monclient: build_initial_monmap 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.054+0000 7fd7d5867640 10 monclient: monmap: 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:25.055324+0000 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:25.055324+0000 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:25.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:25.057 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:25.057 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:25.057 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:25.057 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.054+0000 7fd7d5867640 1 Processor -- start 2024-08-19T10:09:25.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 1 -- start start 2024-08-19T10:09:25.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient: init 2024-08-19T10:09:25.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:25.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient: _add_conns ranks=[5,4,0,3,7,2,6,8,1] 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient(hunting): picked mon.noname-h con 0x7fd7d006e7f0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient(hunting): picked mon.noname-c con 0x7fd7d00a5920 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient(hunting): picked mon.noname-a con 0x7fd7d00a5d40 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d01366e0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d009ddd0 con 0x7fd7d00a5920 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d009e0b0 con 0x7fd7d006e7f0 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7d5867640 10 monclient(hunting): authenticate will time out at 1821.131576s 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7cf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7d00a5d40 0x7fd7d0141b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51538/0 (socket says 172.21.15.1:51538) 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7cf7fe640 1 -- 172.21.15.1:0/2661294277 learned_addr learned my addr 172.21.15.1:0/2661294277 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.055+0000 7fd7ce7fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fd7d00a5920 0x7fd7d013e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48440/0 (socket says 172.21.15.1:48440) 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.057+0000 7fd7cf7fe640 1 -- 172.21.15.1:0/2661294277 >> v1:172.21.15.1:6789/0 conn(0x7fd7d00a5d40 legacy=0x7fd7d0141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.057+0000 7fd7ce7fc640 1 -- 172.21.15.1:0/2661294277 >> v1:172.21.15.1:6791/0 conn(0x7fd7d00a5920 legacy=0x7fd7d013e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403658016 0 0) 0x7fd7d01366e0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): my global_id is 204624 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b8003520 con 0x7fd7d00a5d40 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785429388 0 0) 0x7fd7d009ddd0 con 0x7fd7d00a5920 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient(hunting): my global_id is 194827 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d01366e0 con 0x7fd7d00a5920 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7ceffd640 1 -- 172.21.15.1:0/2661294277 >> v1:172.21.15.82:6791/0 conn(0x7fd7d006e7f0 legacy=0x7fd7d0135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1047559325 0 0) 0x7fd7b8003520 con 0x7fd7d00a5d40 2024-08-19T10:09:25.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 >> v1:172.21.15.82:6791/0 conn(0x7fd7d006e7f0 legacy=0x7fd7d0135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 >> v1:172.21.15.1:6791/0 conn(0x7fd7d00a5920 legacy=0x7fd7d013e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 monclient: found mon.noname-a 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d009e390 con 0x7fd7d00a5d40 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7c8001e60 con 0x7fd7d00a5d40 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 5 monclient: authenticate success, global_id 204624 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:25.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: dump: 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.066 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7c8004c20 con 0x7fd7d00a5d40 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 1 -- 172.21.15.1:0/2661294277 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7c8005190 con 0x7fd7d00a5d40 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7cdffb640 10 monclient: dump: 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 4 set_mon_vals no callback set 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 10 monclient: shutdown 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 1 -- 172.21.15.1:0/2661294277 >> v1:172.21.15.1:6789/0 conn(0x7fd7d00a5d40 legacy=0x7fd7d0141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 1 -- 172.21.15.1:0/2661294277 shutdown_connections 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.058+0000 7fd7d5867640 1 -- 172.21.15.1:0/2661294277 >> 172.21.15.1:0/2661294277 conn(0x7fd7d00ac6a0 msgr2=0x7fd7d00a8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 1 -- 172.21.15.1:0/2661294277 shutdown_connections 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 1 -- 172.21.15.1:0/2661294277 wait complete. 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 10 monclient: build_initial_monmap 2024-08-19T10:09:25.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 10 monclient: monmap: 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 1 librados: starting msgr at 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 1 librados: starting objecter 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.059+0000 7fd7d5867640 1 Processor -- start 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 1 -- start start 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 1 librados: setting wanted keys 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 1 librados: calling monclient init 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient: init 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient: _add_conns ranks=[8,5,2,6,4,3,7,0,1] 2024-08-19T10:09:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient(hunting): picked mon.noname-e con 0x7fd7d006e7f0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient(hunting): picked mon.noname-h con 0x7fd7d00a5920 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient(hunting): picked mon.noname-b con 0x7fd7d00a5d40 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d0132940 con 0x7fd7d00a5d40 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d00a0b00 con 0x7fd7d00a5920 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d00a0260 con 0x7fd7d006e7f0 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7d5867640 10 monclient(hunting): authenticate will time out at 1821.136629s 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7ceffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fd7d006e7f0 0x7fd7d0131b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:49030/0 (socket says 172.21.15.1:49030) 2024-08-19T10:09:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.060+0000 7fd7ceffd640 1 -- 172.21.15.1:0/3653949476 learned_addr learned my addr 172.21.15.1:0/3653949476 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7ceffd640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.1:6793/0 conn(0x7fd7d006e7f0 legacy=0x7fd7d0131b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7b7fff640 10 client.?.objecter ms_handle_connect 0x7fd7d006e7f0 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7b7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7b7fff640 10 client.?.objecter ms_handle_connect 0x7fd7d00a5d40 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7b7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7cf7fe640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.1:6790/0 conn(0x7fd7d00a5d40 legacy=0x7fd7d01d6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7ce7fc640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.82:6791/0 conn(0x7fd7d00a5920 legacy=0x7fd7d0132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.062+0000 7fd7b7fff640 10 client.?.objecter ms_handle_connect 0x7fd7d00a5920 2024-08-19T10:09:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786182010 0 0) 0x7fd7d0132940 con 0x7fd7d00a5d40 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): my global_id is 184988 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7bc004ee0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006865401 0 0) 0x7fd7d00a0260 con 0x7fd7d006e7f0 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): my global_id is 204668 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d0132940 con 0x7fd7d006e7f0 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424391921 0 0) 0x7fd7d00a0b00 con 0x7fd7d00a5920 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): my global_id is 214394 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d00a0260 con 0x7fd7d00a5920 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4286374815 0 0) 0x7fd7bc004ee0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7d00a0b00 con 0x7fd7d00a5d40 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7c8005110 con 0x7fd7d00a5d40 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): dump: 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3319077989 0 0) 0x7fd7d0132940 con 0x7fd7d006e7f0 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7bc004ee0 con 0x7fd7d006e7f0 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7c0003610 con 0x7fd7d006e7f0 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): dump: 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.081 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2457883387 0 0) 0x7fd7d00a0260 con 0x7fd7d00a5920 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7d0132940 con 0x7fd7d00a5920 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7c4001320 con 0x7fd7d00a5920 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:25.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient(hunting): dump: 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.083 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3297310774 0 0) 0x7fd7d00a0b00 con 0x7fd7d00a5d40 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.1:6793/0 conn(0x7fd7d006e7f0 legacy=0x7fd7d0131b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.82:6791/0 conn(0x7fd7d00a5920 legacy=0x7fd7d0132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 monclient: found mon.b 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d00a5240 con 0x7fd7d00a5d40 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: _finish_auth 0 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:25.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 5 monclient: authenticate success, global_id 184988 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 10 monclient: _renew_subs 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d00a54f0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 10 client.184988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 10 monclient: _renew_subs 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7d009f4e0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7d5867640 1 librados: init done 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7c80020b0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fd7c8001e80 con 0x7fd7d00a5d40 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7cd7fa640 4 set_mon_vals no callback set 2024-08-19T10:09:25.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.063+0000 7fd7b7fff640 10 monclient: dump: 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7d5867640 10 client.184988.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7d5867640 15 client.184988.objecter _session_command_op_assign -1 1 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7d5867640 20 client.184988.objecter _get_session osd=-1 returning homeless 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7d5867640 10 client.184988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7d5867640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd7b0002910 con 0x7fd7d00a5d40 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fd7c8011d80 con 0x7fd7d00a5d40 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fd7c804e370 con 0x7fd7d00a5d40 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 10 client.184988.objecter ms_dispatch 0x7fd7d0000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 3 client.184988.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 3 client.184988.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 10 client.184988.objecter checking command 1 2024-08-19T10:09:25.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 20 client.184988.objecter _get_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 15 client.184988.objecter _session_command_op_remove -1 1 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 20 client.184988.objecter _get_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 20 client.184988.objecter get_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 15 client.184988.objecter _session_command_op_assign 2 1 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 4 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 10 client.184988.objecter _send_command 1 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6808/207870648 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7c4001320 con 0x7fd7bc04a170 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 20 client.184988.objecter dump_active .. 0 homeless 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_get_version_reply(handle=1 version=711) ==== 24+0+0 (unknown 3044792690 0 0) 0x7fd7c804e7e0 con 0x7fd7d00a5d40 2024-08-19T10:09:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.064+0000 7fd7b7fff640 10 monclient: handle_get_version_reply finishing 1 version 711 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.065+0000 7fd7ceffd640 10 monclient: get_auth_request con 0x7fd7bc041b70 auth_method 0 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.065+0000 7fd7ce7fc640 10 monclient: get_auth_request con 0x7fd7bc04a170 auth_method 0 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.067+0000 7fd7b7fff640 10 client.184988.objecter ms_handle_connect 0x7fd7bc041b70 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.069+0000 7fd7b7fff640 10 client.184988.objecter ms_handle_connect 0x7fd7bc04a170 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.069+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== osd.2 v1:172.21.15.1:6808/207870648 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd7c4001320 con 0x7fd7bc04a170 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.069+0000 7fd7b7fff640 10 client.184988.objecter ms_dispatch 0x7fd7d0000b90 command_reply(tid 1: 0 ) 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.070+0000 7fd7b7fff640 10 client.184988.objecter _finish_command 1 = system:0 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.070+0000 7fd7b7fff640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 4 2024-08-19T10:09:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.070+0000 7fd7b7fff640 15 client.184988.objecter _session_command_op_remove 2 1 2024-08-19T10:09:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 10 client.184988.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T10:09:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 15 client.184988.objecter _session_command_op_assign -1 2 2024-08-19T10:09:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter _get_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter _get_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 15 client.184988.objecter _session_command_op_remove -1 2 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter get_session s=0x7fd7bc049ac0 osd=2 3 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 15 client.184988.objecter _session_command_op_assign 2 2 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 4 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 10 client.184988.objecter _send_command 2 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 --> v1:172.21.15.1:6808/207870648 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd7b00012b0 con 0x7fd7bc04a170 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7b7fff640 1 -- 172.21.15.1:0/3653949476 <== osd.2 v1:172.21.15.1:6808/207870648 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fd7b00012b0 con 0x7fd7bc04a170 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7b7fff640 10 client.184988.objecter ms_dispatch 0x7fd7d0000b90 command_reply(tid 2: 0 ) 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7b7fff640 10 client.184988.objecter _finish_command 2 = system:0 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7b7fff640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 4 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7b7fff640 15 client.184988.objecter _session_command_op_remove 2 2 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:25.091 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T10:09:25.092 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 10 librados: watch_flush enter 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 10 librados: watch_flush exit 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 10 client.184988.objecter close_session for osd.2 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.1:6808/207870648 conn(0x7fd7bc04a170 legacy=0x7fd7bc04c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter put_session s=0x7fd7bc049ac0 osd=2 1 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 20 client.184988.objecter shutdown clearing up homeless session... 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.082+0000 7fd7d5867640 10 client.184988.objecter successfully canceled tick 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.82:6812/2549104415 conn(0x7fd7bc041b70 legacy=0x7fd7bc044030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 10 monclient: shutdown 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 >> v1:172.21.15.1:6790/0 conn(0x7fd7d00a5d40 legacy=0x7fd7d01d6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7cf7fe640 1 -- 172.21.15.1:0/3653949476 reap_dead start 2024-08-19T10:09:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 shutdown_connections 2024-08-19T10:09:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 >> 172.21.15.1:0/3653949476 conn(0x7fd7d00ac6a0 msgr2=0x7fd7d006d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 shutdown_connections 2024-08-19T10:09:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 -- 172.21.15.1:0/3653949476 wait complete. 2024-08-19T10:09:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.083+0000 7fd7d5867640 1 librados: shutdown 2024-08-19T10:09:25.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-19T10:09:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.194+0000 7f29ad460640 10 monclient: get_monmap_and_config 2024-08-19T10:09:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.194+0000 7f29ad460640 10 monclient: build_initial_monmap 2024-08-19T10:09:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.194+0000 7f29ad460640 10 monclient: monmap: 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:25.195023+0000 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:25.195023+0000 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:25.197 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.194+0000 7f29ad460640 1 Processor -- start 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.194+0000 7f29ad460640 1 -- start start 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.194+0000 7f29ad460640 10 monclient: init 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient: _add_conns ranks=[2,1,5,8,3,6,0,4,7] 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient(hunting): picked mon.noname-b con 0x7f29a80a4f80 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient(hunting): picked mon.noname-f con 0x7f29a80a4650 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient(hunting): picked mon.noname-h con 0x7f29a813fcd0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a80a4a60 con 0x7f29a80a4650 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a8069270 con 0x7f29a80a4f80 2024-08-19T10:09:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a80a3c70 con 0x7f29a813fcd0 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29ad460640 10 monclient(hunting): authenticate will time out at 1821.271240s 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29a6ffd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f29a80a4f80 0x7f29a80a5360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49174/0 (socket says 172.21.15.1:49174) 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29a6ffd640 1 -- 172.21.15.1:0/3694622340 learned_addr learned my addr 172.21.15.1:0/3694622340 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.195+0000 7f29a67fc640 1 -- 172.21.15.1:0/3694622340 >> v1:172.21.15.82:6789/0 conn(0x7f29a80a4650 legacy=0x7f29a813e5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a6ffd640 1 -- 172.21.15.1:0/3694622340 >> v1:172.21.15.1:6790/0 conn(0x7f29a80a4f80 legacy=0x7f29a80a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a77fe640 1 -- 172.21.15.1:0/3694622340 >> v1:172.21.15.82:6791/0 conn(0x7f29a813fcd0 legacy=0x7f29a81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 58506978 0 0) 0x7f29a8069270 con 0x7f29a80a4f80 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): my global_id is 184997 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2994003520 con 0x7f29a80a4f80 2024-08-19T10:09:25.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503103702 0 0) 0x7f29a80a3c70 con 0x7f29a813fcd0 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient(hunting): my global_id is 214403 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a8069270 con 0x7f29a813fcd0 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1203353758 0 0) 0x7f2994003520 con 0x7f29a80a4f80 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 >> v1:172.21.15.82:6791/0 conn(0x7f29a813fcd0 legacy=0x7f29a81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 >> v1:172.21.15.82:6789/0 conn(0x7f29a80a4650 legacy=0x7f29a813e5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 monclient: found mon.noname-b 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a80a3420 con 0x7f29a80a4f80 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2998001e60 con 0x7f29a80a4f80 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.197+0000 7f29a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 10 monclient: dump: 2024-08-19T10:09:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.207 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.207 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.207 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.207 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.208 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.208 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.208 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.208 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.208 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 5 monclient: authenticate success, global_id 184997 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2998004c20 con 0x7f29a80a4f80 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 1 -- 172.21.15.1:0/3694622340 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2998005190 con 0x7f29a80a4f80 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 4 set_mon_vals no callback set 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29a5ffb640 10 monclient: dump: 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.210 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 10 monclient: shutdown 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 1 -- 172.21.15.1:0/3694622340 >> v1:172.21.15.1:6790/0 conn(0x7f29a80a4f80 legacy=0x7f29a80a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 1 -- 172.21.15.1:0/3694622340 shutdown_connections 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 1 -- 172.21.15.1:0/3694622340 >> 172.21.15.1:0/3694622340 conn(0x7f29a80a8ef0 msgr2=0x7f29a8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 1 -- 172.21.15.1:0/3694622340 shutdown_connections 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.198+0000 7f29ad460640 1 -- 172.21.15.1:0/3694622340 wait complete. 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 10 monclient: build_initial_monmap 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 10 monclient: monmap: 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 1 librados: starting msgr at 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 1 librados: starting objecter 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 1 Processor -- start 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 1 -- start start 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 1 librados: setting wanted keys 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 1 librados: calling monclient init 2024-08-19T10:09:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.199+0000 7f29ad460640 10 monclient: init 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient: _add_conns ranks=[5,1,4,8,2,3,0,6,7] 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient(hunting): picked mon.noname-h con 0x7f29a80a4650 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient(hunting): picked mon.noname-f con 0x7f29a80a4f80 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient(hunting): picked mon.noname-c con 0x7f29a813fcd0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a81ce900 con 0x7f29a80a4f80 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a80a1c50 con 0x7f29a813fcd0 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a8139580 con 0x7f29a80a4650 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29ad460640 10 monclient(hunting): authenticate will time out at 1821.276198s 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29a67fc640 1 -- >> v1:172.21.15.82:6789/0 conn(0x7f29a80a4f80 legacy=0x7f29a81d3220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29a77fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f29a813fcd0 0x7f29a81d6950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48446/0 (socket says 172.21.15.1:48446) 2024-08-19T10:09:25.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.200+0000 7f29a77fe640 1 -- 172.21.15.1:0/56249232 learned_addr learned my addr 172.21.15.1:0/56249232 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f29a77fe640 1 -- 172.21.15.1:0/56249232 >> v1:172.21.15.1:6791/0 conn(0x7f29a813fcd0 legacy=0x7f29a81d6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 client.?.objecter ms_handle_connect 0x7f29a813fcd0 2024-08-19T10:09:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 client.?.objecter ms_handle_connect 0x7f29a80a4650 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f29a6ffd640 1 -- 172.21.15.1:0/56249232 >> v1:172.21.15.82:6791/0 conn(0x7f29a80a4650 legacy=0x7f29a813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258350613 0 0) 0x7f29a80a1c50 con 0x7f29a813fcd0 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): my global_id is 194836 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2988004eb0 con 0x7f29a813fcd0 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1803777491 0 0) 0x7f2988004eb0 con 0x7f29a813fcd0 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29a80a1c50 con 0x7f29a813fcd0 2024-08-19T10:09:25.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29a0003610 con 0x7f29a813fcd0 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): dump: 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614574278 0 0) 0x7f29a8139580 con 0x7f29a80a4650 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 10 monclient(hunting): my global_id is 214412 2024-08-19T10:09:25.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2988004eb0 con 0x7f29a80a4650 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 787591125 0 0) 0x7f29a80a1c50 con 0x7f29a813fcd0 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.202+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 >> v1:172.21.15.82:6791/0 conn(0x7f29a80a4650 legacy=0x7f29a813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 >> v1:172.21.15.82:6789/0 conn(0x7f29a80a4f80 legacy=0x7f29a81d3220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 1 monclient: found mon.c 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a8139860 con 0x7f29a813fcd0 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 5 monclient: authenticate success, global_id 194836 2024-08-19T10:09:25.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 monclient: _renew_subs 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29a813f9a0 con 0x7f29a813fcd0 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 client.194836.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 monclient: _renew_subs 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f29a81ce110 con 0x7f29a813fcd0 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 1 librados: init done 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29a77fe640 1 --1- 172.21.15.1:0/56249232 >> v1:172.21.15.1:6791/0 conn(0x7f29a813fcd0 0x7f29a81d6950 :-1 s=OPENED pgs=3933 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f29a0003b50 config(0 keys) 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29a0003b50 con 0x7f29a813fcd0 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29a57fa640 4 set_mon_vals no callback set 2024-08-19T10:09:25.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f29a0004040 con 0x7f29a813fcd0 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.203+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.1:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298c005080 con 0x7f29a813fcd0 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.204+0000 7f299f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.204+0000 7f299f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.204+0000 7f299f7fe640 10 monclient: dump: 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:25.224 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:25.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.204+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f29a00041e0 con 0x7f29a813fcd0 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.204+0000 7f29a6ffd640 10 monclient: get_auth_request con 0x7f2988040e80 auth_method 0 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.206+0000 7f299f7fe640 10 client.194836.objecter ms_handle_connect 0x7f2988040e80 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.208+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f29a004e070 con 0x7f29a813fcd0 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.208+0000 7f299f7fe640 10 client.194836.objecter ms_dispatch 0x7f29a8000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.208+0000 7f299f7fe640 3 client.194836.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.208+0000 7f299f7fe640 3 client.194836.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:25.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:25.208+0000 7f299f7fe640 20 client.194836.objecter dump_active .. 0 homeless 2024-08-19T10:09:26.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.032+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f29a0017670 con 0x7f29a813fcd0 2024-08-19T10:09:26.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.032+0000 7f299f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.032+0000 7f299f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:26.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.223+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f298c000b90 con 0x7f2988040e80 2024-08-19T10:09:26.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.224+0000 7f299f7fe640 1 -- 172.21.15.1:0/56249232 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f298c000b90 con 0x7f2988040e80 2024-08-19T10:09:26.225 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 10 librados: watch_flush enter 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 10 librados: watch_flush exit 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 20 client.194836.objecter shutdown clearing up homeless session... 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 10 client.194836.objecter successfully canceled tick 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2988040e80 legacy=0x7f2988043340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 10 monclient: shutdown 2024-08-19T10:09:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 >> v1:172.21.15.1:6791/0 conn(0x7f29a813fcd0 legacy=0x7f29a81d6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 shutdown_connections 2024-08-19T10:09:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 >> 172.21.15.1:0/56249232 conn(0x7f29a80a8ef0 msgr2=0x7f29a8136ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 shutdown_connections 2024-08-19T10:09:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.226+0000 7f29ad460640 1 -- 172.21.15.1:0/56249232 wait complete. 2024-08-19T10:09:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.227+0000 7f29ad460640 1 librados: shutdown 2024-08-19T10:09:26.238 INFO:tasks.ceph:Scrubbing osd.3 2024-08-19T10:09:26.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.340+0000 7fcbf7be2640 10 monclient: get_monmap_and_config 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.340+0000 7fcbf7be2640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.340+0000 7fcbf7be2640 10 monclient: monmap: 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:26.341640+0000 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:26.341640+0000 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.343 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.341+0000 7fcbf7be2640 1 Processor -- start 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.341+0000 7fcbf7be2640 1 -- start start 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.341+0000 7fcbf7be2640 10 monclient: init 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.341+0000 7fcbf7be2640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.341+0000 7fcbf7be2640 10 monclient: _add_conns ranks=[4,3,7,0,2,5,1,6,8] 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.341+0000 7fcbf7be2640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf00a66b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 10 monclient(hunting): picked mon.noname-g con 0x7fcbf00a71a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 10 monclient(hunting): picked mon.noname-i con 0x7fcbf00a3f10 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf00a2e80 con 0x7fcbf00a71a0 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf006eea0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf00a7f90 con 0x7fcbf00a3f10 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf7be2640 10 monclient(hunting): authenticate will time out at 1822.418121s 2024-08-19T10:09:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf597e640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fcbf00a66b0 0x7fcbf00a6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48456/0 (socket says 172.21.15.1:48456) 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.342+0000 7fcbf597e640 1 -- 172.21.15.1:0/1664713685 learned_addr learned my addr 172.21.15.1:0/1664713685 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf597e640 1 -- 172.21.15.1:0/1664713685 >> v1:172.21.15.1:6791/0 conn(0x7fcbf00a66b0 legacy=0x7fcbf00a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460576456 0 0) 0x7fcbf006eea0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf517d640 1 -- 172.21.15.1:0/1664713685 >> v1:172.21.15.82:6790/0 conn(0x7fcbf00a71a0 legacy=0x7fcbf013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): my global_id is 194845 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf617f640 1 -- 172.21.15.1:0/1664713685 >> v1:172.21.15.82:6792/0 conn(0x7fcbf00a3f10 legacy=0x7fcbf0141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbdc003520 con 0x7fcbf00a66b0 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516354016 0 0) 0x7fcbe8001750 con 0x7fcbf00a3f10 2024-08-19T10:09:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): my global_id is 204604 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf006eea0 con 0x7fcbf00a3f10 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3276686071 0 0) 0x7fcbf00a2e80 con 0x7fcbf00a71a0 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient(hunting): my global_id is 214374 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe8001750 con 0x7fcbf00a71a0 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2398118785 0 0) 0x7fcbdc003520 con 0x7fcbf00a66b0 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 >> v1:172.21.15.82:6792/0 conn(0x7fcbf00a3f10 legacy=0x7fcbf0141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 >> v1:172.21.15.82:6790/0 conn(0x7fcbf00a71a0 legacy=0x7fcbf013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 monclient: found mon.noname-c 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf00a3a10 con 0x7fcbf00a66b0 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbec001ae0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf7be2640 5 monclient: authenticate success, global_id 194845 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf7be2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: dump: 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.353 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf7be2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbec001d30 con 0x7fcbf00a66b0 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 1 -- 172.21.15.1:0/1664713685 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbec004be0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf7be2640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf7be2640 4 set_mon_vals no callback set 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.344+0000 7fcbf497c640 10 monclient: dump: 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 10 monclient: shutdown 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 1 -- 172.21.15.1:0/1664713685 >> v1:172.21.15.1:6791/0 conn(0x7fcbf00a66b0 legacy=0x7fcbf00a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 1 -- 172.21.15.1:0/1664713685 shutdown_connections 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 1 -- 172.21.15.1:0/1664713685 >> 172.21.15.1:0/1664713685 conn(0x7fcbf00ac340 msgr2=0x7fcbf00a87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 1 -- 172.21.15.1:0/1664713685 shutdown_connections 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.345+0000 7fcbf7be2640 1 -- 172.21.15.1:0/1664713685 wait complete. 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient: monmap: 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.356 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.357 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 librados: starting msgr at 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 librados: starting objecter 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 Processor -- start 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 -- start start 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 librados: setting wanted keys 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 librados: calling monclient init 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient: init 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient: _add_conns ranks=[5,2,4,3,6,8,7,0,1] 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient(hunting): picked mon.noname-h con 0x7fcbf00a3f10 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf00a66b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf00a71a0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:26.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf01cea10 con 0x7fcbf00a66b0 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf00a2740 con 0x7fcbf00a71a0 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf01270b0 con 0x7fcbf00a3f10 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.346+0000 7fcbf7be2640 10 monclient(hunting): authenticate will time out at 1822.422953s 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.347+0000 7fcbf617f640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fcbf00a71a0 0x7fcbf01d6560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48468/0 (socket says 172.21.15.1:48468) 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.347+0000 7fcbf617f640 1 -- 172.21.15.1:0/2813872360 learned_addr learned my addr 172.21.15.1:0/2813872360 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.347+0000 7fcbf517d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fcbf00a66b0 0x7fcbf01d2e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49180/0 (socket says 172.21.15.1:49180) 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf517d640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.1:6790/0 conn(0x7fcbf00a66b0 legacy=0x7fcbf01d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf617f640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.1:6791/0 conn(0x7fcbf00a71a0 legacy=0x7fcbf01d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 client.?.objecter ms_handle_connect 0x7fcbf00a66b0 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 client.?.objecter ms_handle_connect 0x7fcbf00a71a0 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 client.?.objecter ms_handle_connect 0x7fcbf00a3f10 2024-08-19T10:09:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf597e640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.82:6791/0 conn(0x7fcbf00a3f10 legacy=0x7fcbf01399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501857410 0 0) 0x7fcbf006eea0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): my global_id is 194854 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbd4004ee0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970106458 0 0) 0x7fcbf01cea10 con 0x7fcbf00a66b0 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): my global_id is 185006 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf006eea0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801495277 0 0) 0x7fcbf01270b0 con 0x7fcbf00a3f10 2024-08-19T10:09:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): my global_id is 214421 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf01cea10 con 0x7fcbf00a3f10 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1564442482 0 0) 0x7fcbd4004ee0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbf01270b0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbe8003cd0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): dump: 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1949908245 0 0) 0x7fcbf006eea0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbd4004ee0 con 0x7fcbf00a66b0 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbe0001c70 con 0x7fcbf00a66b0 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): dump: 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1321411796 0 0) 0x7fcbf01cea10 con 0x7fcbf00a3f10 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbf006eea0 con 0x7fcbf00a3f10 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbec004b20 con 0x7fcbf00a3f10 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient(hunting): dump: 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.370 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3806072366 0 0) 0x7fcbf01270b0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.82:6791/0 conn(0x7fcbf00a3f10 legacy=0x7fcbf01399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.1:6790/0 conn(0x7fcbf00a66b0 legacy=0x7fcbf01d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 monclient: found mon.c 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf0127390 con 0x7fcbf00a71a0 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 5 monclient: authenticate success, global_id 194854 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 10 monclient: _renew_subs 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf0126150 con 0x7fcbf00a71a0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 10 client.194854.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 10 monclient: _renew_subs 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fcbf0125b90 con 0x7fcbf00a71a0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.349+0000 7fcbf7be2640 1 librados: init done 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbe8001fe0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcbe8004510 con 0x7fcbf00a71a0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe7fff640 4 set_mon_vals no callback set 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 10 monclient: dump: 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.373 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbf7be2640 10 client.194854.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbf7be2640 15 client.194854.objecter _session_command_op_assign -1 1 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbf7be2640 20 client.194854.objecter _get_session osd=-1 returning homeless 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbf7be2640 10 client.194854.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbf7be2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.1:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fcbd8002910 con 0x7fcbf00a71a0 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.350+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcbe80046b0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fcbe804e3a0 con 0x7fcbf00a71a0 2024-08-19T10:09:26.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 10 client.194854.objecter ms_dispatch 0x7fcbf0000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 3 client.194854.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 3 client.194854.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 10 client.194854.objecter checking command 1 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 20 client.194854.objecter _get_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 15 client.194854.objecter _session_command_op_remove -1 1 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 20 client.194854.objecter _get_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 20 client.194854.objecter get_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 15 client.194854.objecter _session_command_op_assign 3 1 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 4 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 10 client.194854.objecter _send_command 1 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.82:6804/394826682 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbd404dce0 con 0x7fcbd4049fd0 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 20 client.194854.objecter dump_active .. 0 homeless 2024-08-19T10:09:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_get_version_reply(handle=1 version=711) ==== 24+0+0 (unknown 3044792690 0 0) 0x7fcbe804e810 con 0x7fcbf00a71a0 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbe67fc640 10 monclient: handle_get_version_reply finishing 1 version 711 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbf597e640 10 monclient: get_auth_request con 0x7fcbd4041980 auth_method 0 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.351+0000 7fcbf517d640 10 monclient: get_auth_request con 0x7fcbd4049fd0 auth_method 0 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.353+0000 7fcbe67fc640 10 client.194854.objecter ms_handle_connect 0x7fcbd4041980 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.356+0000 7fcbe67fc640 10 client.194854.objecter ms_handle_connect 0x7fcbd4049fd0 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.356+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== osd.3 v1:172.21.15.82:6804/394826682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcbd404dce0 con 0x7fcbd4049fd0 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.356+0000 7fcbe67fc640 10 client.194854.objecter ms_dispatch 0x7fcbf0000b90 command_reply(tid 1: 0 ) 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.356+0000 7fcbe67fc640 10 client.194854.objecter _finish_command 1 = system:0 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.356+0000 7fcbe67fc640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 4 2024-08-19T10:09:26.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.356+0000 7fcbe67fc640 15 client.194854.objecter _session_command_op_remove 3 1 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 10 client.194854.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 15 client.194854.objecter _session_command_op_assign -1 2 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 20 client.194854.objecter _get_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 20 client.194854.objecter _get_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 15 client.194854.objecter _session_command_op_remove -1 2 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 20 client.194854.objecter get_session s=0x7fcbd4049920 osd=3 3 2024-08-19T10:09:26.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 15 client.194854.objecter _session_command_op_assign 3 2 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 4 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 10 client.194854.objecter _send_command 2 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.369+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 --> v1:172.21.15.82:6804/394826682 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcbd80012b0 con 0x7fcbd4049fd0 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbe67fc640 1 -- 172.21.15.1:0/2813872360 <== osd.3 v1:172.21.15.82:6804/394826682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fcbd80012b0 con 0x7fcbd4049fd0 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbe67fc640 10 client.194854.objecter ms_dispatch 0x7fcbf0000b90 command_reply(tid 2: 0 ) 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbe67fc640 10 client.194854.objecter _finish_command 2 = system:0 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbe67fc640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 4 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbe67fc640 15 client.194854.objecter _session_command_op_remove 3 2 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:26.378 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T10:09:26.379 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 10 librados: watch_flush enter 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 10 librados: watch_flush exit 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 10 client.194854.objecter close_session for osd.3 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.82:6804/394826682 conn(0x7fcbd4049fd0 legacy=0x7fcbd404c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 20 client.194854.objecter put_session s=0x7fcbd4049920 osd=3 1 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 20 client.194854.objecter shutdown clearing up homeless session... 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 10 client.194854.objecter successfully canceled tick 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcbd4041980 legacy=0x7fcbd4043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 10 monclient: shutdown 2024-08-19T10:09:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 >> v1:172.21.15.1:6791/0 conn(0x7fcbf00a71a0 legacy=0x7fcbf01d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf617f640 1 -- 172.21.15.1:0/2813872360 reap_dead start 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 shutdown_connections 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 >> 172.21.15.1:0/2813872360 conn(0x7fcbf00ac340 msgr2=0x7fcbf00aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 shutdown_connections 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 -- 172.21.15.1:0/2813872360 wait complete. 2024-08-19T10:09:26.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.370+0000 7fcbf7be2640 1 librados: shutdown 2024-08-19T10:09:26.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.480+0000 7f2d970b3640 10 monclient: get_monmap_and_config 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.480+0000 7f2d970b3640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.480+0000 7f2d970b3640 10 monclient: monmap: 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:26.481829+0000 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:26.481829+0000 2024-08-19T10:09:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.481+0000 7f2d970b3640 1 Processor -- start 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.481+0000 7f2d970b3640 1 -- start start 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.481+0000 7f2d970b3640 10 monclient: init 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient: _add_conns ranks=[7,5,2,8,4,0,3,1,6] 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient(hunting): picked mon.noname-i con 0x7f2d900a6720 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient(hunting): picked mon.noname-h con 0x7f2d900a7210 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient(hunting): picked mon.noname-b con 0x7f2d900a4500 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9006ed50 con 0x7f2d900a4500 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d900a3b20 con 0x7f2d900a7210 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9006f190 con 0x7f2d900a6720 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d970b3640 10 monclient(hunting): authenticate will time out at 1822.558227s 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d95650640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f2d900a4500 0x7f2d90141ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:49194/0 (socket says 172.21.15.1:49194) 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.482+0000 7f2d95650640 1 -- 172.21.15.1:0/3174857238 learned_addr learned my addr 172.21.15.1:0/3174857238 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d95650640 1 -- 172.21.15.1:0/3174857238 >> v1:172.21.15.1:6790/0 conn(0x7f2d900a4500 legacy=0x7f2d90141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8ffff640 1 -- 172.21.15.1:0/3174857238 >> v1:172.21.15.82:6791/0 conn(0x7f2d900a7210 legacy=0x7f2d9013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d94e4f640 1 -- 172.21.15.1:0/3174857238 >> v1:172.21.15.82:6792/0 conn(0x7f2d900a6720 legacy=0x7f2d900a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 715897673 0 0) 0x7f2d9006ed50 con 0x7f2d900a4500 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): my global_id is 185015 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d7c003520 con 0x7f2d900a4500 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3905066001 0 0) 0x7f2d900a3b20 con 0x7f2d900a7210 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): my global_id is 214430 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d9006ed50 con 0x7f2d900a7210 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997563302 0 0) 0x7f2d9006f190 con 0x7f2d900a6720 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 10 monclient(hunting): my global_id is 204613 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.484+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d900a3b20 con 0x7f2d900a6720 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2823390219 0 0) 0x7f2d7c003520 con 0x7f2d900a4500 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 >> v1:172.21.15.82:6792/0 conn(0x7f2d900a6720 legacy=0x7f2d900a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 >> v1:172.21.15.82:6791/0 conn(0x7f2d900a7210 legacy=0x7f2d9013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 monclient: found mon.noname-b 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d900a9260 con 0x7f2d900a4500 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 5 monclient: authenticate success, global_id 185015 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d78001ae0 con 0x7f2d900a4500 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: dump: 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d780048a0 con 0x7f2d900a4500 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 1 -- 172.21.15.1:0/3174857238 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d78004e10 con 0x7f2d900a4500 2024-08-19T10:09:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d8f7fe640 10 monclient: dump: 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 4 set_mon_vals no callback set 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 10 monclient: shutdown 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 1 -- 172.21.15.1:0/3174857238 >> v1:172.21.15.1:6790/0 conn(0x7f2d900a4500 legacy=0x7f2d90141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 1 -- 172.21.15.1:0/3174857238 shutdown_connections 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 1 -- 172.21.15.1:0/3174857238 >> 172.21.15.1:0/3174857238 conn(0x7f2d900ac440 msgr2=0x7f2d900a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 1 -- 172.21.15.1:0/3174857238 shutdown_connections 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.485+0000 7f2d970b3640 1 -- 172.21.15.1:0/3174857238 wait complete. 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 10 monclient: monmap: 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 1 librados: starting msgr at 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 1 librados: starting objecter 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 1 Processor -- start 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 1 -- start start 2024-08-19T10:09:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 1 librados: setting wanted keys 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 1 librados: calling monclient init 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.486+0000 7f2d970b3640 10 monclient: init 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient: _add_conns ranks=[4,0,6,3,1,2,8,5,7] 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2d900a4500 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient(hunting): picked mon.noname-a con 0x7f2d900a6720 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient(hunting): picked mon.noname-d con 0x7f2d900a7210 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d90129790 con 0x7f2d900a6720 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d900a2610 con 0x7f2d900a4500 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d90127430 con 0x7f2d900a7210 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d970b3640 10 monclient(hunting): authenticate will time out at 1822.563428s 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d94e4f640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f2d900a4500 0x7f2d90141e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:48476/0 (socket says 172.21.15.1:48476) 2024-08-19T10:09:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.487+0000 7f2d94e4f640 1 -- 172.21.15.1:0/4145856226 learned_addr learned my addr 172.21.15.1:0/4145856226 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d94e4f640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.1:6791/0 conn(0x7f2d900a4500 legacy=0x7f2d90141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8ffff640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.1:6789/0 conn(0x7f2d900a6720 legacy=0x7f2d901d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d900a4500 2024-08-19T10:09:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d900a6720 2024-08-19T10:09:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d900a7210 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d95650640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.1:6792/0 conn(0x7f2d900a7210 legacy=0x7f2d901d67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109987481 0 0) 0x7f2d90129790 con 0x7f2d900a6720 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): my global_id is 204633 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d70004eb0 con 0x7f2d900a6720 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261587941 0 0) 0x7f2d900a2610 con 0x7f2d900a4500 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): my global_id is 194863 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d90129790 con 0x7f2d900a4500 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958354207 0 0) 0x7f2d90127430 con 0x7f2d900a7210 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 10 monclient(hunting): my global_id is 204504 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.489+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d900a2610 con 0x7f2d900a7210 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2061686901 0 0) 0x7f2d70004eb0 con 0x7f2d900a6720 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d90127430 con 0x7f2d900a6720 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d800020b0 con 0x7f2d900a6720 2024-08-19T10:09:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): dump: 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2237283488 0 0) 0x7f2d900a2610 con 0x7f2d900a7210 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d70004eb0 con 0x7f2d900a7210 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1190311628 0 0) 0x7f2d90129790 con 0x7f2d900a4500 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d900a2610 con 0x7f2d900a4500 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d78004d90 con 0x7f2d900a7210 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): dump: 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.509 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d88003610 con 0x7f2d900a4500 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient(hunting): dump: 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1672472047 0 0) 0x7f2d90127430 con 0x7f2d900a6720 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.1:6792/0 conn(0x7f2d900a7210 legacy=0x7f2d901d67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.1:6791/0 conn(0x7f2d900a4500 legacy=0x7f2d90141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 monclient: found mon.a 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d90128ef0 con 0x7f2d900a6720 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 5 monclient: authenticate success, global_id 204633 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 10 monclient: _renew_subs 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d90142540 con 0x7f2d900a6720 2024-08-19T10:09:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 10 client.204633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 10 monclient: _renew_subs 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d90128170 con 0x7f2d900a6720 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d970b3640 1 librados: init done 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d80003df0 con 0x7f2d900a6720 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f2d80004320 con 0x7f2d900a6720 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8effd640 4 set_mon_vals no callback set 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.490+0000 7f2d8d7fa640 10 monclient: dump: 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.514 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f2d800044c0 con 0x7f2d900a6720 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d970b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d970b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d970b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d74005080 con 0x7f2d900a6720 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f2d8004e250 con 0x7f2d900a6720 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d8d7fa640 10 client.204633.objecter ms_dispatch 0x7f2d90000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d8d7fa640 3 client.204633.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d8d7fa640 3 client.204633.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d8d7fa640 20 client.204633.objecter dump_active .. 0 homeless 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.491+0000 7f2d94e4f640 10 monclient: get_auth_request con 0x7f2d70041ba0 auth_method 0 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.494+0000 7f2d8d7fa640 10 client.204633.objecter ms_handle_connect 0x7f2d70041ba0 2024-08-19T10:09:26.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.496+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f2d800178b0 con 0x7f2d900a6720 2024-08-19T10:09:26.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.496+0000 7f2d8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.496+0000 7f2d8d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.681+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f2d74000b90 con 0x7f2d70041ba0 2024-08-19T10:09:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.681+0000 7f2d8d7fa640 1 -- 172.21.15.1:0/4145856226 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7f2d74000b90 con 0x7f2d70041ba0 2024-08-19T10:09:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 10 librados: watch_flush enter 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 10 librados: watch_flush exit 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 20 client.204633.objecter shutdown clearing up homeless session... 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 10 client.204633.objecter successfully canceled tick 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.82:6812/2549104415 conn(0x7f2d70041ba0 legacy=0x7f2d70044060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 10 monclient: shutdown 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 >> v1:172.21.15.1:6789/0 conn(0x7f2d900a6720 legacy=0x7f2d901d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 shutdown_connections 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 >> 172.21.15.1:0/4145856226 conn(0x7f2d900ac440 msgr2=0x7f2d90142cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 shutdown_connections 2024-08-19T10:09:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 -- 172.21.15.1:0/4145856226 wait complete. 2024-08-19T10:09:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.684+0000 7f2d970b3640 1 librados: shutdown 2024-08-19T10:09:26.696 INFO:tasks.ceph:Scrubbing osd.4 2024-08-19T10:09:26.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-08-19T10:09:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.798+0000 7f7bcd97f640 10 monclient: get_monmap_and_config 2024-08-19T10:09:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.798+0000 7f7bcd97f640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.798+0000 7f7bcd97f640 10 monclient: monmap: 2024-08-19T10:09:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:26.799180+0000 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:26.799180+0000 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.801 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.798+0000 7f7bcd97f640 1 Processor -- start 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.798+0000 7f7bcd97f640 1 -- start start 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.798+0000 7f7bcd97f640 10 monclient: init 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient: _add_conns ranks=[5,4,3,7,1,0,8,6,2] 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient(hunting): picked mon.noname-h con 0x7f7bc80a6720 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc80a7210 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient(hunting): picked mon.noname-g con 0x7f7bc80a4570 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:26.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc80a3b60 con 0x7f7bc80a4570 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc8069210 con 0x7f7bc80a7210 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc80a32c0 con 0x7f7bc80a6720 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bcd97f640 10 monclient(hunting): authenticate will time out at 1822.875463s 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bc67fc640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f7bc80a7210 0x7f7bc813e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:33600/0 (socket says 172.21.15.1:33600) 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.799+0000 7f7bc67fc640 1 -- 172.21.15.1:0/2104483219 learned_addr learned my addr 172.21.15.1:0/2104483219 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc67fc640 1 -- 172.21.15.1:0/2104483219 >> v1:172.21.15.1:6791/0 conn(0x7f7bc80a7210 legacy=0x7f7bc813e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740174689 0 0) 0x7f7bc8069210 con 0x7f7bc80a7210 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc5ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc5ffb640 10 monclient(hunting): my global_id is 194872 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.801+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bb4003520 con 0x7f7bc80a7210 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc6ffd640 1 -- 172.21.15.1:0/2104483219 >> v1:172.21.15.82:6791/0 conn(0x7f7bc80a6720 legacy=0x7f7bc80a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc77fe640 1 -- 172.21.15.1:0/2104483219 >> v1:172.21.15.82:6790/0 conn(0x7f7bc80a4570 legacy=0x7f7bc8142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1666040438 0 0) 0x7f7bb4003520 con 0x7f7bc80a7210 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 >> v1:172.21.15.82:6791/0 conn(0x7f7bc80a6720 legacy=0x7f7bc80a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 >> v1:172.21.15.82:6790/0 conn(0x7f7bc80a4570 legacy=0x7f7bc8142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 monclient: found mon.noname-c 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc8069610 con 0x7f7bc80a7210 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7bb0001ae0 con 0x7f7bc80a7210 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: dump: 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.814 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 5 monclient: authenticate success, global_id 194872 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7bb00048a0 con 0x7f7bc80a7210 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 1 -- 172.21.15.1:0/2104483219 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7bb0004e10 con 0x7f7bc80a7210 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 4 set_mon_vals no callback set 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bc5ffb640 10 monclient: dump: 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 10 monclient: shutdown 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 1 -- 172.21.15.1:0/2104483219 >> v1:172.21.15.1:6791/0 conn(0x7f7bc80a7210 legacy=0x7f7bc813e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 1 -- 172.21.15.1:0/2104483219 shutdown_connections 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 1 -- 172.21.15.1:0/2104483219 >> 172.21.15.1:0/2104483219 conn(0x7f7bc80a8e50 msgr2=0x7f7bc8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 1 -- 172.21.15.1:0/2104483219 shutdown_connections 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.802+0000 7f7bcd97f640 1 -- 172.21.15.1:0/2104483219 wait complete. 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 10 monclient: monmap: 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:26.818 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 1 librados: starting msgr at 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 1 librados: starting objecter 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 1 Processor -- start 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 1 -- start start 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 1 librados: setting wanted keys 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 1 librados: calling monclient init 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.803+0000 7f7bcd97f640 10 monclient: init 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient: _add_conns ranks=[4,7,8,1,2,6,3,0,5] 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc80a4570 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient(hunting): picked mon.noname-i con 0x7f7bc80a6720 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient(hunting): picked mon.noname-e con 0x7f7bc80a7210 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc80ab680 con 0x7f7bc80a4570 2024-08-19T10:09:26.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc81cf350 con 0x7f7bc80a6720 2024-08-19T10:09:26.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc813ed20 con 0x7f7bc80a7210 2024-08-19T10:09:26.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bcd97f640 10 monclient(hunting): authenticate will time out at 1822.880497s 2024-08-19T10:09:26.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bc6ffd640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f7bc80a4570 0x7f7bc81ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:33602/0 (socket says 172.21.15.1:33602) 2024-08-19T10:09:26.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.804+0000 7f7bc6ffd640 1 -- 172.21.15.1:0/661763819 learned_addr learned my addr 172.21.15.1:0/661763819 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bc77fe640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.1:6793/0 conn(0x7f7bc80a7210 legacy=0x7f7bc81d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 client.?.objecter ms_handle_connect 0x7f7bc80a7210 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 client.?.objecter ms_handle_connect 0x7f7bc80a4570 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bc6ffd640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.1:6791/0 conn(0x7f7bc80a4570 legacy=0x7f7bc81ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1329113555 0 0) 0x7f7bc80ab680 con 0x7f7bc80a4570 2024-08-19T10:09:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): my global_id is 194881 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ba8004ee0 con 0x7f7bc80a4570 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bc67fc640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.82:6792/0 conn(0x7f7bc80a6720 legacy=0x7f7bc81cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 client.?.objecter ms_handle_connect 0x7f7bc80a6720 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319100480 0 0) 0x7f7bc813ed20 con 0x7f7bc80a7210 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 10 monclient(hunting): my global_id is 204677 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.806+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bc80ab680 con 0x7f7bc80a7210 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1287099437 0 0) 0x7f7bc81cf350 con 0x7f7bc80a6720 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): my global_id is 204622 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bc813ed20 con 0x7f7bc80a6720 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3736642891 0 0) 0x7f7ba8004ee0 con 0x7f7bc80a4570 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7bc81cf350 con 0x7f7bc80a4570 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7bb8003610 con 0x7f7bc80a4570 2024-08-19T10:09:26.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): dump: 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 111711746 0 0) 0x7f7bc80ab680 con 0x7f7bc80a7210 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ba8004ee0 con 0x7f7bc80a7210 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7bc0002000 con 0x7f7bc80a7210 2024-08-19T10:09:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient(hunting): dump: 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3205152878 0 0) 0x7f7bc81cf350 con 0x7f7bc80a4570 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.1:6793/0 conn(0x7f7bc80a7210 legacy=0x7f7bc81d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.82:6792/0 conn(0x7f7bc80a6720 legacy=0x7f7bc81cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 monclient: found mon.c 2024-08-19T10:09:26.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc8141080 con 0x7f7bc80a4570 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 5 monclient: authenticate success, global_id 194881 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 10 monclient: _renew_subs 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc81407e0 con 0x7f7bc80a4570 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 10 client.194881.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 10 monclient: _renew_subs 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f7bc8140420 con 0x7f7bc80a4570 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bcd97f640 1 librados: init done 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7bb8003b90 con 0x7f7bc80a4570 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f7bb80040c0 con 0x7f7bc80a4570 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bc57fa640 4 set_mon_vals no callback set 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.807+0000 7f7bbffff640 10 monclient: dump: 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.833 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bcd97f640 10 client.194881.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bcd97f640 15 client.194881.objecter _session_command_op_assign -1 1 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bcd97f640 20 client.194881.objecter _get_session osd=-1 returning homeless 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bcd97f640 10 client.194881.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bcd97f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.1:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7bac002910 con 0x7f7bc80a4570 2024-08-19T10:09:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f7bb8004260 con 0x7f7bc80a4570 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f7bb804dfd0 con 0x7f7bc80a4570 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 10 client.194881.objecter ms_dispatch 0x7f7bc8000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 3 client.194881.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 3 client.194881.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 10 client.194881.objecter checking command 1 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 20 client.194881.objecter _get_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 15 client.194881.objecter _session_command_op_remove -1 1 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 20 client.194881.objecter _get_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 20 client.194881.objecter get_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 15 client.194881.objecter _session_command_op_assign 4 1 2024-08-19T10:09:26.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 4 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 10 client.194881.objecter _send_command 1 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.82:6800/2865329823 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bb0001320 con 0x7f7ba8049d20 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 20 client.194881.objecter dump_active .. 0 homeless 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== mon.4 v1:172.21.15.1:6791/0 9 ==== mon_get_version_reply(handle=1 version=711) ==== 24+0+0 (unknown 3044792690 0 0) 0x7f7bb804e440 con 0x7f7bc80a4570 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bbffff640 10 monclient: handle_get_version_reply finishing 1 version 711 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bc67fc640 10 monclient: get_auth_request con 0x7f7ba80416f0 auth_method 0 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.808+0000 7f7bc77fe640 10 monclient: get_auth_request con 0x7f7ba8049d20 auth_method 0 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.811+0000 7f7bbffff640 10 client.194881.objecter ms_handle_connect 0x7f7ba80416f0 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.813+0000 7f7bbffff640 10 client.194881.objecter ms_handle_connect 0x7f7ba8049d20 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.814+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== osd.4 v1:172.21.15.82:6800/2865329823 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bb0001320 con 0x7f7ba8049d20 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.814+0000 7f7bbffff640 10 client.194881.objecter ms_dispatch 0x7f7bc8000b90 command_reply(tid 1: 0 ) 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.814+0000 7f7bbffff640 10 client.194881.objecter _finish_command 1 = system:0 2024-08-19T10:09:26.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.814+0000 7f7bbffff640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 4 2024-08-19T10:09:26.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.814+0000 7f7bbffff640 15 client.194881.objecter _session_command_op_remove 4 1 2024-08-19T10:09:26.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 10 client.194881.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T10:09:26.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 15 client.194881.objecter _session_command_op_assign -1 2 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 20 client.194881.objecter _get_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 20 client.194881.objecter _get_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 15 client.194881.objecter _session_command_op_remove -1 2 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 20 client.194881.objecter get_session s=0x7f7ba8049670 osd=4 3 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 15 client.194881.objecter _session_command_op_assign 4 2 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 4 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 10 client.194881.objecter _send_command 2 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 --> v1:172.21.15.82:6800/2865329823 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7bac0012b0 con 0x7f7ba8049d20 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bbffff640 1 -- 172.21.15.1:0/661763819 <== osd.4 v1:172.21.15.82:6800/2865329823 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f7bac0012b0 con 0x7f7ba8049d20 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bbffff640 10 client.194881.objecter ms_dispatch 0x7f7bc8000b90 command_reply(tid 2: 0 ) 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bbffff640 10 client.194881.objecter _finish_command 2 = system:0 2024-08-19T10:09:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bbffff640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 4 2024-08-19T10:09:26.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.827+0000 7f7bbffff640 15 client.194881.objecter _session_command_op_remove 4 2 2024-08-19T10:09:26.840 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:26.840 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T10:09:26.840 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 10 librados: watch_flush enter 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 10 librados: watch_flush exit 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 10 client.194881.objecter close_session for osd.4 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.82:6800/2865329823 conn(0x7f7ba8049d20 legacy=0x7f7ba804c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 20 client.194881.objecter put_session s=0x7f7ba8049670 osd=4 1 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 20 client.194881.objecter shutdown clearing up homeless session... 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 10 client.194881.objecter successfully canceled tick 2024-08-19T10:09:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.82:6812/2549104415 conn(0x7f7ba80416f0 legacy=0x7f7ba8043bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 10 monclient: shutdown 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 >> v1:172.21.15.1:6791/0 conn(0x7f7bc80a4570 legacy=0x7f7bc81ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bc77fe640 1 -- 172.21.15.1:0/661763819 reap_dead start 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 shutdown_connections 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 >> 172.21.15.1:0/661763819 conn(0x7f7bc80a8e50 msgr2=0x7f7bc80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 shutdown_connections 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 -- 172.21.15.1:0/661763819 wait complete. 2024-08-19T10:09:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.828+0000 7f7bcd97f640 1 librados: shutdown 2024-08-19T10:09:26.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-08-19T10:09:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.942+0000 7f36477c9640 10 monclient: get_monmap_and_config 2024-08-19T10:09:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.942+0000 7f36477c9640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.943+0000 7f36477c9640 10 monclient: monmap: 2024-08-19T10:09:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:26.943907+0000 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:26.943907+0000 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.945 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.943+0000 7f36477c9640 1 Processor -- start 2024-08-19T10:09:26.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.943+0000 7f36477c9640 1 -- start start 2024-08-19T10:09:26.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.943+0000 7f36477c9640 10 monclient: init 2024-08-19T10:09:26.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient: _add_conns ranks=[2,5,4,1,0,6,3,8,7] 2024-08-19T10:09:26.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient(hunting): picked mon.noname-b con 0x7f36400a6720 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient(hunting): picked mon.noname-h con 0x7f36400a7210 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient(hunting): picked mon.noname-c con 0x7f36400a4500 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364006ed50 con 0x7f36400a6720 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36400a3b20 con 0x7f36400a4500 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364006f190 con 0x7f36400a7210 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f36477c9640 10 monclient(hunting): authenticate will time out at 1823.020301s 2024-08-19T10:09:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f3645565640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f36400a6720 0x7f36400a6b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39864/0 (socket says 172.21.15.1:39864) 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f3645565640 1 -- 172.21.15.1:0/1552652637 learned_addr learned my addr 172.21.15.1:0/1552652637 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.944+0000 7f3645d66640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f36400a4500 0x7f3640141ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:33610/0 (socket says 172.21.15.1:33610) 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f3645d66640 1 -- 172.21.15.1:0/1552652637 >> v1:172.21.15.1:6791/0 conn(0x7f36400a4500 legacy=0x7f3640141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f3645565640 1 -- 172.21.15.1:0/1552652637 >> v1:172.21.15.1:6790/0 conn(0x7f36400a6720 legacy=0x7f36400a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578470427 0 0) 0x7f36400a3b20 con 0x7f36400a4500 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): my global_id is 194890 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3628003520 con 0x7f36400a4500 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f3644d64640 1 -- 172.21.15.1:0/1552652637 >> v1:172.21.15.82:6791/0 conn(0x7f36400a7210 legacy=0x7f364013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575077964 0 0) 0x7f364006ed50 con 0x7f36400a6720 2024-08-19T10:09:26.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): my global_id is 185024 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36400a3b20 con 0x7f36400a6720 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1390561020 0 0) 0x7f364006f190 con 0x7f36400a7210 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient(hunting): my global_id is 214448 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f364006ed50 con 0x7f36400a7210 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1849877491 0 0) 0x7f3628003520 con 0x7f36400a4500 2024-08-19T10:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 >> v1:172.21.15.82:6791/0 conn(0x7f36400a7210 legacy=0x7f364013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 >> v1:172.21.15.1:6790/0 conn(0x7f36400a6720 legacy=0x7f36400a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 monclient: found mon.noname-c 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36400a9260 con 0x7f36400a4500 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f36477c9640 5 monclient: authenticate success, global_id 194890 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3638001ae0 con 0x7f36400a4500 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f36477c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.946+0000 7f363ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 10 monclient: dump: 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f36380047e0 con 0x7f36400a4500 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 1 -- 172.21.15.1:0/1552652637 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3638004d50 con 0x7f36400a4500 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f363ffff640 10 monclient: dump: 2024-08-19T10:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 4 set_mon_vals no callback set 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 10 monclient: shutdown 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 1 -- 172.21.15.1:0/1552652637 >> v1:172.21.15.1:6791/0 conn(0x7f36400a4500 legacy=0x7f3640141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 1 -- 172.21.15.1:0/1552652637 shutdown_connections 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 1 -- 172.21.15.1:0/1552652637 >> 172.21.15.1:0/1552652637 conn(0x7f36400ac440 msgr2=0x7f36400a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 1 -- 172.21.15.1:0/1552652637 shutdown_connections 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.947+0000 7f36477c9640 1 -- 172.21.15.1:0/1552652637 wait complete. 2024-08-19T10:09:26.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 10 monclient: build_initial_monmap 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 10 monclient: monmap: 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 1 librados: starting msgr at 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 1 librados: starting objecter 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 1 Processor -- start 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 1 -- start start 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 1 librados: setting wanted keys 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 1 librados: calling monclient init 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.948+0000 7f36477c9640 10 monclient: init 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient: _add_conns ranks=[8,2,7,3,4,1,6,0,5] 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient(hunting): picked mon.noname-e con 0x7f36400a4500 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient(hunting): picked mon.noname-b con 0x7f36400a6720 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient(hunting): picked mon.noname-i con 0x7f36400a7210 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36400aa4a0 con 0x7f36400a6720 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36400a1e30 con 0x7f36400a7210 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36401273c0 con 0x7f36400a4500 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f36477c9640 10 monclient(hunting): authenticate will time out at 1823.025254s 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f3645565640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f36400a4500 0x7f3640141dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:45372/0 (socket says 172.21.15.1:45372) 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f3644d64640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f36400a6720 0x7f36401d3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39868/0 (socket says 172.21.15.1:39868) 2024-08-19T10:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.949+0000 7f3645565640 1 -- 172.21.15.1:0/1033583349 learned_addr learned my addr 172.21.15.1:0/1033583349 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f3645565640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.1:6793/0 conn(0x7f36400a4500 legacy=0x7f3640141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 client.?.objecter ms_handle_connect 0x7f36400a4500 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f3644d64640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.1:6790/0 conn(0x7f36400a6720 legacy=0x7f36401d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 client.?.objecter ms_handle_connect 0x7f36400a6720 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f3645d66640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.82:6792/0 conn(0x7f36400a7210 legacy=0x7f36401d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 client.?.objecter ms_handle_connect 0x7f36400a7210 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064796084 0 0) 0x7f36400aa4a0 con 0x7f36400a6720 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): my global_id is 185033 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f362c004ee0 con 0x7f36400a6720 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653095192 0 0) 0x7f36401273c0 con 0x7f36400a4500 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): my global_id is 204686 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36400aa4a0 con 0x7f36400a4500 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1402059499 0 0) 0x7f36400a1e30 con 0x7f36400a7210 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 10 monclient(hunting): my global_id is 204631 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36401273c0 con 0x7f36400a7210 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3842247476 0 0) 0x7f362c004ee0 con 0x7f36400a6720 2024-08-19T10:09:26.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.951+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f36400a1e30 con 0x7f36400a6720 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3634001cf0 con 0x7f36400a6720 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): dump: 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1017887096 0 0) 0x7f36400aa4a0 con 0x7f36400a4500 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f362c004ee0 con 0x7f36400a4500 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3630004b60 con 0x7f36400a4500 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient(hunting): dump: 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.972 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1145730240 0 0) 0x7f36400a1e30 con 0x7f36400a6720 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.1:6793/0 conn(0x7f36400a4500 legacy=0x7f3640141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.82:6792/0 conn(0x7f36400a7210 legacy=0x7f36401d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 monclient: found mon.b 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3640129720 con 0x7f36400a6720 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:26.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 5 monclient: authenticate success, global_id 185033 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 10 monclient: _renew_subs 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3640128e80 con 0x7f36400a6720 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3634003a30 con 0x7f36400a6720 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3634003f20 con 0x7f36400a6720 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363dffb640 10 monclient: dump: 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 10 client.185033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f363f7fe640 4 set_mon_vals no callback set 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 10 monclient: _renew_subs 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3640127f20 con 0x7f36400a6720 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.952+0000 7f36477c9640 1 librados: init done 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f36340040c0 con 0x7f36400a6720 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f36477c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f36477c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f36477c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3640051b00 con 0x7f36400a6720 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f363404ddf0 con 0x7f36400a6720 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f363dffb640 10 client.185033.objecter ms_dispatch 0x7f3640000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f363dffb640 3 client.185033.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f363dffb640 3 client.185033.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f363dffb640 20 client.185033.objecter dump_active .. 0 homeless 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.953+0000 7f3645565640 10 monclient: get_auth_request con 0x7f362c041650 auth_method 0 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.956+0000 7f363dffb640 10 client.185033.objecter ms_handle_connect 0x7f362c041650 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.958+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f36340173f0 con 0x7f36400a6720 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.958+0000 7f363dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:26.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:26.958+0000 7f363dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.140+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f3640126e50 con 0x7f362c041650 2024-08-19T10:09:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.141+0000 7f363dffb640 1 -- 172.21.15.1:0/1033583349 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f3640126e50 con 0x7f362c041650 2024-08-19T10:09:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-08-19T10:09:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 10 librados: watch_flush enter 2024-08-19T10:09:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 10 librados: watch_flush exit 2024-08-19T10:09:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 20 client.185033.objecter shutdown clearing up homeless session... 2024-08-19T10:09:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 10 client.185033.objecter successfully canceled tick 2024-08-19T10:09:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.82:6812/2549104415 conn(0x7f362c041650 legacy=0x7f362c043b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 10 monclient: shutdown 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 >> v1:172.21.15.1:6790/0 conn(0x7f36400a6720 legacy=0x7f36401d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 shutdown_connections 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 >> 172.21.15.1:0/1033583349 conn(0x7f36400ac440 msgr2=0x7f3640142a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 shutdown_connections 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 -- 172.21.15.1:0/1033583349 wait complete. 2024-08-19T10:09:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.143+0000 7f36477c9640 1 librados: shutdown 2024-08-19T10:09:27.155 INFO:tasks.ceph:Scrubbing osd.5 2024-08-19T10:09:27.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.260+0000 7fb19beb3640 10 monclient: get_monmap_and_config 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.260+0000 7fb19beb3640 10 monclient: build_initial_monmap 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.260+0000 7fb19beb3640 10 monclient: monmap: 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:27.261630+0000 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:27.261630+0000 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:27.262 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:27.263 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 1 Processor -- start 2024-08-19T10:09:27.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 1 -- start start 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient: init 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient: _add_conns ranks=[2,1,8,5,7,4,3,0,6] 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1940a66b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient(hunting): picked mon.noname-f con 0x7fb1940a71a0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient(hunting): picked mon.noname-e con 0x7fb1940a3f10 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1940a2e80 con 0x7fb1940a71a0 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19406eea0 con 0x7fb1940a66b0 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1940a7f90 con 0x7fb1940a3f10 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.261+0000 7fb19beb3640 10 monclient(hunting): authenticate will time out at 1823.337964s 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.262+0000 7fb19a450640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fb1940a3f10 0x7fb194141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:45376/0 (socket says 172.21.15.1:45376) 2024-08-19T10:09:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.262+0000 7fb19a450640 1 -- 172.21.15.1:0/2230136832 learned_addr learned my addr 172.21.15.1:0/2230136832 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.262+0000 7fb19944e640 1 -- 172.21.15.1:0/2230136832 >> v1:172.21.15.82:6789/0 conn(0x7fb1940a71a0 legacy=0x7fb19413e6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb19a450640 1 -- 172.21.15.1:0/2230136832 >> v1:172.21.15.1:6793/0 conn(0x7fb1940a3f10 legacy=0x7fb194141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb199c4f640 1 -- 172.21.15.1:0/2230136832 >> v1:172.21.15.1:6790/0 conn(0x7fb1940a66b0 legacy=0x7fb1940a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1201989933 0 0) 0x7fb18c0017d0 con 0x7fb1940a3f10 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): my global_id is 204695 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb180003520 con 0x7fb1940a3f10 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004934710 0 0) 0x7fb19406eea0 con 0x7fb1940a66b0 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient(hunting): my global_id is 185042 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb18c0017d0 con 0x7fb1940a66b0 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 362931820 0 0) 0x7fb18c0017d0 con 0x7fb1940a66b0 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 >> v1:172.21.15.1:6793/0 conn(0x7fb1940a3f10 legacy=0x7fb194141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 >> v1:172.21.15.82:6789/0 conn(0x7fb1940a71a0 legacy=0x7fb19413e6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: _finish_hunting 0 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 monclient: found mon.noname-b 2024-08-19T10:09:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1940a3a10 con 0x7fb1940a66b0 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: _finish_auth 0 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: _check_auth_tickets 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb190001b10 con 0x7fb1940a66b0 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb19beb3640 5 monclient: authenticate success, global_id 185042 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb19beb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: dump: 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.272 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb19beb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb190003610 con 0x7fb1940a66b0 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 1 -- 172.21.15.1:0/2230136832 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb190003b80 con 0x7fb1940a66b0 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb19beb3640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb19beb3640 4 set_mon_vals no callback set 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.264+0000 7fb198c4d640 10 monclient: dump: 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.274 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 10 monclient: shutdown 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 1 -- 172.21.15.1:0/2230136832 >> v1:172.21.15.1:6790/0 conn(0x7fb1940a66b0 legacy=0x7fb1940a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 1 -- 172.21.15.1:0/2230136832 shutdown_connections 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 1 -- 172.21.15.1:0/2230136832 >> 172.21.15.1:0/2230136832 conn(0x7fb1940ac340 msgr2=0x7fb1940a87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 1 -- 172.21.15.1:0/2230136832 shutdown_connections 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 1 -- 172.21.15.1:0/2230136832 wait complete. 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 10 monclient: build_initial_monmap 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.265+0000 7fb19beb3640 10 monclient: monmap: 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:27.275 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 librados: starting msgr at 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 librados: starting objecter 2024-08-19T10:09:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 Processor -- start 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 -- start start 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 librados: setting wanted keys 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 librados: calling monclient init 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient: init 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient: _add_conns ranks=[0,7,8,3,4,6,1,5,2] 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient(hunting): picked mon.noname-a con 0x7fb1940a3f10 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient(hunting): picked mon.noname-i con 0x7fb1940a66b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient(hunting): picked mon.noname-e con 0x7fb1940a71a0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb194124f40 con 0x7fb1940a3f10 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1941272a0 con 0x7fb1940a66b0 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb194126a00 con 0x7fb1940a71a0 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19beb3640 10 monclient(hunting): authenticate will time out at 1823.342909s 2024-08-19T10:09:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb199c4f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb1940a3f10 0x7fb1941398e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43372/0 (socket says 172.21.15.1:43372) 2024-08-19T10:09:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb19a450640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fb1940a71a0 0x7fb1941d6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:45380/0 (socket says 172.21.15.1:45380) 2024-08-19T10:09:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.266+0000 7fb199c4f640 1 -- 172.21.15.1:0/2085550313 learned_addr learned my addr 172.21.15.1:0/2085550313 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19a450640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.1:6793/0 conn(0x7fb1940a71a0 legacy=0x7fb1941d6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 client.?.objecter ms_handle_connect 0x7fb1940a71a0 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 client.?.objecter ms_handle_connect 0x7fb1940a3f10 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb199c4f640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.1:6789/0 conn(0x7fb1940a3f10 legacy=0x7fb1941398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175003697 0 0) 0x7fb194126a00 con 0x7fb1940a71a0 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): my global_id is 204704 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb178004ee0 con 0x7fb1940a71a0 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046530763 0 0) 0x7fb194124f40 con 0x7fb1940a3f10 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): my global_id is 204642 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb194126a00 con 0x7fb1940a3f10 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19944e640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.82:6792/0 conn(0x7fb1940a66b0 legacy=0x7fb1941d2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 client.?.objecter ms_handle_connect 0x7fb1940a66b0 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2048589420 0 0) 0x7fb194126a00 con 0x7fb1940a3f10 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb194124f40 con 0x7fb1940a3f10 2024-08-19T10:09:27.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb190003b00 con 0x7fb1940a3f10 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): dump: 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.284 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2941721078 0 0) 0x7fb178004ee0 con 0x7fb1940a71a0 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb194126a00 con 0x7fb1940a71a0 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb18c001ef0 con 0x7fb1940a71a0 2024-08-19T10:09:27.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): dump: 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.286 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859468895 0 0) 0x7fb1941272a0 con 0x7fb1940a66b0 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient(hunting): my global_id is 204640 2024-08-19T10:09:27.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb178004ee0 con 0x7fb1940a66b0 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2663523699 0 0) 0x7fb194124f40 con 0x7fb1940a3f10 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.1:6793/0 conn(0x7fb1940a71a0 legacy=0x7fb1941d6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.82:6792/0 conn(0x7fb1940a66b0 legacy=0x7fb1941d2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 monclient: found mon.a 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb194125aa0 con 0x7fb1940a3f10 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:27.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 5 monclient: authenticate success, global_id 204642 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 10 monclient: _renew_subs 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb194139ff0 con 0x7fb1940a3f10 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 10 client.204642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 10 monclient: _renew_subs 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1941cdd90 con 0x7fb1940a3f10 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb19beb3640 1 librados: init done 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb190001350 con 0x7fb1940a3f10 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:27.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb1900057c0 con 0x7fb1940a3f10 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18bfff640 4 set_mon_vals no callback set 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.269+0000 7fb18a7fc640 10 monclient: dump: 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.290 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb19beb3640 10 client.204642.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb19beb3640 15 client.204642.objecter _session_command_op_assign -1 1 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb19beb3640 20 client.204642.objecter _get_session osd=-1 returning homeless 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb19beb3640 10 client.204642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:27.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb19beb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.1:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fb17c002910 con 0x7fb1940a3f10 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb190011c20 con 0x7fb1940a3f10 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fb19004e060 con 0x7fb1940a3f10 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb18a7fc640 10 client.204642.objecter ms_dispatch 0x7fb194000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb18a7fc640 3 client.204642.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.270+0000 7fb18a7fc640 3 client.204642.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 10 client.204642.objecter checking command 1 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 20 client.204642.objecter _get_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 15 client.204642.objecter _session_command_op_remove -1 1 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 20 client.204642.objecter _get_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 20 client.204642.objecter get_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 15 client.204642.objecter _session_command_op_assign 5 1 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 4 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 10 client.204642.objecter _send_command 1 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.82:6808/1813220032 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb190003610 con 0x7fb178049c50 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 20 client.204642.objecter dump_active .. 0 homeless 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_get_version_reply(handle=1 version=711) ==== 24+0+0 (unknown 3044792690 0 0) 0x7fb19004e4d0 con 0x7fb1940a3f10 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb18a7fc640 10 monclient: handle_get_version_reply finishing 1 version 711 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb19944e640 10 monclient: get_auth_request con 0x7fb1780415b0 auth_method 0 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.271+0000 7fb19a450640 10 monclient: get_auth_request con 0x7fb178049c50 auth_method 0 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.273+0000 7fb18a7fc640 10 client.204642.objecter ms_handle_connect 0x7fb1780415b0 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.275+0000 7fb18a7fc640 10 client.204642.objecter ms_handle_connect 0x7fb178049c50 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.276+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== osd.5 v1:172.21.15.82:6808/1813220032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb190003610 con 0x7fb178049c50 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.276+0000 7fb18a7fc640 10 client.204642.objecter ms_dispatch 0x7fb194000b90 command_reply(tid 1: 0 ) 2024-08-19T10:09:27.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.276+0000 7fb18a7fc640 10 client.204642.objecter _finish_command 1 = system:0 2024-08-19T10:09:27.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.276+0000 7fb18a7fc640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 4 2024-08-19T10:09:27.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.276+0000 7fb18a7fc640 15 client.204642.objecter _session_command_op_remove 5 1 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 10 client.204642.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 15 client.204642.objecter _session_command_op_assign -1 2 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 20 client.204642.objecter _get_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 20 client.204642.objecter _get_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 15 client.204642.objecter _session_command_op_remove -1 2 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 20 client.204642.objecter get_session s=0x7fb1780495a0 osd=5 3 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 15 client.204642.objecter _session_command_op_assign 5 2 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 4 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 10 client.204642.objecter _send_command 2 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.289+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 --> v1:172.21.15.82:6808/1813220032 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb17c0012b0 con 0x7fb178049c50 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb18a7fc640 1 -- 172.21.15.1:0/2085550313 <== osd.5 v1:172.21.15.82:6808/1813220032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fb17c0012b0 con 0x7fb178049c50 2024-08-19T10:09:27.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb18a7fc640 10 client.204642.objecter ms_dispatch 0x7fb194000b90 command_reply(tid 2: 0 ) 2024-08-19T10:09:27.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb18a7fc640 10 client.204642.objecter _finish_command 2 = system:0 2024-08-19T10:09:27.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb18a7fc640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 4 2024-08-19T10:09:27.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb18a7fc640 15 client.204642.objecter _session_command_op_remove 5 2 2024-08-19T10:09:27.296 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T10:09:27.296 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:27.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 10 librados: watch_flush enter 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 10 librados: watch_flush exit 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 10 client.204642.objecter close_session for osd.5 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.82:6808/1813220032 conn(0x7fb178049c50 legacy=0x7fb17804c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 20 client.204642.objecter put_session s=0x7fb1780495a0 osd=5 1 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 20 client.204642.objecter shutdown clearing up homeless session... 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 10 client.204642.objecter successfully canceled tick 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb1780415b0 legacy=0x7fb178043a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 10 monclient: shutdown 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 >> v1:172.21.15.1:6789/0 conn(0x7fb1940a3f10 legacy=0x7fb1941398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19a450640 1 -- 172.21.15.1:0/2085550313 reap_dead start 2024-08-19T10:09:27.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 shutdown_connections 2024-08-19T10:09:27.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 >> 172.21.15.1:0/2085550313 conn(0x7fb1940ac340 msgr2=0x7fb1940aa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 shutdown_connections 2024-08-19T10:09:27.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 -- 172.21.15.1:0/2085550313 wait complete. 2024-08-19T10:09:27.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.290+0000 7fb19beb3640 1 librados: shutdown 2024-08-19T10:09:27.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-08-19T10:09:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.399+0000 7fc9e8284640 10 monclient: get_monmap_and_config 2024-08-19T10:09:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.399+0000 7fc9e8284640 10 monclient: build_initial_monmap 2024-08-19T10:09:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.399+0000 7fc9e8284640 10 monclient: monmap: 2024-08-19T10:09:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:27.400349+0000 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:27.400349+0000 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:27.401 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:27.402 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:27.402 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.399+0000 7fc9e8284640 1 Processor -- start 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 1 -- start start 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient: init 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient: _add_conns ranks=[2,3,7,0,5,8,1,4,6] 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e00a6720 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient(hunting): picked mon.noname-g con 0x7fc9e00a7210 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient(hunting): picked mon.noname-i con 0x7fc9e00a4500 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e006ed50 con 0x7fc9e00a6720 2024-08-19T10:09:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e00a3b20 con 0x7fc9e00a7210 2024-08-19T10:09:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e006f190 con 0x7fc9e00a4500 2024-08-19T10:09:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e8284640 10 monclient(hunting): authenticate will time out at 1823.476695s 2024-08-19T10:09:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e6020640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc9e00a6720 0x7fc9e00a6b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39892/0 (socket says 172.21.15.1:39892) 2024-08-19T10:09:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.400+0000 7fc9e6020640 1 -- 172.21.15.1:0/280993391 learned_addr learned my addr 172.21.15.1:0/280993391 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:27.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.402+0000 7fc9e6020640 1 -- 172.21.15.1:0/280993391 >> v1:172.21.15.1:6790/0 conn(0x7fc9e00a6720 legacy=0x7fc9e00a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231312439 0 0) 0x7fc9e006ed50 con 0x7fc9e00a6720 2024-08-19T10:09:27.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient(hunting): my global_id is 185051 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c8003520 con 0x7fc9e00a6720 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e581f640 1 -- 172.21.15.1:0/280993391 >> v1:172.21.15.82:6790/0 conn(0x7fc9e00a7210 legacy=0x7fc9e013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e6821640 1 -- 172.21.15.1:0/280993391 >> v1:172.21.15.82:6792/0 conn(0x7fc9e00a4500 legacy=0x7fc9e0141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3694717572 0 0) 0x7fc9c8003520 con 0x7fc9e00a6720 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 >> v1:172.21.15.82:6792/0 conn(0x7fc9e00a4500 legacy=0x7fc9e0141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 >> v1:172.21.15.82:6790/0 conn(0x7fc9e00a7210 legacy=0x7fc9e013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: _finish_hunting 0 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 monclient: found mon.noname-b 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e00a9260 con 0x7fc9e00a6720 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: _finish_auth 0 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: _check_auth_tickets 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d8001ae0 con 0x7fc9e00a6720 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 5 monclient: authenticate success, global_id 185051 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: dump: 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9d80048a0 con 0x7fc9e00a6720 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 1 -- 172.21.15.1:0/280993391 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d8004e10 con 0x7fc9e00a6720 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 4 set_mon_vals no callback set 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e501e640 10 monclient: dump: 2024-08-19T10:09:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 10 monclient: shutdown 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.403+0000 7fc9e8284640 1 -- 172.21.15.1:0/280993391 >> v1:172.21.15.1:6790/0 conn(0x7fc9e00a6720 legacy=0x7fc9e00a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 -- 172.21.15.1:0/280993391 shutdown_connections 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 -- 172.21.15.1:0/280993391 >> 172.21.15.1:0/280993391 conn(0x7fc9e00ac440 msgr2=0x7fc9e00a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 -- 172.21.15.1:0/280993391 shutdown_connections 2024-08-19T10:09:27.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 -- 172.21.15.1:0/280993391 wait complete. 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 10 monclient: build_initial_monmap 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 10 monclient: monmap: 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:27.414 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 librados: starting msgr at 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 librados: starting objecter 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.404+0000 7fc9e8284640 1 Processor -- start 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 1 -- start start 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 1 librados: setting wanted keys 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 1 librados: calling monclient init 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient: init 2024-08-19T10:09:27.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient: _add_conns ranks=[3,8,2,0,6,1,5,7,4] 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient(hunting): picked mon.noname-g con 0x7fc9e00a4500 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient(hunting): picked mon.noname-e con 0x7fc9e00a6720 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e00a7210 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e0142600 con 0x7fc9e00a7210 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e00a2750 con 0x7fc9e00a4500 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e0126eb0 con 0x7fc9e00a6720 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e8284640 10 monclient(hunting): authenticate will time out at 1823.481485s 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e6821640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc9e00a7210 0x7fc9e01d6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39900/0 (socket says 172.21.15.1:39900) 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e581f640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc9e00a6720 0x7fc9e01d3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:45384/0 (socket says 172.21.15.1:45384) 2024-08-19T10:09:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.405+0000 7fc9e6821640 1 -- 172.21.15.1:0/2020314074 learned_addr learned my addr 172.21.15.1:0/2020314074 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9e581f640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.1:6793/0 conn(0x7fc9e00a6720 legacy=0x7fc9e01d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9e6821640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.1:6790/0 conn(0x7fc9e00a7210 legacy=0x7fc9e01d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 client.?.objecter ms_handle_connect 0x7fc9e00a6720 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 client.?.objecter ms_handle_connect 0x7fc9e00a7210 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431342649 0 0) 0x7fc9e0142600 con 0x7fc9e00a7210 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): my global_id is 185060 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9cc004eb0 con 0x7fc9e00a7210 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932130332 0 0) 0x7fc9e0126eb0 con 0x7fc9e00a6720 2024-08-19T10:09:27.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 monclient(hunting): my global_id is 204713 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e0142600 con 0x7fc9e00a6720 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 client.?.objecter ms_handle_connect 0x7fc9e00a4500 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.407+0000 7fc9e6020640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.82:6790/0 conn(0x7fc9e00a4500 legacy=0x7fc9e0141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1627459943 0 0) 0x7fc9e00a2750 con 0x7fc9e00a4500 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): my global_id is 214401 2024-08-19T10:09:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e0126eb0 con 0x7fc9e00a4500 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2533827501 0 0) 0x7fc9cc004eb0 con 0x7fc9e00a7210 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e00a2750 con 0x7fc9e00a7210 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1563537573 0 0) 0x7fc9e0142600 con 0x7fc9e00a6720 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9cc004eb0 con 0x7fc9e00a6720 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d00020a0 con 0x7fc9e00a7210 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): dump: 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d4001cf0 con 0x7fc9e00a6720 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient(hunting): dump: 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2744274790 0 0) 0x7fc9cc004eb0 con 0x7fc9e00a6720 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.82:6790/0 conn(0x7fc9e00a4500 legacy=0x7fc9e0141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.1:6790/0 conn(0x7fc9e00a7210 legacy=0x7fc9e01d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 monclient: found mon.e 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e01289e0 con 0x7fc9e00a6720 2024-08-19T10:09:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:27.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 5 monclient: authenticate success, global_id 204713 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 10 monclient: _renew_subs 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e01274f0 con 0x7fc9e00a6720 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 10 client.204713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 10 monclient: _renew_subs 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9e0128050 con 0x7fc9e00a6720 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e8284640 1 librados: init done 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9d4003a20 con 0x7fc9e00a6720 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d4003f50 con 0x7fc9e00a6720 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9e481d640 4 set_mon_vals no callback set 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.408+0000 7fc9deffd640 10 monclient: dump: 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.432 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc9d40041f0 con 0x7fc9e00a6720 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9e8284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9e8284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9e8284640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.1:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c0005080 con 0x7fc9e00a6720 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fc9d404de00 con 0x7fc9e00a6720 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9deffd640 10 client.204713.objecter ms_dispatch 0x7fc9e0000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9deffd640 3 client.204713.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9deffd640 3 client.204713.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9deffd640 20 client.204713.objecter dump_active .. 0 homeless 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.409+0000 7fc9e6020640 10 monclient: get_auth_request con 0x7fc9cc041600 auth_method 0 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.412+0000 7fc9deffd640 10 client.204713.objecter ms_handle_connect 0x7fc9cc041600 2024-08-19T10:09:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.414+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mon.8 v1:172.21.15.1:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fc9d4017460 con 0x7fc9e00a6720 2024-08-19T10:09:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.414+0000 7fc9deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.414+0000 7fc9deffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.595+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fc9c0000b90 con 0x7fc9cc041600 2024-08-19T10:09:27.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.596+0000 7fc9deffd640 1 -- 172.21.15.1:0/2020314074 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7fc9c0000b90 con 0x7fc9cc041600 2024-08-19T10:09:27.597 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-08-19T10:09:27.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 10 librados: watch_flush enter 2024-08-19T10:09:27.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 10 librados: watch_flush exit 2024-08-19T10:09:27.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 20 client.204713.objecter shutdown clearing up homeless session... 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 10 client.204713.objecter successfully canceled tick 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc9cc041600 legacy=0x7fc9cc043ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 10 monclient: shutdown 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 >> v1:172.21.15.1:6793/0 conn(0x7fc9e00a6720 legacy=0x7fc9e01d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 shutdown_connections 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 >> 172.21.15.1:0/2020314074 conn(0x7fc9e00ac440 msgr2=0x7fc9e0142a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 shutdown_connections 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 -- 172.21.15.1:0/2020314074 wait complete. 2024-08-19T10:09:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.598+0000 7fc9e8284640 1 librados: shutdown 2024-08-19T10:09:27.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.707+0000 7fb34c19a640 10 monclient: get_monmap_and_config 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.707+0000 7fb34c19a640 10 monclient: build_initial_monmap 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.707+0000 7fb34c19a640 10 monclient: monmap: 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:27.708690+0000 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:27.708690+0000 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:27.710 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 1 Processor -- start 2024-08-19T10:09:27.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 1 -- start start 2024-08-19T10:09:27.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 10 monclient: init 2024-08-19T10:09:27.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 10 monclient: _add_conns ranks=[0,7,2,3,6,5,8,4,1] 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 10 monclient(hunting): picked mon.noname-a con 0x7fb3440a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 10 monclient(hunting): picked mon.noname-i con 0x7fb3440a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.708+0000 7fb34c19a640 10 monclient(hunting): picked mon.noname-b con 0x7fb34413f5c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb34c19a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3440a3b40 con 0x7fb3440a67c0 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb34c19a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3440a8130 con 0x7fb34413f5c0 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb34c19a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb34406ede0 con 0x7fb3440a6ba0 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb34c19a640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb34c19a640 10 monclient(hunting): authenticate will time out at 1823.785031s 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb349f36640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb3440a67c0 0x7fb3440a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43380/0 (socket says 172.21.15.1:43380) 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb34a737640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb34413f5c0 0x7fb3441419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39914/0 (socket says 172.21.15.1:39914) 2024-08-19T10:09:27.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.709+0000 7fb349f36640 1 -- 172.21.15.1:0/265982142 learned_addr learned my addr 172.21.15.1:0/265982142 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:27.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34a737640 1 -- 172.21.15.1:0/265982142 >> v1:172.21.15.1:6790/0 conn(0x7fb34413f5c0 legacy=0x7fb3441419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb349f36640 1 -- 172.21.15.1:0/265982142 >> v1:172.21.15.1:6789/0 conn(0x7fb3440a67c0 legacy=0x7fb3440a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875193282 0 0) 0x7fb3440a3b40 con 0x7fb3440a67c0 2024-08-19T10:09:27.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): my global_id is 204651 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb330003520 con 0x7fb3440a67c0 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430435564 0 0) 0x7fb3440a8130 con 0x7fb34413f5c0 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient(hunting): my global_id is 185069 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3440a3b40 con 0x7fb34413f5c0 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb349735640 1 -- 172.21.15.1:0/265982142 >> v1:172.21.15.82:6792/0 conn(0x7fb3440a6ba0 legacy=0x7fb3440a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2008887342 0 0) 0x7fb3440a3b40 con 0x7fb34413f5c0 2024-08-19T10:09:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 >> v1:172.21.15.82:6792/0 conn(0x7fb3440a6ba0 legacy=0x7fb3440a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 >> v1:172.21.15.1:6789/0 conn(0x7fb3440a67c0 legacy=0x7fb3440a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: _finish_hunting 0 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 monclient: found mon.noname-b 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3440a92a0 con 0x7fb34413f5c0 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: _finish_auth 0 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: _check_auth_tickets 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb33c001a40 con 0x7fb34413f5c0 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34c19a640 5 monclient: authenticate success, global_id 185069 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34c19a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: dump: 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.719 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34c19a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34a737640 1 --1- 172.21.15.1:0/265982142 >> v1:172.21.15.1:6790/0 conn(0x7fb34413f5c0 0x7fb3441419b0 :-1 s=OPENED pgs=3701 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7fb33c004b20 mon_map magic: 0 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb33c001d10 con 0x7fb34413f5c0 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb348f34640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34c19a640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.711+0000 7fb34c19a640 4 set_mon_vals no callback set 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 10 monclient: shutdown 2024-08-19T10:09:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 1 -- 172.21.15.1:0/265982142 >> v1:172.21.15.1:6790/0 conn(0x7fb34413f5c0 legacy=0x7fb3441419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb348f34640 1 -- 172.21.15.1:0/265982142 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb33c004b20 con 0x7fb34413f5c0 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb348f34640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 1 -- 172.21.15.1:0/265982142 shutdown_connections 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 1 -- 172.21.15.1:0/265982142 >> 172.21.15.1:0/265982142 conn(0x7fb3440ac4b0 msgr2=0x7fb3440a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 1 -- 172.21.15.1:0/265982142 shutdown_connections 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 1 -- 172.21.15.1:0/265982142 wait complete. 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 10 monclient: build_initial_monmap 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.712+0000 7fb34c19a640 10 monclient: monmap: 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:27.722 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 librados: starting msgr at 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 librados: starting objecter 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 Processor -- start 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 -- start start 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 librados: setting wanted keys 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 librados: calling monclient init 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient: init 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient: _add_conns ranks=[5,6,7,0,1,2,3,8,4] 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient(hunting): picked mon.noname-h con 0x7fb3440a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient(hunting): picked mon.noname-d con 0x7fb3440a6ba0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient(hunting): picked mon.noname-i con 0x7fb34413f5c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:27.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3440a2320 con 0x7fb3440a67c0 2024-08-19T10:09:27.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3441cf230 con 0x7fb3440a6ba0 2024-08-19T10:09:27.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb344142d60 con 0x7fb34413f5c0 2024-08-19T10:09:27.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:27.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb34c19a640 10 monclient(hunting): authenticate will time out at 1823.789780s 2024-08-19T10:09:27.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb349735640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fb3440a6ba0 0x7fb3441ceb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:36392/0 (socket says 172.21.15.1:36392) 2024-08-19T10:09:27.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.713+0000 7fb349735640 1 -- 172.21.15.1:0/1273080785 learned_addr learned my addr 172.21.15.1:0/1273080785 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:27.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.715+0000 7fb349735640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.1:6792/0 conn(0x7fb3440a6ba0 legacy=0x7fb3441ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 10 client.?.objecter ms_handle_connect 0x7fb3440a6ba0 2024-08-19T10:09:27.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3455723894 0 0) 0x7fb3441cf230 con 0x7fb3440a6ba0 2024-08-19T10:09:27.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb349f36640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.82:6791/0 conn(0x7fb3440a67c0 legacy=0x7fb3441ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 10 monclient(hunting): my global_id is 204513 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb34a737640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.82:6792/0 conn(0x7fb34413f5c0 legacy=0x7fb3441d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb328004ee0 con 0x7fb3440a6ba0 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.716+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 client.?.objecter ms_handle_connect 0x7fb3440a67c0 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 client.?.objecter ms_handle_connect 0x7fb34413f5c0 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:27.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3093510407 0 0) 0x7fb344142d60 con 0x7fb34413f5c0 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): my global_id is 204667 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3441cf230 con 0x7fb34413f5c0 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141550135 0 0) 0x7fb340001960 con 0x7fb3440a67c0 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): my global_id is 214457 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb344142d60 con 0x7fb3440a67c0 2024-08-19T10:09:27.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2528619028 0 0) 0x7fb328004ee0 con 0x7fb3440a6ba0 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb340001960 con 0x7fb3440a6ba0 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb334003610 con 0x7fb3440a6ba0 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): dump: 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.730 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1404629561 0 0) 0x7fb340001960 con 0x7fb3440a6ba0 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.82:6792/0 conn(0x7fb34413f5c0 legacy=0x7fb3441d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.82:6791/0 conn(0x7fb3440a67c0 legacy=0x7fb3441ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 monclient: found mon.d 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3440a3120 con 0x7fb3440a6ba0 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 5 monclient: authenticate success, global_id 204513 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 10 monclient: _renew_subs 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3440a16b0 con 0x7fb3440a6ba0 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 10 client.204513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 10 monclient: _renew_subs 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb34413f320 con 0x7fb3440a6ba0 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb34c19a640 1 librados: init done 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb334003b10 con 0x7fb3440a6ba0 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb334004040 con 0x7fb3440a6ba0 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33bfff640 4 set_mon_vals no callback set 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.718+0000 7fb33a7fc640 10 monclient: dump: 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:27.733 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.719+0000 7fb34c19a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.719+0000 7fb34c19a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.719+0000 7fb34c19a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.719+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb3340041e0 con 0x7fb3440a6ba0 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.719+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb32c005080 con 0x7fb3440a6ba0 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.720+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fb33404dfa0 con 0x7fb3440a6ba0 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.720+0000 7fb33a7fc640 10 client.204513.objecter ms_dispatch 0x7fb344000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.720+0000 7fb33a7fc640 3 client.204513.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.720+0000 7fb33a7fc640 3 client.204513.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.720+0000 7fb33a7fc640 20 client.204513.objecter dump_active .. 0 homeless 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.720+0000 7fb349f36640 10 monclient: get_auth_request con 0x7fb328041270 auth_method 0 2024-08-19T10:09:27.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.722+0000 7fb33a7fc640 10 client.204513.objecter ms_handle_connect 0x7fb328041270 2024-08-19T10:09:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.724+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb334017600 con 0x7fb3440a6ba0 2024-08-19T10:09:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.724+0000 7fb33a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.724+0000 7fb33a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.905+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb32c000b90 con 0x7fb328041270 2024-08-19T10:09:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.907+0000 7fb33a7fc640 1 -- 172.21.15.1:0/1273080785 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113846 (unknown 2965378022 0 2867107226) 0x7fb32c000b90 con 0x7fb328041270 2024-08-19T10:09:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T10:09:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T10:09:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 10 librados: watch_flush enter 2024-08-19T10:09:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 10 librados: watch_flush exit 2024-08-19T10:09:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 20 client.204513.objecter shutdown clearing up homeless session... 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 10 client.204513.objecter successfully canceled tick 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb328041270 legacy=0x7fb328043730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 10 monclient: shutdown 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.909+0000 7fb34c19a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.910+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 >> v1:172.21.15.1:6792/0 conn(0x7fb3440a6ba0 legacy=0x7fb3441ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.910+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 shutdown_connections 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.910+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 >> 172.21.15.1:0/1273080785 conn(0x7fb3440ac4b0 msgr2=0x7fb3440a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.910+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 shutdown_connections 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.910+0000 7fb34c19a640 1 -- 172.21.15.1:0/1273080785 wait complete. 2024-08-19T10:09:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:27.910+0000 7fb34c19a640 1 librados: shutdown 2024-08-19T10:09:27.921 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1284,"stamp":"2024-08-19T10:09:27.061909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":442656,"kb_used_data":38928,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565788384,"statfs":{"total":579820584960,"available":579367305216,"internally_reserved":0,"allocated":39862272,"data_stored":17623178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":413260450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001523"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1496,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632196+0000","last_change":"2024-08-19T09:50:08.953637+0000","last_active":"2024-08-19T10:09:20.632196+0000","last_peered":"2024-08-19T10:09:20.632196+0000","last_clean":"2024-08-19T10:09:20.632196+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T10:09:20.632196+0000","last_undegraded":"2024-08-19T10:09:20.632196+0000","last_fullsized":"2024-08-19T10:09:20.632196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:07.419178+0000","last_clean_scrub_stamp":"2024-08-19T09:50:07.419178+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:15:12.482947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1404,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631119+0000","last_change":"2024-08-19T09:50:17.768584+0000","last_active":"2024-08-19T10:09:20.631119+0000","last_peered":"2024-08-19T10:09:20.631119+0000","last_clean":"2024-08-19T10:09:20.631119+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T10:09:20.631119+0000","last_undegraded":"2024-08-19T10:09:20.631119+0000","last_fullsized":"2024-08-19T10:09:20.631119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:11:45.274731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1405,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632165+0000","last_change":"2024-08-19T09:50:17.768191+0000","last_active":"2024-08-19T10:09:20.632165+0000","last_peered":"2024-08-19T10:09:20.632165+0000","last_clean":"2024-08-19T10:09:20.632165+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T10:09:20.632165+0000","last_undegraded":"2024-08-19T10:09:20.632165+0000","last_fullsized":"2024-08-19T10:09:20.632165+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:35:12.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1403,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628092+0000","last_change":"2024-08-19T09:50:17.768317+0000","last_active":"2024-08-19T10:09:20.628092+0000","last_peered":"2024-08-19T10:09:20.628092+0000","last_clean":"2024-08-19T10:09:20.628092+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T10:09:20.628092+0000","last_undegraded":"2024-08-19T10:09:20.628092+0000","last_fullsized":"2024-08-19T10:09:20.628092+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:14:28.300620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1405,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631240+0000","last_change":"2024-08-19T09:50:17.768001+0000","last_active":"2024-08-19T10:09:20.631240+0000","last_peered":"2024-08-19T10:09:20.631240+0000","last_clean":"2024-08-19T10:09:20.631240+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T10:09:20.631240+0000","last_undegraded":"2024-08-19T10:09:20.631240+0000","last_fullsized":"2024-08-19T10:09:20.631240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:17.839426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1407,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:25.012786+0000","last_change":"2024-08-19T10:09:25.012809+0000","last_active":"2024-08-19T10:09:25.012786+0000","last_peered":"2024-08-19T10:09:25.012786+0000","last_clean":"2024-08-19T10:09:25.012786+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T10:09:25.012786+0000","last_undegraded":"2024-08-19T10:09:25.012786+0000","last_fullsized":"2024-08-19T10:09:25.012786+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:25.012721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:25.012721+0000","last_clean_scrub_stamp":"2024-08-19T10:09:25.012721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:20:07.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1405,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632353+0000","last_change":"2024-08-19T09:50:17.768483+0000","last_active":"2024-08-19T10:09:20.632353+0000","last_peered":"2024-08-19T10:09:20.632353+0000","last_clean":"2024-08-19T10:09:20.632353+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T10:09:20.632353+0000","last_undegraded":"2024-08-19T10:09:20.632353+0000","last_fullsized":"2024-08-19T10:09:20.632353+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:36:04.512176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1406,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:25.976146+0000","last_change":"2024-08-19T10:09:25.976179+0000","last_active":"2024-08-19T10:09:25.976146+0000","last_peered":"2024-08-19T10:09:25.976146+0000","last_clean":"2024-08-19T10:09:25.976146+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T10:09:25.976146+0000","last_undegraded":"2024-08-19T10:09:25.976146+0000","last_fullsized":"2024-08-19T10:09:25.976146+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:25.976052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:25.976052+0000","last_clean_scrub_stamp":"2024-08-19T10:09:25.976052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:53:02.356791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1404,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632460+0000","last_change":"2024-08-19T09:50:17.768240+0000","last_active":"2024-08-19T10:09:20.632460+0000","last_peered":"2024-08-19T10:09:20.632460+0000","last_clean":"2024-08-19T10:09:20.632460+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T10:09:20.632460+0000","last_undegraded":"2024-08-19T10:09:20.632460+0000","last_fullsized":"2024-08-19T10:09:20.632460+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:50:14.111421+0000","last_clean_scrub_stamp":"2024-08-19T09:50:14.111421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:59:09.686825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1","version":"23'7","reported_seq":1400,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632286+0000","last_change":"2024-08-19T09:51:22.586587+0000","last_active":"2024-08-19T10:09:20.632286+0000","last_peered":"2024-08-19T10:09:20.632286+0000","last_clean":"2024-08-19T10:09:20.632286+0000","last_became_active":"2024-08-19T09:51:22.585789+0000","last_became_peered":"2024-08-19T09:51:22.585789+0000","last_unstale":"2024-08-19T10:09:20.632286+0000","last_undegraded":"2024-08-19T10:09:20.632286+0000","last_fullsized":"2024-08-19T10:09:20.632286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:25:11.479863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1386,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628005+0000","last_change":"2024-08-19T09:51:22.586723+0000","last_active":"2024-08-19T10:09:20.628005+0000","last_peered":"2024-08-19T10:09:20.628005+0000","last_clean":"2024-08-19T10:09:20.628005+0000","last_became_active":"2024-08-19T09:51:22.583401+0000","last_became_peered":"2024-08-19T09:51:22.583401+0000","last_unstale":"2024-08-19T10:09:20.628005+0000","last_undegraded":"2024-08-19T10:09:20.628005+0000","last_fullsized":"2024-08-19T10:09:20.628005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:56:50.500489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628356+0000","last_change":"2024-08-19T09:51:22.588184+0000","last_active":"2024-08-19T10:09:20.628356+0000","last_peered":"2024-08-19T10:09:20.628356+0000","last_clean":"2024-08-19T10:09:20.628356+0000","last_became_active":"2024-08-19T09:51:22.587936+0000","last_became_peered":"2024-08-19T09:51:22.587936+0000","last_unstale":"2024-08-19T10:09:20.628356+0000","last_undegraded":"2024-08-19T10:09:20.628356+0000","last_fullsized":"2024-08-19T10:09:20.628356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:22:07.623601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628204+0000","last_change":"2024-08-19T09:51:22.640927+0000","last_active":"2024-08-19T10:09:20.628204+0000","last_peered":"2024-08-19T10:09:20.628204+0000","last_clean":"2024-08-19T10:09:20.628204+0000","last_became_active":"2024-08-19T09:51:22.640604+0000","last_became_peered":"2024-08-19T09:51:22.640604+0000","last_unstale":"2024-08-19T10:09:20.628204+0000","last_undegraded":"2024-08-19T10:09:20.628204+0000","last_fullsized":"2024-08-19T10:09:20.628204+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:38:41.853584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1387,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632330+0000","last_change":"2024-08-19T09:51:22.581514+0000","last_active":"2024-08-19T10:09:20.632330+0000","last_peered":"2024-08-19T10:09:20.632330+0000","last_clean":"2024-08-19T10:09:20.632330+0000","last_became_active":"2024-08-19T09:51:22.581246+0000","last_became_peered":"2024-08-19T09:51:22.581246+0000","last_unstale":"2024-08-19T10:09:20.632330+0000","last_undegraded":"2024-08-19T10:09:20.632330+0000","last_fullsized":"2024-08-19T10:09:20.632330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:12:50.239096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631344+0000","last_change":"2024-08-19T09:51:22.620559+0000","last_active":"2024-08-19T10:09:20.631344+0000","last_peered":"2024-08-19T10:09:20.631344+0000","last_clean":"2024-08-19T10:09:20.631344+0000","last_became_active":"2024-08-19T09:51:22.620042+0000","last_became_peered":"2024-08-19T09:51:22.620042+0000","last_unstale":"2024-08-19T10:09:20.631344+0000","last_undegraded":"2024-08-19T10:09:20.631344+0000","last_fullsized":"2024-08-19T10:09:20.631344+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:27:35.980026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1384,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628638+0000","last_change":"2024-08-19T09:51:22.606649+0000","last_active":"2024-08-19T10:09:20.628638+0000","last_peered":"2024-08-19T10:09:20.628638+0000","last_clean":"2024-08-19T10:09:20.628638+0000","last_became_active":"2024-08-19T09:51:22.606054+0000","last_became_peered":"2024-08-19T09:51:22.606054+0000","last_unstale":"2024-08-19T10:09:20.628638+0000","last_undegraded":"2024-08-19T10:09:20.628638+0000","last_fullsized":"2024-08-19T10:09:20.628638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:23:52.344010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628354+0000","last_change":"2024-08-19T09:51:22.653723+0000","last_active":"2024-08-19T10:09:20.628354+0000","last_peered":"2024-08-19T10:09:20.628354+0000","last_clean":"2024-08-19T10:09:20.628354+0000","last_became_active":"2024-08-19T09:51:22.653455+0000","last_became_peered":"2024-08-19T09:51:22.653455+0000","last_unstale":"2024-08-19T10:09:20.628354+0000","last_undegraded":"2024-08-19T10:09:20.628354+0000","last_fullsized":"2024-08-19T10:09:20.628354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:25:13.028210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1378,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631422+0000","last_change":"2024-08-19T09:51:22.560997+0000","last_active":"2024-08-19T10:09:20.631422+0000","last_peered":"2024-08-19T10:09:20.631422+0000","last_clean":"2024-08-19T10:09:20.631422+0000","last_became_active":"2024-08-19T09:51:22.553419+0000","last_became_peered":"2024-08-19T09:51:22.553419+0000","last_unstale":"2024-08-19T10:09:20.631422+0000","last_undegraded":"2024-08-19T10:09:20.631422+0000","last_fullsized":"2024-08-19T10:09:20.631422+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:25.353142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.627921+0000","last_change":"2024-08-19T09:51:22.554649+0000","last_active":"2024-08-19T10:09:20.627921+0000","last_peered":"2024-08-19T10:09:20.627921+0000","last_clean":"2024-08-19T10:09:20.627921+0000","last_became_active":"2024-08-19T09:51:22.554397+0000","last_became_peered":"2024-08-19T09:51:22.554397+0000","last_unstale":"2024-08-19T10:09:20.627921+0000","last_undegraded":"2024-08-19T10:09:20.627921+0000","last_fullsized":"2024-08-19T10:09:20.627921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:36:10.709773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1387,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:24.939724+0000","last_change":"2024-08-19T09:55:22.703762+0000","last_active":"2024-08-19T10:09:24.939724+0000","last_peered":"2024-08-19T10:09:24.939724+0000","last_clean":"2024-08-19T10:09:24.939724+0000","last_became_active":"2024-08-19T09:55:22.703438+0000","last_became_peered":"2024-08-19T09:55:22.703438+0000","last_unstale":"2024-08-19T10:09:24.939724+0000","last_undegraded":"2024-08-19T10:09:24.939724+0000","last_fullsized":"2024-08-19T10:09:24.939724+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1381,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631337+0000","last_change":"2024-08-19T09:51:22.529738+0000","last_active":"2024-08-19T10:09:20.631337+0000","last_peered":"2024-08-19T10:09:20.631337+0000","last_clean":"2024-08-19T10:09:20.631337+0000","last_became_active":"2024-08-19T09:51:22.528219+0000","last_became_peered":"2024-08-19T09:51:22.528219+0000","last_unstale":"2024-08-19T10:09:20.631337+0000","last_undegraded":"2024-08-19T10:09:20.631337+0000","last_fullsized":"2024-08-19T10:09:20.631337+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631324+0000","last_change":"2024-08-19T09:51:22.614370+0000","last_active":"2024-08-19T10:09:20.631324+0000","last_peered":"2024-08-19T10:09:20.631324+0000","last_clean":"2024-08-19T10:09:20.631324+0000","last_became_active":"2024-08-19T09:51:22.614136+0000","last_became_peered":"2024-08-19T09:51:22.614136+0000","last_unstale":"2024-08-19T10:09:20.631324+0000","last_undegraded":"2024-08-19T10:09:20.631324+0000","last_fullsized":"2024-08-19T10:09:20.631324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:14:54.869367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628283+0000","last_change":"2024-08-19T09:51:22.587447+0000","last_active":"2024-08-19T10:09:20.628283+0000","last_peered":"2024-08-19T10:09:20.628283+0000","last_clean":"2024-08-19T10:09:20.628283+0000","last_became_active":"2024-08-19T09:51:22.587149+0000","last_became_peered":"2024-08-19T09:51:22.587149+0000","last_unstale":"2024-08-19T10:09:20.628283+0000","last_undegraded":"2024-08-19T10:09:20.628283+0000","last_fullsized":"2024-08-19T10:09:20.628283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:08:36.757587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1377,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632076+0000","last_change":"2024-08-19T09:51:22.638208+0000","last_active":"2024-08-19T10:09:20.632076+0000","last_peered":"2024-08-19T10:09:20.632076+0000","last_clean":"2024-08-19T10:09:20.632076+0000","last_became_active":"2024-08-19T09:51:22.634477+0000","last_became_peered":"2024-08-19T09:51:22.634477+0000","last_unstale":"2024-08-19T10:09:20.632076+0000","last_undegraded":"2024-08-19T10:09:20.632076+0000","last_fullsized":"2024-08-19T10:09:20.632076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:01:43.314792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628301+0000","last_change":"2024-08-19T09:51:22.594096+0000","last_active":"2024-08-19T10:09:20.628301+0000","last_peered":"2024-08-19T10:09:20.628301+0000","last_clean":"2024-08-19T10:09:20.628301+0000","last_became_active":"2024-08-19T09:51:22.590688+0000","last_became_peered":"2024-08-19T09:51:22.590688+0000","last_unstale":"2024-08-19T10:09:20.628301+0000","last_undegraded":"2024-08-19T10:09:20.628301+0000","last_fullsized":"2024-08-19T10:09:20.628301+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:51.718926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631182+0000","last_change":"2024-08-19T09:51:22.537238+0000","last_active":"2024-08-19T10:09:20.631182+0000","last_peered":"2024-08-19T10:09:20.631182+0000","last_clean":"2024-08-19T10:09:20.631182+0000","last_became_active":"2024-08-19T09:51:22.536584+0000","last_became_peered":"2024-08-19T09:51:22.536584+0000","last_unstale":"2024-08-19T10:09:20.631182+0000","last_undegraded":"2024-08-19T10:09:20.631182+0000","last_fullsized":"2024-08-19T10:09:20.631182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1377,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631530+0000","last_change":"2024-08-19T09:51:22.556447+0000","last_active":"2024-08-19T10:09:20.631530+0000","last_peered":"2024-08-19T10:09:20.631530+0000","last_clean":"2024-08-19T10:09:20.631530+0000","last_became_active":"2024-08-19T09:51:22.556182+0000","last_became_peered":"2024-08-19T09:51:22.556182+0000","last_unstale":"2024-08-19T10:09:20.631530+0000","last_undegraded":"2024-08-19T10:09:20.631530+0000","last_fullsized":"2024-08-19T10:09:20.631530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:04:06.155397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631467+0000","last_change":"2024-08-19T09:51:22.672202+0000","last_active":"2024-08-19T10:09:20.631467+0000","last_peered":"2024-08-19T10:09:20.631467+0000","last_clean":"2024-08-19T10:09:20.631467+0000","last_became_active":"2024-08-19T09:51:22.671918+0000","last_became_peered":"2024-08-19T09:51:22.671918+0000","last_unstale":"2024-08-19T10:09:20.631467+0000","last_undegraded":"2024-08-19T10:09:20.631467+0000","last_fullsized":"2024-08-19T10:09:20.631467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631211+0000","last_change":"2024-08-19T09:51:22.605834+0000","last_active":"2024-08-19T10:09:20.631211+0000","last_peered":"2024-08-19T10:09:20.631211+0000","last_clean":"2024-08-19T10:09:20.631211+0000","last_became_active":"2024-08-19T09:51:22.605586+0000","last_became_peered":"2024-08-19T09:51:22.605586+0000","last_unstale":"2024-08-19T10:09:20.631211+0000","last_undegraded":"2024-08-19T10:09:20.631211+0000","last_fullsized":"2024-08-19T10:09:20.631211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631661+0000","last_change":"2024-08-19T09:51:22.551390+0000","last_active":"2024-08-19T10:09:20.631661+0000","last_peered":"2024-08-19T10:09:20.631661+0000","last_clean":"2024-08-19T10:09:20.631661+0000","last_became_active":"2024-08-19T09:51:22.551119+0000","last_became_peered":"2024-08-19T09:51:22.551119+0000","last_unstale":"2024-08-19T10:09:20.631661+0000","last_undegraded":"2024-08-19T10:09:20.631661+0000","last_fullsized":"2024-08-19T10:09:20.631661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1384,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628146+0000","last_change":"2024-08-19T09:51:22.631476+0000","last_active":"2024-08-19T10:09:20.628146+0000","last_peered":"2024-08-19T10:09:20.628146+0000","last_clean":"2024-08-19T10:09:20.628146+0000","last_became_active":"2024-08-19T09:51:22.630612+0000","last_became_peered":"2024-08-19T09:51:22.630612+0000","last_unstale":"2024-08-19T10:09:20.628146+0000","last_undegraded":"2024-08-19T10:09:20.628146+0000","last_fullsized":"2024-08-19T10:09:20.628146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:15.371670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1377,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631498+0000","last_change":"2024-08-19T09:51:22.490285+0000","last_active":"2024-08-19T10:09:20.631498+0000","last_peered":"2024-08-19T10:09:20.631498+0000","last_clean":"2024-08-19T10:09:20.631498+0000","last_became_active":"2024-08-19T09:51:22.490018+0000","last_became_peered":"2024-08-19T09:51:22.490018+0000","last_unstale":"2024-08-19T10:09:20.631498+0000","last_undegraded":"2024-08-19T10:09:20.631498+0000","last_fullsized":"2024-08-19T10:09:20.631498+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:57:54.192511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1376,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632082+0000","last_change":"2024-08-19T09:51:22.646201+0000","last_active":"2024-08-19T10:09:20.632082+0000","last_peered":"2024-08-19T10:09:20.632082+0000","last_clean":"2024-08-19T10:09:20.632082+0000","last_became_active":"2024-08-19T09:51:22.644785+0000","last_became_peered":"2024-08-19T09:51:22.644785+0000","last_unstale":"2024-08-19T10:09:20.632082+0000","last_undegraded":"2024-08-19T10:09:20.632082+0000","last_fullsized":"2024-08-19T10:09:20.632082+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:30:51.586745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1376,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631888+0000","last_change":"2024-08-19T09:51:22.633554+0000","last_active":"2024-08-19T10:09:20.631888+0000","last_peered":"2024-08-19T10:09:20.631888+0000","last_clean":"2024-08-19T10:09:20.631888+0000","last_became_active":"2024-08-19T09:51:22.618248+0000","last_became_peered":"2024-08-19T09:51:22.618248+0000","last_unstale":"2024-08-19T10:09:20.631888+0000","last_undegraded":"2024-08-19T10:09:20.631888+0000","last_fullsized":"2024-08-19T10:09:20.631888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:04:45.586830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631208+0000","last_change":"2024-08-19T09:51:22.559064+0000","last_active":"2024-08-19T10:09:20.631208+0000","last_peered":"2024-08-19T10:09:20.631208+0000","last_clean":"2024-08-19T10:09:20.631208+0000","last_became_active":"2024-08-19T09:51:22.558815+0000","last_became_peered":"2024-08-19T09:51:22.558815+0000","last_unstale":"2024-08-19T10:09:20.631208+0000","last_undegraded":"2024-08-19T10:09:20.631208+0000","last_fullsized":"2024-08-19T10:09:20.631208+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1380,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631442+0000","last_change":"2024-08-19T09:51:22.582602+0000","last_active":"2024-08-19T10:09:20.631442+0000","last_peered":"2024-08-19T10:09:20.631442+0000","last_clean":"2024-08-19T10:09:20.631442+0000","last_became_active":"2024-08-19T09:51:22.581840+0000","last_became_peered":"2024-08-19T09:51:22.581840+0000","last_unstale":"2024-08-19T10:09:20.631442+0000","last_undegraded":"2024-08-19T10:09:20.631442+0000","last_fullsized":"2024-08-19T10:09:20.631442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1383,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631391+0000","last_change":"2024-08-19T09:55:22.701986+0000","last_active":"2024-08-19T10:09:20.631391+0000","last_peered":"2024-08-19T10:09:20.631391+0000","last_clean":"2024-08-19T10:09:20.631391+0000","last_became_active":"2024-08-19T09:55:22.701660+0000","last_became_peered":"2024-08-19T09:55:22.701660+0000","last_unstale":"2024-08-19T10:09:20.631391+0000","last_undegraded":"2024-08-19T10:09:20.631391+0000","last_fullsized":"2024-08-19T10:09:20.631391+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:56:19.752844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1376,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632018+0000","last_change":"2024-08-19T09:51:22.565151+0000","last_active":"2024-08-19T10:09:20.632018+0000","last_peered":"2024-08-19T10:09:20.632018+0000","last_clean":"2024-08-19T10:09:20.632018+0000","last_became_active":"2024-08-19T09:51:22.564919+0000","last_became_peered":"2024-08-19T09:51:22.564919+0000","last_unstale":"2024-08-19T10:09:20.632018+0000","last_undegraded":"2024-08-19T10:09:20.632018+0000","last_fullsized":"2024-08-19T10:09:20.632018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:09:05.826534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1384,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.631129+0000","last_change":"2024-08-19T09:51:22.612829+0000","last_active":"2024-08-19T10:09:20.631129+0000","last_peered":"2024-08-19T10:09:20.631129+0000","last_clean":"2024-08-19T10:09:20.631129+0000","last_became_active":"2024-08-19T09:51:22.612552+0000","last_became_peered":"2024-08-19T09:51:22.612552+0000","last_unstale":"2024-08-19T10:09:20.631129+0000","last_undegraded":"2024-08-19T10:09:20.631129+0000","last_fullsized":"2024-08-19T10:09:20.631129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:58:05.134880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1384,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.628549+0000","last_change":"2024-08-19T09:51:22.609929+0000","last_active":"2024-08-19T10:09:20.628549+0000","last_peered":"2024-08-19T10:09:20.628549+0000","last_clean":"2024-08-19T10:09:20.628549+0000","last_became_active":"2024-08-19T09:51:22.609674+0000","last_became_peered":"2024-08-19T09:51:22.609674+0000","last_unstale":"2024-08-19T10:09:20.628549+0000","last_undegraded":"2024-08-19T10:09:20.628549+0000","last_fullsized":"2024-08-19T10:09:20.628549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:41:23.025542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1378,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:20.632208+0000","last_change":"2024-08-19T09:51:22.652416+0000","last_active":"2024-08-19T10:09:20.632208+0000","last_peered":"2024-08-19T10:09:20.632208+0000","last_clean":"2024-08-19T10:09:20.632208+0000","last_became_active":"2024-08-19T09:51:22.652083+0000","last_became_peered":"2024-08-19T09:51:22.652083+0000","last_unstale":"2024-08-19T10:09:20.632208+0000","last_undegraded":"2024-08-19T10:09:20.632208+0000","last_fullsized":"2024-08-19T10:09:20.632208+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T09:51:21.222127+0000","last_clean_scrub_stamp":"2024-08-19T09:51:21.222127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:47.783611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575087,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73596,"kb_used_data":6312,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298244,"statfs":{"total":96636764160,"available":96561401856,"internally_reserved":0,"allocated":6463488,"data_stored":2760879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74120,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297720,"statfs":{"total":96636764160,"available":96560865280,"internally_reserved":0,"allocated":7000064,"data_stored":3289812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69448,"kb_used_data":6316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302392,"statfs":{"total":96636764160,"available":96565649408,"internally_reserved":0,"allocated":6467584,"data_stored":2760898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81928,"kb_used_data":6316,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289912,"statfs":{"total":96636764160,"available":96552869888,"internally_reserved":0,"allocated":6467584,"data_stored":2760884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607788,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69968,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301872,"statfs":{"total":96636764160,"available":96565116928,"internally_reserved":0,"allocated":6995968,"data_stored":3289807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73596,"kb_used_data":6316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298244,"statfs":{"total":96636764160,"available":96561401856,"internally_reserved":0,"allocated":6467584,"data_stored":2760898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-19T09:50:07.419178+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=7, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-19T09:50:14.111421+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=14, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-19T09:50:14.111421+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=14, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-19T09:50:14.111421+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=14, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-19T09:50:14.111421+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=14, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-19T09:50:14.111421+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=14, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.927 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-19T09:50:14.111421+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=50, tm_sec=14, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.1 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.0 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.3 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.2 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.5 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.4 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.7 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.6 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.928 INFO:tasks.ceph:pgid 33.9 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.8 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.b last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.a last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.d last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.c last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.f last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.e last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.11 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.929 INFO:tasks.ceph:pgid 33.10 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.13 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.12 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.15 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.14 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.17 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.16 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.19 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.18 last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.930 INFO:tasks.ceph:pgid 33.1b last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.931 INFO:tasks.ceph:pgid 33.1a last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.931 INFO:tasks.ceph:pgid 33.1d last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.931 INFO:tasks.ceph:pgid 33.1c last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.931 INFO:tasks.ceph:pgid 33.1f last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.931 INFO:tasks.ceph:pgid 33.1e last_scrub_stamp 2024-08-19T09:51:21.222127+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=9, tm_min=51, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=10, tm_min=9, tm_sec=23, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T10:09:27.931 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-19T10:09:36.266 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-08-19T10:09:36.267 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T10:09:36.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T10:09:36.270 INFO:tasks.ceph.mon.f:Started 2024-08-19T10:09:36.271 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T10:09:36.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:09:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.379+0000 7fb6dbff7640 10 monclient: get_monmap_and_config 2024-08-19T10:09:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.379+0000 7fb6dbff7640 10 monclient: build_initial_monmap 2024-08-19T10:09:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.379+0000 7fb6dbff7640 10 monclient: monmap: 2024-08-19T10:09:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:36.380260+0000 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:36.380260+0000 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:36.382 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.379+0000 7fb6dbff7640 1 Processor -- start 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 1 -- start start 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient: init 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient: _add_conns ranks=[4,2,3,6,1,7,0,5,8] 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d40a6830 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d40694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient(hunting): picked mon.noname-g con 0x7fb6d40a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d40a3c70 con 0x7fb6d40694b0 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d40a33d0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d40a36b0 con 0x7fb6d40a6830 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6dbff7640 10 monclient(hunting): authenticate will time out at 1832.456744s 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6da594640 1 --1- >> v1:172.21.15.82:6790/0 conn(0x7fb6d40a46a0 0x7fb6d4141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6790/0 says I am v1:172.21.15.1:33258/0 (socket says 172.21.15.1:33258) 2024-08-19T10:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.380+0000 7fb6da594640 1 -- 172.21.15.1:0/318852643 learned_addr learned my addr 172.21.15.1:0/318852643 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6da594640 1 -- 172.21.15.1:0/318852643 >> v1:172.21.15.82:6790/0 conn(0x7fb6d40a46a0 legacy=0x7fb6d4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 920450823 0 0) 0x7fb6d40a33d0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient(hunting): _negotiate 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient(hunting): my global_id is 214410 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6bc003520 con 0x7fb6d40a46a0 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3531399266 0 0) 0x7fb6bc003520 con 0x7fb6d40a46a0 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 >> v1:172.21.15.1:6791/0 conn(0x7fb6d40a6830 legacy=0x7fb6d40a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 >> v1:172.21.15.1:6790/0 conn(0x7fb6d40694b0 legacy=0x7fb6d40a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T10:09:36.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: _finish_hunting 0 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 monclient: found mon.noname-g 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.82:6790/0 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 --> v1:172.21.15.82:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d406dbb0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: _finish_auth 0 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: _check_auth_tickets 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb6c4001ae0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T10:09:36.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: dump: 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6dbff7640 5 monclient: authenticate success, global_id 214410 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6dbff7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 <== mon.3 v1:172.21.15.82:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6c40048a0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 1 -- 172.21.15.1:0/318852643 <== mon.3 v1:172.21.15.82:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb6c4004e10 con 0x7fb6d40a46a0 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6dbff7640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6dbff7640 4 set_mon_vals no callback set 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.383+0000 7fb6d8d91640 10 monclient: dump: 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 10 monclient: shutdown 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 1 -- 172.21.15.1:0/318852643 >> v1:172.21.15.82:6790/0 conn(0x7fb6d40a46a0 legacy=0x7fb6d4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 1 -- 172.21.15.1:0/318852643 shutdown_connections 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 1 -- 172.21.15.1:0/318852643 >> 172.21.15.1:0/318852643 conn(0x7fb6d40a9020 msgr2=0x7fb6d4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 1 -- 172.21.15.1:0/318852643 shutdown_connections 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 1 -- 172.21.15.1:0/318852643 wait complete. 2024-08-19T10:09:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 10 monclient: build_initial_monmap 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.384+0000 7fb6dbff7640 10 monclient: monmap: 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 librados: starting msgr at 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 librados: starting objecter 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 Processor -- start 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 -- start start 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 librados: setting wanted keys 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 librados: calling monclient init 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient: init 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient: _add_conns ranks=[4,3,0,6,2,8,1,5,7] 2024-08-19T10:09:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d40694b0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient(hunting): picked mon.noname-g con 0x7fb6d40a46a0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d40a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d4138890 con 0x7fb6d40a6830 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d40ab9c0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d4136530 con 0x7fb6d40694b0 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6dbff7640 10 monclient(hunting): authenticate will time out at 1832.461791s 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6da594640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb6d40a6830 0x7fb6d41d64a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43400/0 (socket says 172.21.15.1:43400) 2024-08-19T10:09:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.385+0000 7fb6da594640 1 -- 172.21.15.1:0/1674901837 learned_addr learned my addr 172.21.15.1:0/1674901837 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6da594640 1 -- 172.21.15.1:0/1674901837 >> v1:172.21.15.1:6789/0 conn(0x7fb6d40a6830 legacy=0x7fb6d41d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 client.?.objecter ms_handle_connect 0x7fb6d40a6830 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 client.?.objecter ms_handle_connect 0x7fb6d40a46a0 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d9592640 1 -- 172.21.15.1:0/1674901837 >> v1:172.21.15.82:6790/0 conn(0x7fb6d40a46a0 legacy=0x7fb6d41d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3657755230 0 0) 0x7fb6d4138890 con 0x7fb6d40a6830 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): my global_id is 204660 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c0004eb0 con 0x7fb6d40a6830 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283098105 0 0) 0x7fb6d40ab9c0 con 0x7fb6d40a46a0 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): my global_id is 214419 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6d4138890 con 0x7fb6d40a46a0 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2582706319 0 0) 0x7fb6c0004eb0 con 0x7fb6d40a6830 2024-08-19T10:09:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6d40ab9c0 con 0x7fb6d40a6830 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb6c4004d90 con 0x7fb6d40a6830 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): dump: 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 371365798 0 0) 0x7fb6d40ab9c0 con 0x7fb6d40a6830 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 >> v1:172.21.15.1:6791/0 conn(0x7fb6d40694b0 legacy=0x7fb6d4141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 >> v1:172.21.15.82:6790/0 conn(0x7fb6d40a46a0 legacy=0x7fb6d41d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 monclient: found mon.a 2024-08-19T10:09:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d4137ff0 con 0x7fb6d40a6830 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 5 monclient: authenticate success, global_id 204660 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 10 monclient: _renew_subs 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d41caef0 con 0x7fb6d40a6830 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 10 client.204660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 10 monclient: _renew_subs 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6d41ca850 con 0x7fb6d40a6830 2024-08-19T10:09:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.388+0000 7fb6dbff7640 1 librados: init done 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6c40048a0 con 0x7fb6d40a6830 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fb6c4001a80 con 0x7fb6d40a6830 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d3fff640 4 set_mon_vals no callback set 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6d27fc640 10 monclient: dump: 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6dbff7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6dbff7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6dbff7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.389+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b4005080 con 0x7fb6d40a6830 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fb6c4011a00 con 0x7fb6d40a6830 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fb6c404e010 con 0x7fb6d40a6830 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d27fc640 10 client.204660.objecter ms_dispatch 0x7fb6d4000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d27fc640 3 client.204660.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d27fc640 3 client.204660.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d27fc640 20 client.204660.objecter dump_active .. 0 homeless 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.390+0000 7fb6d9d93640 10 monclient: get_auth_request con 0x7fb6c0040fc0 auth_method 0 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.392+0000 7fb6d27fc640 10 client.204660.objecter ms_handle_connect 0x7fb6c0040fc0 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.394+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb6d41ca850 con 0x7fb6d40a6830 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.394+0000 7fb6d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.394+0000 7fb6d27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.651+0000 7fb6dbff7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.651+0000 7fb6dbff7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.651+0000 7fb6dbff7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.651+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb6b4005370 con 0x7fb6d40a6830 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.652+0000 7fb6d27fc640 1 -- 172.21.15.1:0/1674901837 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2698 (unknown 254746657 0 114447854) 0x7fb6c4017670 con 0x7fb6d40a6830 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.652+0000 7fb6d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:09:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.652+0000 7fb6d27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:36.654 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":146,"quorum":[0,2,3,4,5,6,7,8],"quorum_names":["a","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":93,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 10 librados: watch_flush enter 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 10 librados: watch_flush exit 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 20 client.204660.objecter shutdown clearing up homeless session... 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 10 client.204660.objecter successfully canceled tick 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 >> v1:172.21.15.82:6812/2549104415 conn(0x7fb6c0040fc0 legacy=0x7fb6c0043480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 10 monclient: shutdown 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 >> v1:172.21.15.1:6789/0 conn(0x7fb6d40a6830 legacy=0x7fb6d41d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 shutdown_connections 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 >> 172.21.15.1:0/1674901837 conn(0x7fb6d40a9020 msgr2=0x7fb6d40a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 shutdown_connections 2024-08-19T10:09:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 -- 172.21.15.1:0/1674901837 wait complete. 2024-08-19T10:09:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:36.654+0000 7fb6dbff7640 1 librados: shutdown 2024-08-19T10:09:39.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.771+0000 7fe488121640 10 monclient: get_monmap_and_config 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.771+0000 7fe488121640 10 monclient: build_initial_monmap 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.771+0000 7fe488121640 10 monclient: monmap: 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:39.772883+0000 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:39.772883+0000 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:39.774 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.772+0000 7fe488121640 1 Processor -- start 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.772+0000 7fe488121640 1 -- start start 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.772+0000 7fe488121640 10 monclient: init 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient: _add_conns ranks=[5,7,8,6,0,2,4,3,1] 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient(hunting): picked mon.noname-h con 0x7fe4800a6720 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient(hunting): picked mon.noname-i con 0x7fe4800a4590 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient(hunting): picked mon.noname-e con 0x7fe48013f730 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4800a32c0 con 0x7fe4800a6720 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4800a35a0 con 0x7fe4800a4590 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4800a3880 con 0x7fe48013f730 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe488121640 10 monclient(hunting): authenticate will time out at 1835.849406s 2024-08-19T10:09:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe4866be640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fe48013f730 0x7fe480141b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:57142/0 (socket says 172.21.15.1:57142) 2024-08-19T10:09:39.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.773+0000 7fe4866be640 1 -- 172.21.15.1:0/2680656750 learned_addr learned my addr 172.21.15.1:0/2680656750 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe4866be640 1 -- 172.21.15.1:0/2680656750 >> v1:172.21.15.1:6793/0 conn(0x7fe48013f730 legacy=0x7fe480141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe4856bc640 1 -- 172.21.15.1:0/2680656750 >> v1:172.21.15.82:6792/0 conn(0x7fe4800a4590 legacy=0x7fe4800a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe485ebd640 1 -- 172.21.15.1:0/2680656750 >> v1:172.21.15.82:6791/0 conn(0x7fe4800a6720 legacy=0x7fe4800a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278664230 0 0) 0x7fe4800a3880 con 0x7fe48013f730 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): my global_id is 204722 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe468003520 con 0x7fe48013f730 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355853532 0 0) 0x7fe4800a35a0 con 0x7fe4800a4590 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): my global_id is 204676 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4800a3880 con 0x7fe4800a4590 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922484488 0 0) 0x7fe4780014a0 con 0x7fe4800a6720 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 10 monclient(hunting): my global_id is 214466 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.775+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4800a35a0 con 0x7fe4800a6720 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2027221857 0 0) 0x7fe468003520 con 0x7fe48013f730 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 >> v1:172.21.15.82:6792/0 conn(0x7fe4800a4590 legacy=0x7fe4800a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 >> v1:172.21.15.82:6791/0 conn(0x7fe4800a6720 legacy=0x7fe4800a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 monclient: found mon.noname-e 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.1:6793/0 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4800a92b0 con 0x7fe48013f730 2024-08-19T10:09:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: _finish_auth 0 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe470001e60 con 0x7fe48013f730 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 5 monclient: authenticate success, global_id 204722 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: dump: 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.8 v1:172.21.15.1:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe470004c20 con 0x7fe48013f730 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 1 -- 172.21.15.1:0/2680656750 <== mon.8 v1:172.21.15.1:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe470005190 con 0x7fe48013f730 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 4 set_mon_vals no callback set 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe484ebb640 10 monclient: dump: 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 10 monclient: shutdown 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 1 -- 172.21.15.1:0/2680656750 >> v1:172.21.15.1:6793/0 conn(0x7fe48013f730 legacy=0x7fe480141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 1 -- 172.21.15.1:0/2680656750 shutdown_connections 2024-08-19T10:09:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.776+0000 7fe488121640 1 -- 172.21.15.1:0/2680656750 >> 172.21.15.1:0/2680656750 conn(0x7fe4800ac410 msgr2=0x7fe4800a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 1 -- 172.21.15.1:0/2680656750 shutdown_connections 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 1 -- 172.21.15.1:0/2680656750 wait complete. 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 10 monclient: build_initial_monmap 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 10 monclient: monmap: 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 1 librados: starting msgr at 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 1 librados: starting objecter 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.777+0000 7fe488121640 1 Processor -- start 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 1 -- start start 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 1 librados: setting wanted keys 2024-08-19T10:09:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 1 librados: calling monclient init 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient: init 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient: _add_conns ranks=[5,1,6,4,2,7,8,3,0] 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient(hunting): picked mon.noname-h con 0x7fe4800a4590 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient(hunting): picked mon.noname-f con 0x7fe4800a6720 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient(hunting): picked mon.noname-d con 0x7fe48013f730 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe48013da40 con 0x7fe4800a6720 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4800a2550 con 0x7fe4800a4590 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4800a2830 con 0x7fe48013f730 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe488121640 10 monclient(hunting): authenticate will time out at 1835.854548s 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe4866be640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fe48013f730 0x7fe4801d6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34452/0 (socket says 172.21.15.1:34452) 2024-08-19T10:09:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.778+0000 7fe4866be640 1 -- 172.21.15.1:0/3697939825 learned_addr learned my addr 172.21.15.1:0/3697939825 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe4866be640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.1:6792/0 conn(0x7fe48013f730 legacy=0x7fe4801d6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 client.?.objecter ms_handle_connect 0x7fe48013f730 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 client.?.objecter ms_handle_connect 0x7fe4800a4590 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe485ebd640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.82:6791/0 conn(0x7fe4800a4590 legacy=0x7fe4801ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 client.?.objecter ms_handle_connect 0x7fe4800a6720 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe4856bc640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.82:6789/0 conn(0x7fe4800a6720 legacy=0x7fe4801d36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 497590993 0 0) 0x7fe4800a2830 con 0x7fe48013f730 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.780+0000 7fe47e7fc640 10 monclient(hunting): my global_id is 204522 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe46c004ee0 con 0x7fe48013f730 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2508997741 0 0) 0x7fe4800a2550 con 0x7fe4800a4590 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): my global_id is 214475 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4800a2830 con 0x7fe4800a4590 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2685916510 0 0) 0x7fe46c004ee0 con 0x7fe48013f730 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4800a2550 con 0x7fe48013f730 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe4700020b0 con 0x7fe48013f730 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): dump: 2024-08-19T10:09:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2385151706 0 0) 0x7fe4800a2830 con 0x7fe4800a4590 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe46c004ee0 con 0x7fe4800a4590 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe478003950 con 0x7fe4800a4590 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient(hunting): dump: 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 644068289 0 0) 0x7fe4800a2550 con 0x7fe48013f730 2024-08-19T10:09:39.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.82:6791/0 conn(0x7fe4800a4590 legacy=0x7fe4801ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.82:6789/0 conn(0x7fe4800a6720 legacy=0x7fe4801d36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 monclient: found mon.d 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4800a0a90 con 0x7fe48013f730 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 5 monclient: authenticate success, global_id 204522 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 10 monclient: _renew_subs 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4800a1370 con 0x7fe48013f730 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 10 client.204522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 10 monclient: _renew_subs 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4800a0280 con 0x7fe48013f730 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe488121640 1 librados: init done 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe470004c20 con 0x7fe48013f730 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fe470005700 con 0x7fe48013f730 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47ffff640 4 set_mon_vals no callback set 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.781+0000 7fe47e7fc640 10 monclient: dump: 2024-08-19T10:09:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:39.802 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:39.802 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.802 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.782+0000 7fe488121640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.782+0000 7fe488121640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.782+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fe470011b60 con 0x7fe48013f730 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.782+0000 7fe488121640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.782+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe460005080 con 0x7fe48013f730 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.783+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fe47004e1b0 con 0x7fe48013f730 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.783+0000 7fe47e7fc640 10 client.204522.objecter ms_dispatch 0x7fe480000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.783+0000 7fe47e7fc640 3 client.204522.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.783+0000 7fe47e7fc640 3 client.204522.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.783+0000 7fe47e7fc640 20 client.204522.objecter dump_active .. 0 homeless 2024-08-19T10:09:39.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.783+0000 7fe485ebd640 10 monclient: get_auth_request con 0x7fe46c0417a0 auth_method 0 2024-08-19T10:09:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.785+0000 7fe47e7fc640 10 client.204522.objecter ms_handle_connect 0x7fe46c0417a0 2024-08-19T10:09:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.791+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe470017890 con 0x7fe48013f730 2024-08-19T10:09:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.791+0000 7fe47e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:39.791+0000 7fe47e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe488121640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T10:09:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe488121640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T10:09:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe488121640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe460005370 con 0x7fe48013f730 2024-08-19T10:09:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe47e7fc640 1 -- 172.21.15.1:0/3697939825 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2703 (unknown 254746657 0 2182118418) 0x7fe470017190 con 0x7fe48013f730 2024-08-19T10:09:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe47e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T10:09:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.049+0000 7fe47e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:40.051 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":152,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0e9f827b-9129-41b7-89ad-24f744d49b77","modified":"2024-08-19T09:49:28.615648Z","created":"2024-08-19T09:49:28.615648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6790","nonce":0}]},"addr":"172.21.15.1:6790/0","public_addr":"172.21.15.1:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6791","nonce":0}]},"addr":"172.21.15.1:6791/0","public_addr":"172.21.15.1:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6791","nonce":0}]},"addr":"172.21.15.82:6791/0","public_addr":"172.21.15.82:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6792","nonce":0}]},"addr":"172.21.15.1:6792/0","public_addr":"172.21.15.1:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6792","nonce":0}]},"addr":"172.21.15.82:6792/0","public_addr":"172.21.15.82:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6793","nonce":0}]},"addr":"172.21.15.1:6793/0","public_addr":"172.21.15.1:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 10 librados: watch_flush enter 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 10 librados: watch_flush exit 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 20 client.204522.objecter shutdown clearing up homeless session... 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 10 client.204522.objecter successfully canceled tick 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.82:6812/2549104415 conn(0x7fe46c0417a0 legacy=0x7fe46c043c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 10 monclient: shutdown 2024-08-19T10:09:40.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 >> v1:172.21.15.1:6792/0 conn(0x7fe48013f730 legacy=0x7fe4801d6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 shutdown_connections 2024-08-19T10:09:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.051+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 >> 172.21.15.1:0/3697939825 conn(0x7fe4800ac410 msgr2=0x7fe4800a4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.052+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 shutdown_connections 2024-08-19T10:09:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.052+0000 7fe488121640 1 -- 172.21.15.1:0/3697939825 wait complete. 2024-08-19T10:09:40.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.052+0000 7fe488121640 1 librados: shutdown 2024-08-19T10:09:40.063 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T10:09:40.063 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T10:09:40.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T10:09:40.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.162+0000 7f3d2f4c5640 10 monclient: get_monmap_and_config 2024-08-19T10:09:40.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.162+0000 7f3d2f4c5640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.162+0000 7f3d2f4c5640 10 monclient: monmap: 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:40.163108+0000 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:40.163108+0000 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.164 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.165 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.165 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.162+0000 7f3d2f4c5640 1 Processor -- start 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.162+0000 7f3d2f4c5640 1 -- start start 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.162+0000 7f3d2f4c5640 10 monclient: init 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient: _add_conns ranks=[0,1,6,4,2,3,8,7,5] 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f3d280a6830 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient(hunting): picked mon.noname-f con 0x7f3d280694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:40.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient(hunting): picked mon.noname-d con 0x7f3d280a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d280a3c70 con 0x7f3d280a6830 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d280a33d0 con 0x7f3d280694b0 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d280a36b0 con 0x7f3d280a46a0 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2f4c5640 10 monclient(hunting): authenticate will time out at 1836.239490s 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2d261640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3d280a6830 0x7f3d280a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60510/0 (socket says 172.21.15.1:60510) 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2d261640 1 -- 172.21.15.1:0/1029264241 learned_addr learned my addr 172.21.15.1:0/1029264241 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.163+0000 7f3d2da62640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f3d280a46a0 0x7f3d28141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34476/0 (socket says 172.21.15.1:34476) 2024-08-19T10:09:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d2d261640 1 -- 172.21.15.1:0/1029264241 >> v1:172.21.15.1:6789/0 conn(0x7f3d280a6830 legacy=0x7f3d280a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d2da62640 1 -- 172.21.15.1:0/1029264241 >> v1:172.21.15.1:6792/0 conn(0x7f3d280a46a0 legacy=0x7f3d28141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578817248 0 0) 0x7f3d280a3c70 con 0x7f3d280a6830 2024-08-19T10:09:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): my global_id is 204669 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d14003520 con 0x7f3d280a6830 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1930162052 0 0) 0x7f3d280a36b0 con 0x7f3d280a46a0 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 10 monclient(hunting): my global_id is 204531 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.165+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d280a3c70 con 0x7f3d280a46a0 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2ca60640 1 -- 172.21.15.1:0/1029264241 >> v1:172.21.15.82:6789/0 conn(0x7f3d280694b0 legacy=0x7f3d280a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 472943028 0 0) 0x7f3d280a3c70 con 0x7f3d280a46a0 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 >> v1:172.21.15.82:6789/0 conn(0x7f3d280694b0 legacy=0x7f3d280a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 >> v1:172.21.15.1:6789/0 conn(0x7f3d280a6830 legacy=0x7f3d280a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 monclient: found mon.noname-d 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2806dbb0 con 0x7f3d280a46a0 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2f4c5640 5 monclient: authenticate success, global_id 204531 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d20001e40 con 0x7f3d280a46a0 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2f4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: dump: 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2f4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2da62640 1 --1- 172.21.15.1:0/1029264241 >> v1:172.21.15.1:6792/0 conn(0x7f3d280a46a0 0x7f3d28141850 :-1 s=OPENED pgs=3654 cs=1 l=1).queue_received will delay after 1.72406e+09 on 0x7f3d200050d0 mon_map magic: 0 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d20004b60 con 0x7f3d280a46a0 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d1ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2f4c5640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.166+0000 7f3d2f4c5640 4 set_mon_vals no callback set 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 10 monclient: shutdown 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/1029264241 >> v1:172.21.15.1:6792/0 conn(0x7f3d280a46a0 legacy=0x7f3d28141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1029264241 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d200050d0 con 0x7f3d280a46a0 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d1ffff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/1029264241 shutdown_connections 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/1029264241 >> 172.21.15.1:0/1029264241 conn(0x7f3d280a9020 msgr2=0x7f3d28068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/1029264241 shutdown_connections 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/1029264241 wait complete. 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.167+0000 7f3d2f4c5640 10 monclient: monmap: 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 librados: starting msgr at 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 librados: starting objecter 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 Processor -- start 2024-08-19T10:09:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 -- start start 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 librados: setting wanted keys 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 librados: calling monclient init 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient: init 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient: _add_conns ranks=[8,5,3,7,2,0,6,4,1] 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient(hunting): picked mon.noname-e con 0x7f3d280694b0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient(hunting): picked mon.noname-h con 0x7f3d280a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient(hunting): picked mon.noname-g con 0x7f3d280a6830 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d28137e70 con 0x7f3d280a6830 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d28139100 con 0x7f3d280a46a0 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d28138440 con 0x7f3d280694b0 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2f4c5640 10 monclient(hunting): authenticate will time out at 1836.244711s 2024-08-19T10:09:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2d261640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f3d280694b0 0x7f3d281350f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:57156/0 (socket says 172.21.15.1:57156) 2024-08-19T10:09:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.168+0000 7f3d2d261640 1 -- 172.21.15.1:0/3482707677 learned_addr learned my addr 172.21.15.1:0/3482707677 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.170+0000 7f3d2d261640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6793/0 conn(0x7f3d280694b0 legacy=0x7f3d281350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.170+0000 7f3d1dffb640 10 client.?.objecter ms_handle_connect 0x7f3d280694b0 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.170+0000 7f3d1dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.170+0000 7f3d1dffb640 10 client.?.objecter ms_handle_connect 0x7f3d280a6830 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.170+0000 7f3d1dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.170+0000 7f3d2da62640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6790/0 conn(0x7f3d280a6830 legacy=0x7f3d281d6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2ca60640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6791/0 conn(0x7f3d280a46a0 legacy=0x7f3d28135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565247227 0 0) 0x7f3d240015e0 con 0x7f3d280694b0 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): my global_id is 204731 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d0c004eb0 con 0x7f3d280694b0 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 client.?.objecter ms_handle_connect 0x7f3d280a46a0 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771468839 0 0) 0x7f3d28137e70 con 0x7f3d280a6830 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): my global_id is 214428 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d240015e0 con 0x7f3d280a6830 2024-08-19T10:09:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741661322 0 0) 0x7f3d280a33d0 con 0x7f3d280a46a0 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): my global_id is 214484 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d28137e70 con 0x7f3d280a46a0 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3596235663 0 0) 0x7f3d0c004eb0 con 0x7f3d280694b0 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d280a33d0 con 0x7f3d280694b0 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d24001d30 con 0x7f3d280694b0 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): dump: 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.186 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.187 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2da62640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6790/0 conn(0x7f3d280a6830 legacy=0x7f3d281d6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2da62640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6790/0 conn(0x7f3d280a6830 legacy=0x7f3d281d6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2da62640 1 --1- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6790/0 conn(0x7f3d280a6830 0x7f3d281d6580 :-1 s=OPENED pgs=817 cs=1 l=1).handle_message read tag failed 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1183355892 0 0) 0x7f3d280a33d0 con 0x7f3d280694b0 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2da62640 1 --1- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6790/0 conn(0x7f3d280a6830 0x7f3d281d6580 :-1 s=OPENED pgs=817 cs=1 l=1).fault on lossy channel, failing 2024-08-19T10:09:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6791/0 conn(0x7f3d280a46a0 legacy=0x7f3d28135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6790/0 conn(0x7f3d280a6830 legacy=0x7f3d281d6580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 monclient: found mon.e 2024-08-19T10:09:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d28136980 con 0x7f3d280694b0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: ms_handle_reset stray mon v1:172.21.15.82:6790/0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 5 monclient: authenticate success, global_id 204731 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 10 monclient: _renew_subs 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d28135f10 con 0x7f3d280694b0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d24004dd0 con 0x7f3d280694b0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d24005340 con 0x7f3d280694b0 2024-08-19T10:09:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1dffb640 10 monclient: dump: 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d1f7fe640 4 set_mon_vals no callback set 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 10 client.204731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 10 monclient: _renew_subs 2024-08-19T10:09:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d281374e0 con 0x7f3d280694b0 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.171+0000 7f3d2f4c5640 1 librados: init done 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d2f4c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d280abc50 con 0x7f3d28143180 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f3d240055e0 con 0x7f3d280694b0 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f3d2404e130 con 0x7f3d280694b0 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d1dffb640 10 client.204731.objecter ms_dispatch 0x7f3d28000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d1dffb640 3 client.204731.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.172+0000 7f3d1dffb640 3 client.204731.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.173+0000 7f3d1dffb640 20 client.204731.objecter dump_active .. 0 homeless 2024-08-19T10:09:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.173+0000 7f3d2da62640 10 monclient: get_auth_request con 0x7f3d0c041270 auth_method 0 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.174+0000 7f3d2ca60640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6789/0 conn(0x7f3d28143180 legacy=0x7f3d2809e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.174+0000 7f3d1dffb640 10 client.204731.objecter ms_handle_connect 0x7f3d28143180 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.174+0000 7f3d1dffb640 10 client.204731.objecter resend_mon_ops 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922658298 0 0) 0x7f3d28137e70 con 0x7f3d28143180 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient(hunting): my global_id is 204678 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d280a33d0 con 0x7f3d28143180 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1964656321 0 0) 0x7f3d280a33d0 con 0x7f3d28143180 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d28137e70 con 0x7f3d28143180 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d18001cb0 con 0x7f3d28143180 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient: dump: 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 70727252 0 0) 0x7f3d28137e70 con 0x7f3d28143180 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d2806be10 con 0x7f3d28143180 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 client.204731.objecter ms_handle_connect 0x7f3d0c041270 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3d180057c0 con 0x7f3d28143180 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6789/0 conn(0x7f3d28143180 legacy=0x7f3d2809e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.175+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6789/0 conn(0x7f3d28143180 legacy=0x7f3d2809e640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.181+0000 7f3d2f4c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:40.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.181+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d28138d90 con 0x7f3d2802e1d0 2024-08-19T10:09:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d2ca60640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6789/0 conn(0x7f3d2802e1d0 legacy=0x7f3d2809ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 client.204731.objecter ms_handle_connect 0x7f3d2802e1d0 2024-08-19T10:09:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 client.204731.objecter resend_mon_ops 2024-08-19T10:09:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037229241 0 0) 0x7f3d28138d90 con 0x7f3d2802e1d0 2024-08-19T10:09:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient(hunting): my global_id is 204687 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d28137e70 con 0x7f3d2802e1d0 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2282345264 0 0) 0x7f3d28137e70 con 0x7f3d2802e1d0 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d28138d90 con 0x7f3d2802e1d0 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f3d18003ce0 con 0x7f3d2802e1d0 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient: dump: 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3032480406 0 0) 0x7f3d28138d90 con 0x7f3d2802e1d0 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 --> v1:172.21.15.1:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d28072fb0 con 0x7f3d2802e1d0 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 <== mon.0 v1:172.21.15.1:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7854 (unknown 0 0 252177725) 0x7f3d18003fd0 con 0x7f3d2802e1d0 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6789/0 conn(0x7f3d2802e1d0 legacy=0x7f3d2809ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.184+0000 7f3d1dffb640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6789/0 conn(0x7f3d2802e1d0 legacy=0x7f3d2809ed50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:40.201 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 246055, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.203 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.204 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.205 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.206 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:40.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:40.209 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:40.210 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.211 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.212 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.215 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.216 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.218 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.219 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.220 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.221 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:40.222 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 10 librados: watch_flush enter 2024-08-19T10:09:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 10 librados: watch_flush exit 2024-08-19T10:09:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 20 client.204731.objecter shutdown clearing up homeless session... 2024-08-19T10:09:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 10 client.204731.objecter successfully canceled tick 2024-08-19T10:09:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.82:6812/2549104415 conn(0x7f3d0c041270 legacy=0x7f3d0c043730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 10 monclient: shutdown 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 >> v1:172.21.15.1:6793/0 conn(0x7f3d280694b0 legacy=0x7f3d281350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2da62640 1 -- 172.21.15.1:0/3482707677 reap_dead start 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2da62640 1 -- 172.21.15.1:0/3482707677 reap_dead start 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 shutdown_connections 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 >> 172.21.15.1:0/3482707677 conn(0x7f3d280a9020 msgr2=0x7f3d280a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 shutdown_connections 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 -- 172.21.15.1:0/3482707677 wait complete. 2024-08-19T10:09:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.185+0000 7f3d2f4c5640 1 librados: shutdown 2024-08-19T10:09:40.226 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:40.226 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T10:09:40.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T10:09:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.324+0000 7f9daad06640 10 monclient: get_monmap_and_config 2024-08-19T10:09:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.324+0000 7f9daad06640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.324+0000 7f9daad06640 10 monclient: monmap: 2024-08-19T10:09:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.326 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:40.325655+0000 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:40.325655+0000 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.327 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.328 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.328 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.328 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 1 Processor -- start 2024-08-19T10:09:40.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 1 -- start start 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 10 monclient: init 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 10 monclient: _add_conns ranks=[0,4,1,8,6,7,5,2,3] 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 10 monclient(hunting): picked mon.noname-a con 0x7f9da40a67c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 10 monclient(hunting): picked mon.noname-c con 0x7f9da40a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 10 monclient(hunting): picked mon.noname-f con 0x7f9da413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.325+0000 7f9daad06640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da40a3b40 con 0x7f9da40a67c0 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9daad06640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da40a8130 con 0x7f9da413f5c0 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9daad06640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da406ede0 con 0x7f9da40a6ba0 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9daad06640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9daad06640 10 monclient(hunting): authenticate will time out at 1836.402032s 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9da8aa2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9da40a67c0 0x7f9da40a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60534/0 (socket says 172.21.15.1:60534) 2024-08-19T10:09:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9da3fff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f9da40a6ba0 0x7f9da40a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:58066/0 (socket says 172.21.15.1:58066) 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.326+0000 7f9da8aa2640 1 -- 172.21.15.1:0/1343941197 learned_addr learned my addr 172.21.15.1:0/1343941197 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da8aa2640 1 -- 172.21.15.1:0/1343941197 >> v1:172.21.15.1:6789/0 conn(0x7f9da40a67c0 legacy=0x7f9da40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da3fff640 1 -- 172.21.15.1:0/1343941197 >> v1:172.21.15.1:6791/0 conn(0x7f9da40a6ba0 legacy=0x7f9da40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056655220 0 0) 0x7f9da40a3b40 con 0x7f9da40a67c0 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): my global_id is 204696 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d94003520 con 0x7f9da40a67c0 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1829149460 0 0) 0x7f9da406ede0 con 0x7f9da40a6ba0 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient(hunting): my global_id is 194899 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da40a3b40 con 0x7f9da40a6ba0 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da92a3640 1 -- 172.21.15.1:0/1343941197 >> v1:172.21.15.82:6789/0 conn(0x7f9da413f5c0 legacy=0x7f9da41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2105151931 0 0) 0x7f9da40a3b40 con 0x7f9da40a6ba0 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 >> v1:172.21.15.82:6789/0 conn(0x7f9da413f5c0 legacy=0x7f9da41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 >> v1:172.21.15.1:6789/0 conn(0x7f9da40a67c0 legacy=0x7f9da40a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 monclient: found mon.noname-c 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da40a92a0 con 0x7f9da40a6ba0 2024-08-19T10:09:40.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d90001940 con 0x7f9da40a6ba0 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9daad06640 5 monclient: authenticate success, global_id 194899 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9daad06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: dump: 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.336 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9daad06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d90001c10 con 0x7f9da40a6ba0 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9daad06640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 1 -- 172.21.15.1:0/1343941197 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d90004b20 con 0x7f9da40a6ba0 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9daad06640 4 set_mon_vals no callback set 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.328+0000 7f9da37fe640 10 monclient: dump: 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.338 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 10 monclient: shutdown 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 1 -- 172.21.15.1:0/1343941197 >> v1:172.21.15.1:6791/0 conn(0x7f9da40a6ba0 legacy=0x7f9da40a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 1 -- 172.21.15.1:0/1343941197 shutdown_connections 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 1 -- 172.21.15.1:0/1343941197 >> 172.21.15.1:0/1343941197 conn(0x7f9da40ac4b0 msgr2=0x7f9da40a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 1 -- 172.21.15.1:0/1343941197 shutdown_connections 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 1 -- 172.21.15.1:0/1343941197 wait complete. 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.329+0000 7f9daad06640 10 monclient: monmap: 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.340 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.341 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 librados: starting msgr at 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 librados: starting objecter 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 Processor -- start 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 -- start start 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 librados: setting wanted keys 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 librados: calling monclient init 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient: init 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient: _add_conns ranks=[4,2,1,8,5,7,6,0,3] 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient(hunting): picked mon.noname-c con 0x7f9da40a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient(hunting): picked mon.noname-b con 0x7f9da40a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient(hunting): picked mon.noname-f con 0x7f9da413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da4139ce0 con 0x7f9da413f5c0 2024-08-19T10:09:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da413a830 con 0x7f9da40a6ba0 2024-08-19T10:09:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da413f310 con 0x7f9da40a67c0 2024-08-19T10:09:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9daad06640 10 monclient(hunting): authenticate will time out at 1836.406918s 2024-08-19T10:09:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9da8aa2640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f9da40a67c0 0x7f9da41ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:58074/0 (socket says 172.21.15.1:58074) 2024-08-19T10:09:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.330+0000 7f9da8aa2640 1 -- 172.21.15.1:0/4100951334 learned_addr learned my addr 172.21.15.1:0/4100951334 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da8aa2640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6791/0 conn(0x7f9da40a67c0 legacy=0x7f9da41ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 client.?.objecter ms_handle_connect 0x7f9da40a67c0 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 client.?.objecter ms_handle_connect 0x7f9da40a6ba0 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da3fff640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9da40a6ba0 legacy=0x7f9da41d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4065264410 0 0) 0x7f9da413f310 con 0x7f9da40a67c0 2024-08-19T10:09:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): my global_id is 194908 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d88004eb0 con 0x7f9da40a67c0 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975144196 0 0) 0x7f9da413a830 con 0x7f9da40a6ba0 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): my global_id is 185078 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da413f310 con 0x7f9da40a6ba0 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da92a3640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.82:6789/0 conn(0x7f9da413f5c0 legacy=0x7f9da41d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 client.?.objecter ms_handle_connect 0x7f9da413f5c0 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3948579637 0 0) 0x7f9d88004eb0 con 0x7f9da40a67c0 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9da413a830 con 0x7f9da40a67c0 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d980047e0 con 0x7f9da40a67c0 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): dump: 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3499508185 0 0) 0x7f9da413f310 con 0x7f9da40a6ba0 2024-08-19T10:09:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d88004eb0 con 0x7f9da40a6ba0 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d90004b20 con 0x7f9da40a6ba0 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient(hunting): dump: 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2166056532 0 0) 0x7f9da413a830 con 0x7f9da40a67c0 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9da40a6ba0 legacy=0x7f9da41d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.82:6789/0 conn(0x7f9da413f5c0 legacy=0x7f9da41d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 monclient: found mon.c 2024-08-19T10:09:40.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da40a9d30 con 0x7f9da40a67c0 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 5 monclient: authenticate success, global_id 194908 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 10 monclient: _renew_subs 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9da4142bf0 con 0x7f9da40a67c0 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 10 client.194908.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 10 monclient: _renew_subs 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6791/0 2024-08-19T10:09:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9da40a0d30 con 0x7f9da40a67c0 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9daad06640 1 librados: init done 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d98004cb0 con 0x7f9da40a67c0 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d98005220 con 0x7f9da40a67c0 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da2ffd640 4 set_mon_vals no callback set 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.333+0000 7f9da17fa640 10 monclient: dump: 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.356 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f9d980054c0 con 0x7f9da40a67c0 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9daad06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9d840065c0 con 0x7f9d84002860 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.4 v1:172.21.15.1:6791/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f9d9804e0a0 con 0x7f9da40a67c0 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9da17fa640 10 client.194908.objecter ms_dispatch 0x7f9da4000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9da17fa640 3 client.194908.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.334+0000 7f9da17fa640 3 client.194908.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.335+0000 7f9da17fa640 20 client.194908.objecter dump_active .. 0 homeless 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.335+0000 7f9da92a3640 10 monclient: get_auth_request con 0x7f9d88041540 auth_method 0 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da3fff640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9d84002860 legacy=0x7f9d84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 client.194908.objecter ms_handle_connect 0x7f9d84002860 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 client.194908.objecter resend_mon_ops 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1128357287 0 0) 0x7f9d840065c0 con 0x7f9d84002860 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient(hunting): my global_id is 185087 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da413a830 con 0x7f9d84002860 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 812678381 0 0) 0x7f9da413a830 con 0x7f9d84002860 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d840065c0 con 0x7f9d84002860 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d90001bd0 con 0x7f9d84002860 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient: dump: 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.359 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3788433425 0 0) 0x7f9d840065c0 con 0x7f9d84002860 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d8400a160 con 0x7f9d84002860 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 client.194908.objecter ms_handle_connect 0x7f9d88041540 2024-08-19T10:09:40.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9d900051a0 con 0x7f9d84002860 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9d84002860 legacy=0x7f9d84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.337+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9d84002860 legacy=0x7f9d84004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.343+0000 7f9daad06640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.344+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9d84009e50 con 0x7f9d84009110 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 client.194908.objecter ms_handle_connect 0x7f9d84009110 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da3fff640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9d84009110 legacy=0x7f9d840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 client.194908.objecter resend_mon_ops 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1627694121 0 0) 0x7f9d84009e50 con 0x7f9d84009110 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient(hunting): my global_id is 185096 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d840065c0 con 0x7f9d84009110 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3560261679 0 0) 0x7f9d840065c0 con 0x7f9d84009110 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d84009e50 con 0x7f9d84009110 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f9d90005a00 con 0x7f9d84009110 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient: dump: 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.362 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1300816493 0 0) 0x7f9d84009e50 con 0x7f9d84009110 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 --> v1:172.21.15.1:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d8400c3a0 con 0x7f9d84009110 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 <== mon.2 v1:172.21.15.1:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 4119927616) 0x7f9d90007ec0 con 0x7f9d84009110 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:40.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9d84009110 legacy=0x7f9d840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.346+0000 7f9da17fa640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6790/0 conn(0x7f9d84009110 legacy=0x7f9d840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 497594, 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:40.364 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.365 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:40.366 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:40.367 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:40.368 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.369 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:40.371 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.372 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:40.374 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.380 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.382 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.383 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.384 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:09:40.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.384 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.384 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:40.384 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 10 librados: watch_flush enter 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 10 librados: watch_flush exit 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 20 client.194908.objecter shutdown clearing up homeless session... 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 10 client.194908.objecter successfully canceled tick 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.82:6812/2549104415 conn(0x7f9d88041540 legacy=0x7f9d88043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 10 monclient: shutdown 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 >> v1:172.21.15.1:6791/0 conn(0x7f9da40a67c0 legacy=0x7f9da41ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9da92a3640 1 -- 172.21.15.1:0/4100951334 reap_dead start 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9da92a3640 1 -- 172.21.15.1:0/4100951334 reap_dead start 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 shutdown_connections 2024-08-19T10:09:40.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 >> 172.21.15.1:0/4100951334 conn(0x7f9da40ac4b0 msgr2=0x7f9da40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 shutdown_connections 2024-08-19T10:09:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 -- 172.21.15.1:0/4100951334 wait complete. 2024-08-19T10:09:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.347+0000 7f9daad06640 1 librados: shutdown 2024-08-19T10:09:40.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:40.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T10:09:40.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T10:09:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.486+0000 7fc06e1f7640 10 monclient: get_monmap_and_config 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.486+0000 7fc06e1f7640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.486+0000 7fc06e1f7640 10 monclient: monmap: 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:40.487540+0000 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:40.487540+0000 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.489 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 1 Processor -- start 2024-08-19T10:09:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 1 -- start start 2024-08-19T10:09:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient: init 2024-08-19T10:09:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient: _add_conns ranks=[2,7,4,5,8,0,1,3,6] 2024-08-19T10:09:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient(hunting): picked mon.noname-b con 0x7fc0680a67c0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient(hunting): picked mon.noname-i con 0x7fc0680a6ba0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient(hunting): picked mon.noname-c con 0x7fc06813f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a3b40 con 0x7fc0680a67c0 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a8130 con 0x7fc06813f5c0 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06806ede0 con 0x7fc0680a6ba0 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.487+0000 7fc06e1f7640 10 monclient(hunting): authenticate will time out at 1836.563895s 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.488+0000 7fc0677fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc0680a67c0 0x7fc0680a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:44946/0 (socket says 172.21.15.1:44946) 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.488+0000 7fc0677fe640 1 -- 172.21.15.1:0/1969257725 learned_addr learned my addr 172.21.15.1:0/1969257725 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.488+0000 7fc067fff640 1 --1- 172.21.15.1:0/1969257725 >> v1:172.21.15.1:6791/0 conn(0x7fc06813f5c0 0x7fc0681419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:58092/0 (socket says 172.21.15.1:58092) 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0677fe640 1 -- 172.21.15.1:0/1969257725 >> v1:172.21.15.1:6790/0 conn(0x7fc0680a67c0 legacy=0x7fc0680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc067fff640 1 -- 172.21.15.1:0/1969257725 >> v1:172.21.15.1:6791/0 conn(0x7fc06813f5c0 legacy=0x7fc0681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771138792 0 0) 0x7fc0680a3b40 con 0x7fc0680a67c0 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): my global_id is 185105 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc066ffd640 1 -- 172.21.15.1:0/1969257725 >> v1:172.21.15.82:6792/0 conn(0x7fc0680a6ba0 legacy=0x7fc0680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc054003520 con 0x7fc0680a67c0 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062479240 0 0) 0x7fc0680a8130 con 0x7fc06813f5c0 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient(hunting): my global_id is 194917 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0680a3b40 con 0x7fc06813f5c0 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3827622909 0 0) 0x7fc054003520 con 0x7fc0680a67c0 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 >> v1:172.21.15.82:6792/0 conn(0x7fc0680a6ba0 legacy=0x7fc0680a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 >> v1:172.21.15.1:6791/0 conn(0x7fc06813f5c0 legacy=0x7fc0681419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 monclient: found mon.noname-b 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0680a92a0 con 0x7fc0680a67c0 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc058001ae0 con 0x7fc0680a67c0 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc06e1f7640 5 monclient: authenticate success, global_id 185105 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc06e1f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: dump: 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc06e1f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0580048a0 con 0x7fc0680a67c0 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 1 -- 172.21.15.1:0/1969257725 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc058004e10 con 0x7fc0680a67c0 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc06e1f7640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc06e1f7640 4 set_mon_vals no callback set 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.490+0000 7fc0667fc640 10 monclient: dump: 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 10 monclient: shutdown 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 1 -- 172.21.15.1:0/1969257725 >> v1:172.21.15.1:6790/0 conn(0x7fc0680a67c0 legacy=0x7fc0680a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 1 -- 172.21.15.1:0/1969257725 shutdown_connections 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 1 -- 172.21.15.1:0/1969257725 >> 172.21.15.1:0/1969257725 conn(0x7fc0680ac4b0 msgr2=0x7fc0680a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 1 -- 172.21.15.1:0/1969257725 shutdown_connections 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 1 -- 172.21.15.1:0/1969257725 wait complete. 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.491+0000 7fc06e1f7640 10 monclient: monmap: 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 librados: starting msgr at 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 librados: starting objecter 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 Processor -- start 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 -- start start 2024-08-19T10:09:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 librados: setting wanted keys 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 librados: calling monclient init 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient: init 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient: _add_conns ranks=[7,1,8,6,2,0,5,3,4] 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient(hunting): picked mon.noname-i con 0x7fc0680a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient(hunting): picked mon.noname-f con 0x7fc0680a6ba0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient(hunting): picked mon.noname-e con 0x7fc06813f5c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a18a0 con 0x7fc0680a6ba0 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc068142bb0 con 0x7fc0680a67c0 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06813a930 con 0x7fc06813f5c0 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.492+0000 7fc06e1f7640 10 monclient(hunting): authenticate will time out at 1836.568943s 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.493+0000 7fc067fff640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc06813f5c0 0x7fc0681d6be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:57170/0 (socket says 172.21.15.1:57170) 2024-08-19T10:09:40.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.493+0000 7fc067fff640 1 -- 172.21.15.1:0/3096847354 learned_addr learned my addr 172.21.15.1:0/3096847354 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc067fff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6793/0 conn(0x7fc06813f5c0 legacy=0x7fc0681d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 client.?.objecter ms_handle_connect 0x7fc06813f5c0 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3761801522 0 0) 0x7fc06813a930 con 0x7fc06813f5c0 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): my global_id is 204740 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc048004eb0 con 0x7fc06813f5c0 2024-08-19T10:09:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 client.?.objecter ms_handle_connect 0x7fc0680a6ba0 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc066ffd640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.82:6789/0 conn(0x7fc0680a6ba0 legacy=0x7fc0681d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 client.?.objecter ms_handle_connect 0x7fc0680a67c0 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc0677fe640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.82:6792/0 conn(0x7fc0680a67c0 legacy=0x7fc0681ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 292107099 0 0) 0x7fc048004eb0 con 0x7fc06813f5c0 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc06813a930 con 0x7fc06813f5c0 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc060001fe0 con 0x7fc06813f5c0 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): dump: 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.511 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005129128 0 0) 0x7fc0680a18a0 con 0x7fc0680a6ba0 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): my global_id is 224128 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc048004eb0 con 0x7fc0680a6ba0 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1620560143 0 0) 0x7fc0680a92a0 con 0x7fc0680a67c0 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient(hunting): my global_id is 204694 2024-08-19T10:09:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0680a18a0 con 0x7fc0680a67c0 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4287721306 0 0) 0x7fc06813a930 con 0x7fc06813f5c0 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.82:6792/0 conn(0x7fc0680a67c0 legacy=0x7fc0681ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.82:6789/0 conn(0x7fc0680a6ba0 legacy=0x7fc0681d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 monclient: found mon.e 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06813ac10 con 0x7fc06813f5c0 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 5 monclient: authenticate success, global_id 204740 2024-08-19T10:09:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 10 monclient: _renew_subs 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc0680a2400 con 0x7fc06813f5c0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 10 client.204740.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 10 monclient: _renew_subs 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0680a11a0 con 0x7fc06813f5c0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc06e1f7640 1 librados: init done 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc060004e90 con 0x7fc06813f5c0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc0600053c0 con 0x7fc06813f5c0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc065ffb640 4 set_mon_vals no callback set 2024-08-19T10:09:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.495+0000 7fc05ffff640 10 monclient: dump: 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.496+0000 7fc06e1f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.496+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc04c0065c0 con 0x7fc04c002860 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.496+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc060005690 con 0x7fc06813f5c0 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.497+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fc06004e230 con 0x7fc06813f5c0 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.497+0000 7fc05ffff640 10 client.204740.objecter ms_dispatch 0x7fc068000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.497+0000 7fc05ffff640 3 client.204740.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.497+0000 7fc05ffff640 3 client.204740.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.497+0000 7fc05ffff640 20 client.204740.objecter dump_active .. 0 homeless 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.497+0000 7fc066ffd640 10 monclient: get_auth_request con 0x7fc048041090 auth_method 0 2024-08-19T10:09:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 10 client.204740.objecter ms_handle_connect 0x7fc04c002860 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 10 client.204740.objecter resend_mon_ops 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc0677fe640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6791/0 conn(0x7fc04c002860 legacy=0x7fc04c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2252343167 0 0) 0x7fc04c0065c0 con 0x7fc04c002860 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 10 monclient(hunting): my global_id is 194926 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.498+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc06813a930 con 0x7fc04c002860 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2159142988 0 0) 0x7fc06813a930 con 0x7fc04c002860 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc04c0065c0 con 0x7fc04c002860 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc058001d30 con 0x7fc04c002860 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 10 monclient: dump: 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.521 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1267219681 0 0) 0x7fc04c0065c0 con 0x7fc04c002860 2024-08-19T10:09:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc04c00a160 con 0x7fc04c002860 2024-08-19T10:09:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 10 client.204740.objecter ms_handle_connect 0x7fc048041090 2024-08-19T10:09:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc058004b90 con 0x7fc04c002860 2024-08-19T10:09:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6791/0 conn(0x7fc04c002860 legacy=0x7fc04c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.499+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6791/0 conn(0x7fc04c002860 legacy=0x7fc04c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.505+0000 7fc06e1f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.505+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc04c009e70 con 0x7fc04c009130 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc0677fe640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6791/0 conn(0x7fc04c009130 legacy=0x7fc04c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 client.204740.objecter ms_handle_connect 0x7fc04c009130 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 client.204740.objecter resend_mon_ops 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311120963 0 0) 0x7fc04c009e70 con 0x7fc04c009130 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient(hunting): my global_id is 194935 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04c0065c0 con 0x7fc04c009130 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 144569662 0 0) 0x7fc04c0065c0 con 0x7fc04c009130 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc04c009e70 con 0x7fc04c009130 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc058005490 con 0x7fc04c009130 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient: dump: 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.523 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2056993908 0 0) 0x7fc04c009e70 con 0x7fc04c009130 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 --> v1:172.21.15.1:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc04c00c3f0 con 0x7fc04c009130 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 <== mon.4 v1:172.21.15.1:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 1767918468) 0x7fc058008600 con 0x7fc04c009130 2024-08-19T10:09:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6791/0 conn(0x7fc04c009130 legacy=0x7fc04c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.508+0000 7fc05ffff640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6791/0 conn(0x7fc04c009130 legacy=0x7fc04c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:40.525 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 368195, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:40.526 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.527 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:40.529 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:40.530 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.531 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.533 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.535 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.537 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.539 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.542 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.543 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.544 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.546 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:40.546 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:40.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 10 librados: watch_flush enter 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 10 librados: watch_flush exit 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 20 client.204740.objecter shutdown clearing up homeless session... 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 10 client.204740.objecter successfully canceled tick 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc048041090 legacy=0x7fc048043550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 10 monclient: shutdown 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 >> v1:172.21.15.1:6793/0 conn(0x7fc06813f5c0 legacy=0x7fc0681d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc067fff640 1 -- 172.21.15.1:0/3096847354 reap_dead start 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc067fff640 1 -- 172.21.15.1:0/3096847354 reap_dead start 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 shutdown_connections 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 >> 172.21.15.1:0/3096847354 conn(0x7fc0680ac4b0 msgr2=0x7fc068138980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 shutdown_connections 2024-08-19T10:09:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 -- 172.21.15.1:0/3096847354 wait complete. 2024-08-19T10:09:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.509+0000 7fc06e1f7640 1 librados: shutdown 2024-08-19T10:09:40.549 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:40.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T10:09:40.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T10:09:40.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.648+0000 7fc8e0a7e640 10 monclient: get_monmap_and_config 2024-08-19T10:09:40.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.648+0000 7fc8e0a7e640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.648+0000 7fc8e0a7e640 10 monclient: monmap: 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:40.649502+0000 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:40.649502+0000 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.650 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.651 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 1 Processor -- start 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 1 -- start start 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient: init 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient: _add_conns ranks=[7,1,6,4,0,2,8,5,3] 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient(hunting): picked mon.noname-i con 0x7fc8dc0a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient(hunting): picked mon.noname-f con 0x7fc8dc0694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient(hunting): picked mon.noname-d con 0x7fc8dc0a46a0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc0a3c70 con 0x7fc8dc0694b0 2024-08-19T10:09:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc0a33d0 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc0a36b0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8e0a7e640 10 monclient(hunting): authenticate will time out at 1836.725708s 2024-08-19T10:09:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8dad9d640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fc8dc0a46a0 0x7fc8dc141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34478/0 (socket says 172.21.15.1:34478) 2024-08-19T10:09:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.649+0000 7fc8dad9d640 1 -- 172.21.15.1:0/815764244 learned_addr learned my addr 172.21.15.1:0/815764244 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8dad9d640 1 -- 172.21.15.1:0/815764244 >> v1:172.21.15.1:6792/0 conn(0x7fc8dc0a46a0 legacy=0x7fc8dc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550656355 0 0) 0x7fc8dc0a33d0 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient(hunting): my global_id is 204540 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8c8003520 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d1d9b640 1 -- 172.21.15.1:0/815764244 >> v1:172.21.15.82:6789/0 conn(0x7fc8dc0694b0 legacy=0x7fc8dc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8da59c640 1 -- 172.21.15.1:0/815764244 >> v1:172.21.15.82:6792/0 conn(0x7fc8dc0a6830 legacy=0x7fc8dc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3847725074 0 0) 0x7fc8c8003520 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 >> v1:172.21.15.82:6792/0 conn(0x7fc8dc0a6830 legacy=0x7fc8dc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 >> v1:172.21.15.82:6789/0 conn(0x7fc8dc0694b0 legacy=0x7fc8dc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 monclient: found mon.noname-d 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc06dbb0 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 5 monclient: authenticate success, global_id 204540 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8d4001e60 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: dump: 2024-08-19T10:09:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8d4004c20 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 1 -- 172.21.15.1:0/815764244 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8d4005190 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 4 set_mon_vals no callback set 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8d9d9b640 10 monclient: dump: 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 10 monclient: shutdown 2024-08-19T10:09:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.652+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/815764244 >> v1:172.21.15.1:6792/0 conn(0x7fc8dc0a46a0 legacy=0x7fc8dc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/815764244 shutdown_connections 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/815764244 >> 172.21.15.1:0/815764244 conn(0x7fc8dc0a9020 msgr2=0x7fc8dc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/815764244 shutdown_connections 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/815764244 wait complete. 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 10 monclient: monmap: 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 1 librados: starting msgr at 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.653+0000 7fc8e0a7e640 1 librados: starting objecter 2024-08-19T10:09:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 Processor -- start 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 -- start start 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 librados: setting wanted keys 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 librados: calling monclient init 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient: init 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient: _add_conns ranks=[3,0,2,8,4,5,1,6,7] 2024-08-19T10:09:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient(hunting): picked mon.noname-g con 0x7fc8dc0694b0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc0a6830 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc0aa010 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc142ce0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc138580 con 0x7fc8dc0694b0 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8e0a7e640 10 monclient(hunting): authenticate will time out at 1836.730788s 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8d1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc8dc0a46a0 0x7fc8dc1372e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60538/0 (socket says 172.21.15.1:60538) 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.654+0000 7fc8d1d9b640 1 -- 172.21.15.1:0/3435841808 learned_addr learned my addr 172.21.15.1:0/3435841808 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8dad9d640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6790/0 conn(0x7fc8dc0a6830 legacy=0x7fc8dc1d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 client.?.objecter ms_handle_connect 0x7fc8dc0a6830 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d1d9b640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6789/0 conn(0x7fc8dc0a46a0 legacy=0x7fc8dc1372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 client.?.objecter ms_handle_connect 0x7fc8dc0a46a0 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1855723883 0 0) 0x7fc8dc142ce0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): my global_id is 185114 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8bc004eb0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356658697 0 0) 0x7fc8dc0a3c70 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): my global_id is 204705 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8da59c640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.82:6790/0 conn(0x7fc8dc0694b0 legacy=0x7fc8dc136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8dc142ce0 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 client.?.objecter ms_handle_connect 0x7fc8dc0694b0 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 765868248 0 0) 0x7fc8bc004eb0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8dc0a3c70 con 0x7fc8dc0a6830 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8d4005110 con 0x7fc8dc0a6830 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): dump: 2024-08-19T10:09:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1281839005 0 0) 0x7fc8dc142ce0 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8bc004eb0 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8c4001d10 con 0x7fc8dc0a46a0 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): dump: 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474613030 0 0) 0x7fc8dc138580 con 0x7fc8dc0694b0 2024-08-19T10:09:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient(hunting): my global_id is 214437 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8dc142ce0 con 0x7fc8dc0694b0 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3751533554 0 0) 0x7fc8dc0a3c70 con 0x7fc8dc0a6830 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.82:6790/0 conn(0x7fc8dc0694b0 legacy=0x7fc8dc136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6789/0 conn(0x7fc8dc0a46a0 legacy=0x7fc8dc1372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 monclient: found mon.b 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc1379f0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 5 monclient: authenticate success, global_id 185114 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 10 monclient: _renew_subs 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc13ae10 con 0x7fc8dc0a6830 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 10 client.185114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 10 monclient: _renew_subs 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc8dc13b230 con 0x7fc8dc0a6830 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8e0a7e640 1 librados: init done 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8d40020b0 con 0x7fc8dc0a6830 2024-08-19T10:09:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8d4004c20 con 0x7fc8dc0a6830 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d959a640 4 set_mon_vals no callback set 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.657+0000 7fc8d37fe640 10 monclient: dump: 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8e0a7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc8c00065c0 con 0x7fc8c0002860 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc8d4011d80 con 0x7fc8dc0a6830 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fc8d404e220 con 0x7fc8dc0a6830 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8d37fe640 10 client.185114.objecter ms_dispatch 0x7fc8dc000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8d37fe640 3 client.185114.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.658+0000 7fc8d37fe640 3 client.185114.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.659+0000 7fc8d37fe640 20 client.185114.objecter dump_active .. 0 homeless 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.659+0000 7fc8d1d9b640 10 monclient: get_auth_request con 0x7fc8bc0417d0 auth_method 0 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 10 client.185114.objecter ms_handle_connect 0x7fc8c0002860 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 10 client.185114.objecter resend_mon_ops 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8da59c640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6792/0 conn(0x7fc8c0002860 legacy=0x7fc8c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971373374 0 0) 0x7fc8c00065c0 con 0x7fc8c0002860 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 10 monclient(hunting): my global_id is 204549 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.660+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8dc0a3c70 con 0x7fc8c0002860 2024-08-19T10:09:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3443825907 0 0) 0x7fc8dc0a3c70 con 0x7fc8c0002860 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8c00065c0 con 0x7fc8c0002860 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8cc001bf0 con 0x7fc8c0002860 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 10 monclient: dump: 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.681 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3995201158 0 0) 0x7fc8c00065c0 con 0x7fc8c0002860 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8c000a160 con 0x7fc8c0002860 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 10 client.185114.objecter ms_handle_connect 0x7fc8bc0417d0 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc8cc0057c0 con 0x7fc8c0002860 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6792/0 conn(0x7fc8c0002860 legacy=0x7fc8c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.661+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6792/0 conn(0x7fc8c0002860 legacy=0x7fc8c0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.667+0000 7fc8e0a7e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.667+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc8c0009e50 con 0x7fc8c0009110 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8da59c640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6792/0 conn(0x7fc8c0009110 legacy=0x7fc8c00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 client.185114.objecter ms_handle_connect 0x7fc8c0009110 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 client.185114.objecter resend_mon_ops 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33768881 0 0) 0x7fc8c0009e50 con 0x7fc8c0009110 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient(hunting): my global_id is 204558 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8c00065c0 con 0x7fc8c0009110 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 138045484 0 0) 0x7fc8c00065c0 con 0x7fc8c0009110 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8c0009e50 con 0x7fc8c0009110 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc8cc003cc0 con 0x7fc8c0009110 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient: dump: 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1138621753 0 0) 0x7fc8c0009e50 con 0x7fc8c0009110 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 --> v1:172.21.15.1:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc8c000c3a0 con 0x7fc8c0009110 2024-08-19T10:09:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 <== mon.6 v1:172.21.15.1:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1342925338) 0x7fc8cc005970 con 0x7fc8c0009110 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6792/0 conn(0x7fc8c0009110 legacy=0x7fc8c00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.670+0000 7fc8d37fe640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6792/0 conn(0x7fc8c0009110 legacy=0x7fc8c00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:40.688 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 246540, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:09:40.689 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.690 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:40.692 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.694 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 10 librados: watch_flush enter 2024-08-19T10:09:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 10 librados: watch_flush exit 2024-08-19T10:09:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 20 client.185114.objecter shutdown clearing up homeless session... 2024-08-19T10:09:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 10 client.185114.objecter successfully canceled tick 2024-08-19T10:09:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc8bc0417d0 legacy=0x7fc8bc043c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 10 monclient: shutdown 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 >> v1:172.21.15.1:6790/0 conn(0x7fc8dc0a6830 legacy=0x7fc8dc1d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8dad9d640 1 -- 172.21.15.1:0/3435841808 reap_dead start 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8dad9d640 1 -- 172.21.15.1:0/3435841808 reap_dead start 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 shutdown_connections 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 >> 172.21.15.1:0/3435841808 conn(0x7fc8dc0a9020 msgr2=0x7fc8dc0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 shutdown_connections 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 -- 172.21.15.1:0/3435841808 wait complete. 2024-08-19T10:09:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.671+0000 7fc8e0a7e640 1 librados: shutdown 2024-08-19T10:09:40.711 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:40.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T10:09:40.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.810+0000 7f4dbdade640 10 monclient: get_monmap_and_config 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.810+0000 7f4dbdade640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.811+0000 7f4dbdade640 10 monclient: monmap: 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:40.811931+0000 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:40.811931+0000 2024-08-19T10:09:40.812 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.811+0000 7f4dbdade640 1 Processor -- start 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.811+0000 7f4dbdade640 1 -- start start 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.811+0000 7f4dbdade640 10 monclient: init 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient: _add_conns ranks=[6,4,5,7,2,1,8,3,0] 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient(hunting): picked mon.noname-d con 0x7f4db80a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient(hunting): picked mon.noname-c con 0x7f4db80a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient(hunting): picked mon.noname-h con 0x7f4db813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db80a3b40 con 0x7f4db80a6ba0 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db80a8130 con 0x7f4db813f5c0 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db806ede0 con 0x7f4db80a67c0 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4dbdade640 10 monclient(hunting): authenticate will time out at 1836.888415s 2024-08-19T10:09:40.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4daffff640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4db80a6ba0 0x7f4db80a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:58124/0 (socket says 172.21.15.1:58124) 2024-08-19T10:09:40.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.812+0000 7f4daffff640 1 -- 172.21.15.1:0/462904023 learned_addr learned my addr 172.21.15.1:0/462904023 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4daffff640 1 -- 172.21.15.1:0/462904023 >> v1:172.21.15.1:6791/0 conn(0x7f4db80a6ba0 legacy=0x7f4db80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db77fe640 1 -- 172.21.15.1:0/462904023 >> v1:172.21.15.1:6792/0 conn(0x7f4db80a67c0 legacy=0x7f4db80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972627419 0 0) 0x7f4db80a3b40 con 0x7f4db80a6ba0 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): my global_id is 194944 2024-08-19T10:09:40.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4da4003520 con 0x7f4db80a6ba0 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213849057 0 0) 0x7f4db806ede0 con 0x7f4db80a67c0 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 10 monclient(hunting): my global_id is 204567 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.814+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4db80a3b40 con 0x7f4db80a67c0 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db7fff640 1 -- 172.21.15.1:0/462904023 >> v1:172.21.15.82:6791/0 conn(0x7f4db813f5c0 legacy=0x7f4db81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2874584849 0 0) 0x7f4da4003520 con 0x7f4db80a6ba0 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 >> v1:172.21.15.1:6792/0 conn(0x7f4db80a67c0 legacy=0x7f4db80a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 >> v1:172.21.15.82:6791/0 conn(0x7f4db813f5c0 legacy=0x7f4db81419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 monclient: found mon.noname-c 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db80a92a0 con 0x7f4db80a6ba0 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da0001ae0 con 0x7f4db80a6ba0 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: dump: 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.821 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 5 monclient: authenticate success, global_id 194944 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4da00048a0 con 0x7f4db80a6ba0 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 1 -- 172.21.15.1:0/462904023 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da0004e10 con 0x7f4db80a6ba0 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 4 set_mon_vals no callback set 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4db6ffd640 10 monclient: dump: 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.823 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 10 monclient: shutdown 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 1 -- 172.21.15.1:0/462904023 >> v1:172.21.15.1:6791/0 conn(0x7f4db80a6ba0 legacy=0x7f4db80a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 1 -- 172.21.15.1:0/462904023 shutdown_connections 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 1 -- 172.21.15.1:0/462904023 >> 172.21.15.1:0/462904023 conn(0x7f4db80ac4b0 msgr2=0x7f4db80a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 1 -- 172.21.15.1:0/462904023 shutdown_connections 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.815+0000 7f4dbdade640 1 -- 172.21.15.1:0/462904023 wait complete. 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 10 monclient: monmap: 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.825 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 1 librados: starting msgr at 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 1 librados: starting objecter 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 1 Processor -- start 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 1 -- start start 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 1 librados: setting wanted keys 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 1 librados: calling monclient init 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.816+0000 7f4dbdade640 10 monclient: init 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient: _add_conns ranks=[4,0,5,8,7,6,2,3,1] 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient(hunting): picked mon.noname-c con 0x7f4db80a67c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient(hunting): picked mon.noname-a con 0x7f4db80a6ba0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient(hunting): picked mon.noname-h con 0x7f4db813f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db813ae90 con 0x7f4db80a6ba0 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db813aa70 con 0x7f4db80a67c0 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db80a9d50 con 0x7f4db813f5c0 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4dbdade640 10 monclient(hunting): authenticate will time out at 1836.893307s 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4db77fe640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7f4db80a67c0 0x7f4db809fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:58130/0 (socket says 172.21.15.1:58130) 2024-08-19T10:09:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.817+0000 7f4db77fe640 1 -- 172.21.15.1:0/2003777108 learned_addr learned my addr 172.21.15.1:0/2003777108 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db77fe640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6791/0 conn(0x7f4db80a67c0 legacy=0x7f4db809fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4daffff640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6789/0 conn(0x7f4db80a6ba0 legacy=0x7f4db80a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 client.?.objecter ms_handle_connect 0x7f4db80a67c0 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 client.?.objecter ms_handle_connect 0x7f4db80a6ba0 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 37364655 0 0) 0x7f4db813aa70 con 0x7f4db80a67c0 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): my global_id is 194953 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d98004eb0 con 0x7f4db80a67c0 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1732709477 0 0) 0x7f4db80a92a0 con 0x7f4db80a6ba0 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): my global_id is 204714 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4db813aa70 con 0x7f4db80a6ba0 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 client.?.objecter ms_handle_connect 0x7f4db813f5c0 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db7fff640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.82:6791/0 conn(0x7f4db813f5c0 legacy=0x7f4db81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3132079109 0 0) 0x7f4db813aa70 con 0x7f4db80a6ba0 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4db80a92a0 con 0x7f4db80a6ba0 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da0004d90 con 0x7f4db80a6ba0 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): dump: 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.834 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3168734923 0 0) 0x7f4d98004eb0 con 0x7f4db80a67c0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4db813aa70 con 0x7f4db80a67c0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244354121 0 0) 0x7f4db80a9d50 con 0x7f4db813f5c0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.819+0000 7f4db4ff9640 10 monclient(hunting): my global_id is 214502 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d98004eb0 con 0x7f4db813f5c0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da8003610 con 0x7f4db80a67c0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:40.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient(hunting): dump: 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3580480510 0 0) 0x7f4db80a92a0 con 0x7f4db80a6ba0 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.82:6791/0 conn(0x7f4db813f5c0 legacy=0x7f4db81d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6791/0 conn(0x7f4db80a67c0 legacy=0x7f4db809fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 monclient: found mon.a 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db80a0c90 con 0x7f4db80a6ba0 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 5 monclient: authenticate success, global_id 204714 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 10 monclient: _renew_subs 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4db813f320 con 0x7f4db80a6ba0 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 10 client.204714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 10 monclient: _renew_subs 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4db81d7970 con 0x7f4db80a6ba0 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4dbdade640 1 librados: init done 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4da0001d30 con 0x7f4db80a6ba0 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da0001a80 con 0x7f4db80a6ba0 2024-08-19T10:09:40.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db67fc640 4 set_mon_vals no callback set 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.820+0000 7f4db4ff9640 10 monclient: dump: 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f4da0011a00 con 0x7f4db80a6ba0 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4dbdade640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4d9c0065c0 con 0x7f4d9c002860 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f4da004e010 con 0x7f4db80a6ba0 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db4ff9640 10 client.204714.objecter ms_dispatch 0x7f4db8000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db4ff9640 3 client.204714.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db4ff9640 3 client.204714.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:40.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db4ff9640 20 client.204714.objecter dump_active .. 0 homeless 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.821+0000 7f4db7fff640 10 monclient: get_auth_request con 0x7f4d98041610 auth_method 0 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db77fe640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6793/0 conn(0x7f4d9c002860 legacy=0x7f4d9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 client.204714.objecter ms_handle_connect 0x7f4d9c002860 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 client.204714.objecter resend_mon_ops 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736879735 0 0) 0x7f4d9c0065c0 con 0x7f4d9c002860 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient(hunting): my global_id is 204749 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4db80a92a0 con 0x7f4d9c002860 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 57068236 0 0) 0x7f4db80a92a0 con 0x7f4d9c002860 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d9c0065c0 con 0x7f4d9c002860 2024-08-19T10:09:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da8004210 con 0x7f4d9c002860 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 monclient: dump: 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3780283574 0 0) 0x7f4d9c0065c0 con 0x7f4d9c002860 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d9c00a160 con 0x7f4d9c002860 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.823+0000 7f4db4ff9640 10 client.204714.objecter ms_handle_connect 0x7f4d98041610 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.824+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4da8004430 con 0x7f4d9c002860 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.824+0000 7f4db4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.824+0000 7f4db4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.824+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6793/0 conn(0x7f4d9c002860 legacy=0x7f4d9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.824+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6793/0 conn(0x7f4d9c002860 legacy=0x7f4d9c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.830+0000 7f4dbdade640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.830+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4d9c009e50 con 0x7f4d9c009110 2024-08-19T10:09:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db77fe640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6793/0 conn(0x7f4d9c009110 legacy=0x7f4d9c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 client.204714.objecter ms_handle_connect 0x7f4d9c009110 2024-08-19T10:09:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 client.204714.objecter resend_mon_ops 2024-08-19T10:09:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441229524 0 0) 0x7f4d9c009e50 con 0x7f4d9c009110 2024-08-19T10:09:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient(hunting): my global_id is 204758 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d9c0065c0 con 0x7f4d9c009110 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1620013505 0 0) 0x7f4d9c0065c0 con 0x7f4d9c009110 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d9c009e50 con 0x7f4d9c009110 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f4da8005e80 con 0x7f4d9c009110 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.832+0000 7f4db4ff9640 10 monclient: dump: 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2581508538 0 0) 0x7f4d9c009e50 con 0x7f4d9c009110 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 --> v1:172.21.15.1:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4d9c00c3f0 con 0x7f4d9c009110 2024-08-19T10:09:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 <== mon.8 v1:172.21.15.1:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 3442421698) 0x7f4da800ad70 con 0x7f4d9c009110 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6793/0 conn(0x7f4d9c009110 legacy=0x7f4d9c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db4ff9640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6793/0 conn(0x7f4d9c009110 legacy=0x7f4d9c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 246705, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:09:40.850 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.851 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.852 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:40.854 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.855 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.856 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.860 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.862 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.863 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:40.869 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 10 librados: watch_flush enter 2024-08-19T10:09:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 10 librados: watch_flush exit 2024-08-19T10:09:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 20 client.204714.objecter shutdown clearing up homeless session... 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 10 client.204714.objecter successfully canceled tick 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.82:6812/2549104415 conn(0x7f4d98041610 legacy=0x7f4d98043ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 10 monclient: shutdown 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 >> v1:172.21.15.1:6789/0 conn(0x7f4db80a6ba0 legacy=0x7f4db80a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db7fff640 1 -- 172.21.15.1:0/2003777108 reap_dead start 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4db7fff640 1 -- 172.21.15.1:0/2003777108 reap_dead start 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 shutdown_connections 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 >> 172.21.15.1:0/2003777108 conn(0x7f4db80ac4b0 msgr2=0x7f4db80a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 shutdown_connections 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 -- 172.21.15.1:0/2003777108 wait complete. 2024-08-19T10:09:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.833+0000 7f4dbdade640 1 librados: shutdown 2024-08-19T10:09:40.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:40.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T10:09:40.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T10:09:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.972+0000 7f8df4807640 10 monclient: get_monmap_and_config 2024-08-19T10:09:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.972+0000 7f8df4807640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.972+0000 7f8df4807640 10 monclient: monmap: 2024-08-19T10:09:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:40.973568+0000 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:40.973568+0000 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.976 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 1 Processor -- start 2024-08-19T10:09:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 1 -- start start 2024-08-19T10:09:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient: init 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient: _add_conns ranks=[6,3,5,7,0,2,8,4,1] 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient(hunting): picked mon.noname-d con 0x7f8df00a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient(hunting): picked mon.noname-g con 0x7f8df00a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient(hunting): picked mon.noname-h con 0x7f8df013f5c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a3b40 con 0x7f8df00a6ba0 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a8130 con 0x7f8df013f5c0 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df006ede0 con 0x7f8df00a67c0 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.973+0000 7f8df4807640 10 monclient(hunting): authenticate will time out at 1837.049946s 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.974+0000 7f8dee59c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f8df00a67c0 0x7f8df00a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34524/0 (socket says 172.21.15.1:34524) 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.974+0000 7f8dee59c640 1 -- 172.21.15.1:0/3098323541 learned_addr learned my addr 172.21.15.1:0/3098323541 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8dee59c640 1 -- 172.21.15.1:0/3098323541 >> v1:172.21.15.1:6792/0 conn(0x7f8df00a67c0 legacy=0x7f8df00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400938901 0 0) 0x7f8df006ede0 con 0x7f8df00a67c0 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): my global_id is 204576 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8deed9d640 1 -- 172.21.15.1:0/3098323541 >> v1:172.21.15.82:6791/0 conn(0x7f8df013f5c0 legacy=0x7f8df01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd8003520 con 0x7f8df00a67c0 2024-08-19T10:09:40.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8dedd9b640 1 -- 172.21.15.1:0/3098323541 >> v1:172.21.15.82:6790/0 conn(0x7f8df00a6ba0 legacy=0x7f8df00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372418633 0 0) 0x7f8df00a8130 con 0x7f8df013f5c0 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): my global_id is 214511 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df006ede0 con 0x7f8df013f5c0 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185762951 0 0) 0x7f8df00a3b40 con 0x7f8df00a6ba0 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient(hunting): my global_id is 214446 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df00a8130 con 0x7f8df00a6ba0 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4193850519 0 0) 0x7f8dd8003520 con 0x7f8df00a67c0 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 >> v1:172.21.15.82:6791/0 conn(0x7f8df013f5c0 legacy=0x7f8df01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 >> v1:172.21.15.82:6790/0 conn(0x7f8df00a6ba0 legacy=0x7f8df00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: _finish_hunting 0 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 monclient: found mon.noname-d 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df00a92a0 con 0x7f8df00a67c0 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: _finish_auth 0 2024-08-19T10:09:40.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: _check_auth_tickets 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de0001ad0 con 0x7f8df00a67c0 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: dump: 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.985 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8df4807640 5 monclient: authenticate success, global_id 204576 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8df4807640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:40.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de0001d20 con 0x7f8df00a67c0 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8df4807640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 1 -- 172.21.15.1:0/3098323541 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de0004be0 con 0x7f8df00a67c0 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8df4807640 4 set_mon_vals no callback set 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.976+0000 7f8ded59a640 10 monclient: dump: 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:40.987 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:40.988 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 10 monclient: shutdown 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 1 -- 172.21.15.1:0/3098323541 >> v1:172.21.15.1:6792/0 conn(0x7f8df00a67c0 legacy=0x7f8df00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 1 -- 172.21.15.1:0/3098323541 shutdown_connections 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 1 -- 172.21.15.1:0/3098323541 >> 172.21.15.1:0/3098323541 conn(0x7f8df00ac4b0 msgr2=0x7f8df00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 1 -- 172.21.15.1:0/3098323541 shutdown_connections 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.977+0000 7f8df4807640 1 -- 172.21.15.1:0/3098323541 wait complete. 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient: build_initial_monmap 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient: monmap: 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 librados: starting msgr at 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 librados: starting objecter 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 Processor -- start 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 -- start start 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 librados: setting wanted keys 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 librados: calling monclient init 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient: init 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient: _add_conns ranks=[7,5,0,4,3,8,2,6,1] 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient(hunting): picked mon.noname-i con 0x7f8df00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient(hunting): picked mon.noname-h con 0x7f8df00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient(hunting): picked mon.noname-a con 0x7f8df013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df013ad70 con 0x7f8df013f5c0 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df0139880 con 0x7f8df00a6ba0 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df013bbe0 con 0x7f8df00a67c0 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8df4807640 10 monclient(hunting): authenticate will time out at 1837.054881s 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.978+0000 7f8deed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8df013f5c0 0x7f8df01d6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60556/0 (socket says 172.21.15.1:60556) 2024-08-19T10:09:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.979+0000 7f8deed9d640 1 -- 172.21.15.1:0/2566125446 learned_addr learned my addr 172.21.15.1:0/2566125446 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8deed9d640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.1:6789/0 conn(0x7f8df013f5c0 legacy=0x7f8df01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 client.?.objecter ms_handle_connect 0x7f8df013f5c0 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 client.?.objecter ms_handle_connect 0x7f8df00a67c0 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8dee59c640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6792/0 conn(0x7f8df00a67c0 legacy=0x7f8df013c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 client.?.objecter ms_handle_connect 0x7f8df00a6ba0 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8dedd9b640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6791/0 conn(0x7f8df00a6ba0 legacy=0x7f8df013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268463159 0 0) 0x7f8df013ad70 con 0x7f8df013f5c0 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): my global_id is 204723 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd0004ee0 con 0x7f8df013f5c0 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234935131 0 0) 0x7f8df0139880 con 0x7f8df00a6ba0 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): my global_id is 214520 2024-08-19T10:09:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df013ad70 con 0x7f8df00a6ba0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415663554 0 0) 0x7f8df013bbe0 con 0x7f8df00a67c0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): my global_id is 204712 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df0139880 con 0x7f8df00a67c0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 974604704 0 0) 0x7f8dd0004ee0 con 0x7f8df013f5c0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df013bbe0 con 0x7f8df013f5c0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de8003990 con 0x7f8df013f5c0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): dump: 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 772885381 0 0) 0x7f8df013ad70 con 0x7f8df00a6ba0 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd0004ee0 con 0x7f8df00a6ba0 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de4001d00 con 0x7f8df00a6ba0 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient(hunting): dump: 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3456227620 0 0) 0x7f8df013bbe0 con 0x7f8df013f5c0 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6792/0 conn(0x7f8df00a67c0 legacy=0x7f8df013c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6791/0 conn(0x7f8df00a6ba0 legacy=0x7f8df013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 monclient: found mon.a 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df013b340 con 0x7f8df013f5c0 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 5 monclient: authenticate success, global_id 204723 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 10 monclient: _renew_subs 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8df013a680 con 0x7f8df013f5c0 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 10 client.204723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 10 monclient: _renew_subs 2024-08-19T10:09:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8df013cf00 con 0x7f8df013f5c0 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8df4807640 1 librados: init done 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de8003e90 con 0x7f8df013f5c0 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de80043c0 con 0x7f8df013f5c0 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8decd99640 4 set_mon_vals no callback set 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.981+0000 7f8ddeffd640 10 monclient: dump: 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8df4807640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dd40065c0 con 0x7f8dd4002860 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f8de8004560 con 0x7f8df013f5c0 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f8de804e2b0 con 0x7f8df013f5c0 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8ddeffd640 10 client.204723.objecter ms_dispatch 0x7f8df0000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8ddeffd640 3 client.204723.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.982+0000 7f8ddeffd640 3 client.204723.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.983+0000 7f8ddeffd640 20 client.204723.objecter dump_active .. 0 homeless 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.983+0000 7f8dedd9b640 10 monclient: get_auth_request con 0x7f8dd0041700 auth_method 0 2024-08-19T10:09:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 client.204723.objecter ms_handle_connect 0x7f8dd4002860 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 client.204723.objecter resend_mon_ops 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8dee59c640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6789/0 conn(0x7f8dd4002860 legacy=0x7f8dd4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355379863 0 0) 0x7f8dd40065c0 con 0x7f8dd4002860 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient(hunting): my global_id is 224146 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df013bbe0 con 0x7f8dd4002860 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 client.204723.objecter ms_handle_connect 0x7f8dd0041700 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 654491782 0 0) 0x7f8df013bbe0 con 0x7f8dd4002860 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd40065c0 con 0x7f8dd4002860 2024-08-19T10:09:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de0001d40 con 0x7f8dd4002860 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 10 monclient: dump: 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 342391161 0 0) 0x7f8dd40065c0 con 0x7f8dd4002860 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.985+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8dd400a160 con 0x7f8dd4002860 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.986+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8de00048d0 con 0x7f8dd4002860 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.986+0000 7f8ddeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.986+0000 7f8ddeffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.986+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6789/0 conn(0x7f8dd4002860 legacy=0x7f8dd4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.986+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6789/0 conn(0x7f8dd4002860 legacy=0x7f8dd4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.992+0000 7f8df4807640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.992+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dd4009e70 con 0x7f8dd4009130 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.994+0000 7f8dee59c640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6789/0 conn(0x7f8dd4009130 legacy=0x7f8dd40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.994+0000 7f8ddeffd640 10 client.204723.objecter ms_handle_connect 0x7f8dd4009130 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.994+0000 7f8ddeffd640 10 client.204723.objecter resend_mon_ops 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531373378 0 0) 0x7f8dd4009e70 con 0x7f8dd4009130 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient(hunting): my global_id is 224155 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd40065c0 con 0x7f8dd4009130 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3601569258 0 0) 0x7f8dd40065c0 con 0x7f8dd4009130 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd4009e70 con 0x7f8dd4009130 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f8de0005340 con 0x7f8dd4009130 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient: dump: 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 152647924 0 0) 0x7f8dd4009e70 con 0x7f8dd4009130 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 --> v1:172.21.15.82:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8dd400c410 con 0x7f8dd4009130 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 <== mon.1 v1:172.21.15.82:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7516 (unknown 0 0 2021838811) 0x7f8de0005470 con 0x7f8dd4009130 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6789/0 conn(0x7f8dd4009130 legacy=0x7f8dd40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.997+0000 7f8ddeffd640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6789/0 conn(0x7f8dd4009130 legacy=0x7f8dd40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 4628, 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:41.018 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.021 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.022 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.023 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.030 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.032 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.033 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:41.039 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 10 librados: watch_flush enter 2024-08-19T10:09:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 10 librados: watch_flush exit 2024-08-19T10:09:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 20 client.204723.objecter shutdown clearing up homeless session... 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 10 client.204723.objecter successfully canceled tick 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.82:6812/2549104415 conn(0x7f8dd0041700 legacy=0x7f8dd0043bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 10 monclient: shutdown 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 >> v1:172.21.15.1:6789/0 conn(0x7f8df013f5c0 legacy=0x7f8df01d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8deed9d640 1 -- 172.21.15.1:0/2566125446 reap_dead start 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8deed9d640 1 -- 172.21.15.1:0/2566125446 reap_dead start 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 shutdown_connections 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 >> 172.21.15.1:0/2566125446 conn(0x7f8df00ac4b0 msgr2=0x7f8df013ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 shutdown_connections 2024-08-19T10:09:41.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 -- 172.21.15.1:0/2566125446 wait complete. 2024-08-19T10:09:41.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:40.998+0000 7f8df4807640 1 librados: shutdown 2024-08-19T10:09:41.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:41.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T10:09:41.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T10:09:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.142+0000 7f6f09638640 10 monclient: get_monmap_and_config 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.142+0000 7f6f09638640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.142+0000 7f6f09638640 10 monclient: monmap: 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:41.143529+0000 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:41.143529+0000 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.145 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.145 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.145 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.145 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.145 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.145 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 1 Processor -- start 2024-08-19T10:09:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 1 -- start start 2024-08-19T10:09:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient: init 2024-08-19T10:09:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient: _add_conns ranks=[5,2,6,1,8,4,7,0,3] 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient(hunting): picked mon.noname-h con 0x7f6f040a67c0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient(hunting): picked mon.noname-b con 0x7f6f040a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient(hunting): picked mon.noname-d con 0x7f6f0413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f040a3b40 con 0x7f6f040a6ba0 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f040a8130 con 0x7f6f040a67c0 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0406ede0 con 0x7f6f0413f5c0 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f09638640 10 monclient(hunting): authenticate will time out at 1837.219725s 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f027fc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f6f040a6ba0 0x7f6f040a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:44966/0 (socket says 172.21.15.1:44966) 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f037fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f6f0413f5c0 0x7f6f041419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34538/0 (socket says 172.21.15.1:34538) 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.143+0000 7f6f027fc640 1 -- 172.21.15.1:0/651013578 learned_addr learned my addr 172.21.15.1:0/651013578 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f037fe640 1 -- 172.21.15.1:0/651013578 >> v1:172.21.15.1:6792/0 conn(0x7f6f0413f5c0 legacy=0x7f6f041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f027fc640 1 -- 172.21.15.1:0/651013578 >> v1:172.21.15.1:6790/0 conn(0x7f6f040a6ba0 legacy=0x7f6f040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961837259 0 0) 0x7f6f0406ede0 con 0x7f6f0413f5c0 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): my global_id is 204585 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eec003520 con 0x7f6f0413f5c0 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4044210033 0 0) 0x7f6f040a3b40 con 0x7f6f040a6ba0 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f02ffd640 1 -- 172.21.15.1:0/651013578 >> v1:172.21.15.82:6791/0 conn(0x7f6f040a67c0 legacy=0x7f6f040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient(hunting): my global_id is 185123 2024-08-19T10:09:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0406ede0 con 0x7f6f040a6ba0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2024104404 0 0) 0x7f6eec003520 con 0x7f6f0413f5c0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 >> v1:172.21.15.82:6791/0 conn(0x7f6f040a67c0 legacy=0x7f6f040a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 >> v1:172.21.15.1:6790/0 conn(0x7f6f040a6ba0 legacy=0x7f6f040a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 monclient: found mon.noname-d 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f040a92a0 con 0x7f6f0413f5c0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6efc001e50 con 0x7f6f0413f5c0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: dump: 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f09638640 5 monclient: authenticate success, global_id 204585 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f09638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6efc004b60 con 0x7f6f0413f5c0 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f09638640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 1 -- 172.21.15.1:0/651013578 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6efc005090 con 0x7f6f0413f5c0 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f09638640 4 set_mon_vals no callback set 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.146+0000 7f6f01ffb640 10 monclient: dump: 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 10 monclient: shutdown 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 1 -- 172.21.15.1:0/651013578 >> v1:172.21.15.1:6792/0 conn(0x7f6f0413f5c0 legacy=0x7f6f041419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 1 -- 172.21.15.1:0/651013578 shutdown_connections 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 1 -- 172.21.15.1:0/651013578 >> 172.21.15.1:0/651013578 conn(0x7f6f040ac4b0 msgr2=0x7f6f040a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 1 -- 172.21.15.1:0/651013578 shutdown_connections 2024-08-19T10:09:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 1 -- 172.21.15.1:0/651013578 wait complete. 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.147+0000 7f6f09638640 10 monclient: monmap: 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 librados: starting msgr at 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 librados: starting objecter 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 Processor -- start 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 -- start start 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 librados: setting wanted keys 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 librados: calling monclient init 2024-08-19T10:09:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient: init 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient: _add_conns ranks=[8,3,4,1,0,6,2,7,5] 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient(hunting): picked mon.noname-e con 0x7f6f040a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient(hunting): picked mon.noname-g con 0x7f6f040a6ba0 addr v1:172.21.15.82:6790/0 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0413f5c0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 -- --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f04142e00 con 0x7f6f040a6ba0 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f040a1650 con 0x7f6f0413f5c0 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0413a810 con 0x7f6f040a67c0 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f09638640 10 monclient(hunting): authenticate will time out at 1837.224944s 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f02ffd640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f6f040a67c0 0x7f6f0409fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:57202/0 (socket says 172.21.15.1:57202) 2024-08-19T10:09:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.148+0000 7f6f02ffd640 1 -- 172.21.15.1:0/3869718666 learned_addr learned my addr 172.21.15.1:0/3869718666 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f02ffd640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.1:6793/0 conn(0x7f6f040a67c0 legacy=0x7f6f0409fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 client.?.objecter ms_handle_connect 0x7f6f040a67c0 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 client.?.objecter ms_handle_connect 0x7f6f0413f5c0 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f037fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.1:6791/0 conn(0x7f6f0413f5c0 legacy=0x7f6f040a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794486260 0 0) 0x7f6f0413a810 con 0x7f6f040a67c0 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): my global_id is 204767 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ef0004eb0 con 0x7f6f040a67c0 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571073048 0 0) 0x7f6f040a1650 con 0x7f6f0413f5c0 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): my global_id is 194962 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0413a810 con 0x7f6f0413f5c0 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 client.?.objecter ms_handle_connect 0x7f6f040a6ba0 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f027fc640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6f040a6ba0 legacy=0x7f6f040a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4022160472 0 0) 0x7f6ef0004eb0 con 0x7f6f040a67c0 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f040a1650 con 0x7f6f040a67c0 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3853444746 0 0) 0x7f6f0413a810 con 0x7f6f0413f5c0 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ef0004eb0 con 0x7f6f0413f5c0 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6ef4003610 con 0x7f6f040a67c0 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): dump: 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585249877 0 0) 0x7f6f04142e00 con 0x7f6f040a6ba0 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): my global_id is 214455 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0413a810 con 0x7f6f040a6ba0 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6efc0020a0 con 0x7f6f0413f5c0 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient(hunting): dump: 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.171 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.172 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2125046980 0 0) 0x7f6f040a1650 con 0x7f6f040a67c0 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.1:6791/0 conn(0x7f6f0413f5c0 legacy=0x7f6f040a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6f040a6ba0 legacy=0x7f6f040a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 monclient: found mon.e 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f04139ab0 con 0x7f6f040a67c0 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6eeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 5 monclient: authenticate success, global_id 204767 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 10 monclient: _renew_subs 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f040a21b0 con 0x7f6f040a67c0 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 10 client.204767.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 10 monclient: _renew_subs 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f041d7440 con 0x7f6f040a67c0 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.151+0000 7f6f09638640 1 librados: init done 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ef4003be0 con 0x7f6f040a67c0 2024-08-19T10:09:41.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6ef40040d0 con 0x7f6f040a67c0 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6f017fa640 4 set_mon_vals no callback set 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 10 monclient: dump: 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.175 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6f09638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6ee40065c0 con 0x7f6ee4002860 2024-08-19T10:09:41.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.152+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6ef4011a80 con 0x7f6f040a67c0 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.153+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f6ef404dfe0 con 0x7f6f040a67c0 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.153+0000 7f6eeb7fe640 10 client.204767.objecter ms_dispatch 0x7f6f04000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.153+0000 7f6eeb7fe640 3 client.204767.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.153+0000 7f6eeb7fe640 3 client.204767.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.153+0000 7f6eeb7fe640 20 client.204767.objecter dump_active .. 0 homeless 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.153+0000 7f6f037fe640 10 monclient: get_auth_request con 0x7f6ef00416d0 auth_method 0 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 client.204767.objecter ms_handle_connect 0x7f6ee4002860 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6f027fc640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6ee4002860 legacy=0x7f6ee4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 client.204767.objecter resend_mon_ops 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172883524 0 0) 0x7f6ee40065c0 con 0x7f6ee4002860 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient(hunting): my global_id is 214464 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f040a1650 con 0x7f6ee4002860 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 964394977 0 0) 0x7f6f040a1650 con 0x7f6ee4002860 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ee40065c0 con 0x7f6ee4002860 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 client.204767.objecter ms_handle_connect 0x7f6ef00416d0 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6ef8001af0 con 0x7f6ee4002860 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 10 monclient: dump: 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.178 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.179 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3771472267 0 0) 0x7f6ee40065c0 con 0x7f6ee4002860 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.155+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ee400a160 con 0x7f6ee4002860 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.156+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6ef8004870 con 0x7f6ee4002860 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.156+0000 7f6eeb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.156+0000 7f6eeb7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.156+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6ee4002860 legacy=0x7f6ee4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.156+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6ee4002860 legacy=0x7f6ee4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.162+0000 7f6f09638640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.162+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6ee400c4f0 con 0x7f6ee4009110 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.164+0000 7f6f027fc640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6ee4009110 legacy=0x7f6ee4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.164+0000 7f6eeb7fe640 10 client.204767.objecter ms_handle_connect 0x7f6ee4009110 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.164+0000 7f6eeb7fe640 10 client.204767.objecter resend_mon_ops 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179623998 0 0) 0x7f6ee400c4f0 con 0x7f6ee4009110 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient(hunting): my global_id is 214473 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ee40065c0 con 0x7f6ee4009110 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1586656423 0 0) 0x7f6ee40065c0 con 0x7f6ee4009110 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ee400c4f0 con 0x7f6ee4009110 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6ef80050a0 con 0x7f6ee4009110 2024-08-19T10:09:41.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 10 monclient: dump: 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.182 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 289143668 0 0) 0x7f6ee400c4f0 con 0x7f6ee4009110 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.165+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 --> v1:172.21.15.82:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ee4009ce0 con 0x7f6ee4009110 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 <== mon.3 v1:172.21.15.82:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 1079414993) 0x7f6ef8007420 con 0x7f6ee4009110 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6eeb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6eeb7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6ee4009110 legacy=0x7f6ee4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6eeb7fe640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6790/0 conn(0x7f6ee4009110 legacy=0x7f6ee4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 126039, 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:41.184 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:41.188 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.189 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.192 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:41.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:41.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.195 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.198 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.199 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:41.200 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:41.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.202 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:41.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:41.204 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:41.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 10 librados: watch_flush enter 2024-08-19T10:09:41.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 10 librados: watch_flush exit 2024-08-19T10:09:41.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 20 client.204767.objecter shutdown clearing up homeless session... 2024-08-19T10:09:41.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 10 client.204767.objecter successfully canceled tick 2024-08-19T10:09:41.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6ef00416d0 legacy=0x7f6ef0043b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 10 monclient: shutdown 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 >> v1:172.21.15.1:6793/0 conn(0x7f6f040a67c0 legacy=0x7f6f0409fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f037fe640 1 -- 172.21.15.1:0/3869718666 reap_dead start 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f037fe640 1 -- 172.21.15.1:0/3869718666 reap_dead start 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 shutdown_connections 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 >> 172.21.15.1:0/3869718666 conn(0x7f6f040ac4b0 msgr2=0x7f6f040a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 shutdown_connections 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 -- 172.21.15.1:0/3869718666 wait complete. 2024-08-19T10:09:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.166+0000 7f6f09638640 1 librados: shutdown 2024-08-19T10:09:41.207 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:41.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T10:09:41.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.307+0000 7fcb604bc640 10 monclient: get_monmap_and_config 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.307+0000 7fcb604bc640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.307+0000 7fcb604bc640 10 monclient: monmap: 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:41.308026+0000 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:41.308026+0000 2024-08-19T10:09:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.309 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.307+0000 7fcb604bc640 1 Processor -- start 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.307+0000 7fcb604bc640 1 -- start start 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.307+0000 7fcb604bc640 10 monclient: init 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient: _add_conns ranks=[6,1,5,8,3,2,0,7,4] 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient(hunting): picked mon.noname-d con 0x7fcb580a6830 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient(hunting): picked mon.noname-f con 0x7fcb580694b0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient(hunting): picked mon.noname-h con 0x7fcb580a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb580a3c70 con 0x7fcb580694b0 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb580a33d0 con 0x7fcb580a46a0 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb580a36b0 con 0x7fcb580a6830 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb604bc640 10 monclient(hunting): authenticate will time out at 1837.384504s 2024-08-19T10:09:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb5e258640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7fcb580a6830 0x7fcb580a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34554/0 (socket says 172.21.15.1:34554) 2024-08-19T10:09:41.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.308+0000 7fcb5e258640 1 -- 172.21.15.1:0/1845397998 learned_addr learned my addr 172.21.15.1:0/1845397998 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.310+0000 7fcb5e258640 1 -- 172.21.15.1:0/1845397998 >> v1:172.21.15.1:6792/0 conn(0x7fcb580a6830 legacy=0x7fcb580a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806985724 0 0) 0x7fcb580a36b0 con 0x7fcb580a6830 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient(hunting): my global_id is 204594 2024-08-19T10:09:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb40003520 con 0x7fcb580a6830 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5ea59640 1 -- 172.21.15.1:0/1845397998 >> v1:172.21.15.82:6791/0 conn(0x7fcb580a46a0 legacy=0x7fcb58141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5da57640 1 -- 172.21.15.1:0/1845397998 >> v1:172.21.15.82:6789/0 conn(0x7fcb580694b0 legacy=0x7fcb580a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2018129874 0 0) 0x7fcb40003520 con 0x7fcb580a6830 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 >> v1:172.21.15.82:6791/0 conn(0x7fcb580a46a0 legacy=0x7fcb58141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 >> v1:172.21.15.82:6789/0 conn(0x7fcb580694b0 legacy=0x7fcb580a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 monclient: found mon.noname-d 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5806dbb0 con 0x7fcb580a6830 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb50001ae0 con 0x7fcb580a6830 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: dump: 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 5 monclient: authenticate success, global_id 204594 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb500048a0 con 0x7fcb580a6830 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 1 -- 172.21.15.1:0/1845397998 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb50004e10 con 0x7fcb580a6830 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 4 set_mon_vals no callback set 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb5d256640 10 monclient: dump: 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 10 monclient: shutdown 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.311+0000 7fcb604bc640 1 -- 172.21.15.1:0/1845397998 >> v1:172.21.15.1:6792/0 conn(0x7fcb580a6830 legacy=0x7fcb580a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 -- 172.21.15.1:0/1845397998 shutdown_connections 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 -- 172.21.15.1:0/1845397998 >> 172.21.15.1:0/1845397998 conn(0x7fcb580a9020 msgr2=0x7fcb58068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 -- 172.21.15.1:0/1845397998 shutdown_connections 2024-08-19T10:09:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 -- 172.21.15.1:0/1845397998 wait complete. 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 10 monclient: monmap: 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 librados: starting msgr at 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 librados: starting objecter 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.312+0000 7fcb604bc640 1 Processor -- start 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 1 -- start start 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 1 librados: setting wanted keys 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 1 librados: calling monclient init 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient: init 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient: _add_conns ranks=[2,5,7,1,8,0,3,6,4] 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient(hunting): picked mon.noname-b con 0x7fcb580694b0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient(hunting): picked mon.noname-h con 0x7fcb580a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient(hunting): picked mon.noname-i con 0x7fcb580a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb58142a40 con 0x7fcb580694b0 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb58138660 con 0x7fcb580a46a0 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb58138940 con 0x7fcb580a6830 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb604bc640 10 monclient(hunting): authenticate will time out at 1837.389551s 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb5e258640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fcb580694b0 0x7fcb581ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:44982/0 (socket says 172.21.15.1:44982) 2024-08-19T10:09:41.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.313+0000 7fcb5e258640 1 -- 172.21.15.1:0/2220476834 learned_addr learned my addr 172.21.15.1:0/2220476834 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb5e258640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.1:6790/0 conn(0x7fcb580694b0 legacy=0x7fcb581ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 client.?.objecter ms_handle_connect 0x7fcb580694b0 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb5ea59640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6792/0 conn(0x7fcb580a6830 legacy=0x7fcb581d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525485773 0 0) 0x7fcb5806dbb0 con 0x7fcb580694b0 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 monclient(hunting): my global_id is 185132 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb5da57640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb580a46a0 legacy=0x7fcb581ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb44004eb0 con 0x7fcb580694b0 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 client.?.objecter ms_handle_connect 0x7fcb580a6830 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 client.?.objecter ms_handle_connect 0x7fcb580a46a0 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.315+0000 7fcb56ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 90350909 0 0) 0x7fcb44004eb0 con 0x7fcb580694b0 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb5806dbb0 con 0x7fcb580694b0 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb50001d30 con 0x7fcb580694b0 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.329 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3198573724 0 0) 0x7fcb58138660 con 0x7fcb580a46a0 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): my global_id is 214547 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb44004eb0 con 0x7fcb580a46a0 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507198246 0 0) 0x7fcb58138940 con 0x7fcb580a6830 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient(hunting): my global_id is 204721 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb58138660 con 0x7fcb580a6830 2024-08-19T10:09:41.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2388966321 0 0) 0x7fcb5806dbb0 con 0x7fcb580694b0 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6792/0 conn(0x7fcb580a6830 legacy=0x7fcb581d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb580a46a0 legacy=0x7fcb581ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 monclient: found mon.b 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb581cf1d0 con 0x7fcb580694b0 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 5 monclient: authenticate success, global_id 185132 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 10 monclient: _renew_subs 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:41.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb580a9f70 con 0x7fcb580694b0 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 10 client.185132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 10 monclient: _renew_subs 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.1:6790/0 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb58138380 con 0x7fcb580694b0 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb604bc640 1 librados: init done 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb50005040 con 0x7fcb580694b0 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb50005530 con 0x7fcb580694b0 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb5ca55640 4 set_mon_vals no callback set 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.316+0000 7fcb56ffd640 10 monclient: dump: 2024-08-19T10:09:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.334 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb604bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcb380065c0 con 0x7fcb38002860 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fcb50011990 con 0x7fcb580694b0 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fcb5004df90 con 0x7fcb580694b0 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb56ffd640 10 client.185132.objecter ms_dispatch 0x7fcb58000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb56ffd640 3 client.185132.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb56ffd640 3 client.185132.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb56ffd640 20 client.185132.objecter dump_active .. 0 homeless 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.317+0000 7fcb5ea59640 10 monclient: get_auth_request con 0x7fcb440412d0 auth_method 0 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 client.185132.objecter ms_handle_connect 0x7fcb38002860 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 client.185132.objecter resend_mon_ops 2024-08-19T10:09:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb5da57640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb38002860 legacy=0x7fcb38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104898546 0 0) 0x7fcb380065c0 con 0x7fcb38002860 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient(hunting): my global_id is 214556 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb5806dbb0 con 0x7fcb38002860 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3127992483 0 0) 0x7fcb5806dbb0 con 0x7fcb38002860 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb380065c0 con 0x7fcb38002860 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb4c001b50 con 0x7fcb38002860 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 10 monclient: dump: 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.336 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3945112107 0 0) 0x7fcb380065c0 con 0x7fcb38002860 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.319+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb3800a160 con 0x7fcb38002860 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.320+0000 7fcb56ffd640 10 client.185132.objecter ms_handle_connect 0x7fcb440412d0 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.320+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcb4c0057c0 con 0x7fcb38002860 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.320+0000 7fcb56ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.320+0000 7fcb56ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.320+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb38002860 legacy=0x7fcb38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.320+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb38002860 legacy=0x7fcb38004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.326+0000 7fcb604bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.326+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcb38009e50 con 0x7fcb38009110 2024-08-19T10:09:41.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb5da57640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb38009110 legacy=0x7fcb380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 client.185132.objecter ms_handle_connect 0x7fcb38009110 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 client.185132.objecter resend_mon_ops 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154266001 0 0) 0x7fcb38009e50 con 0x7fcb38009110 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient(hunting): my global_id is 214565 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb380065c0 con 0x7fcb38009110 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 447577981 0 0) 0x7fcb380065c0 con 0x7fcb38009110 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb38009e50 con 0x7fcb38009110 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fcb4c003d80 con 0x7fcb38009110 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 10 monclient: dump: 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.343 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 856892561 0 0) 0x7fcb38009e50 con 0x7fcb38009110 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.329+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 --> v1:172.21.15.82:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb3800c3f0 con 0x7fcb38009110 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 <== mon.5 v1:172.21.15.82:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7518 (unknown 0 0 4260822793) 0x7fcb4c005a70 con 0x7fcb38009110 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb56ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb56ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb38009110 legacy=0x7fcb380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb56ffd640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6791/0 conn(0x7fcb38009110 legacy=0x7fcb380095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 126203, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:41.345 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.347 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:41.348 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:41.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.354 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.356 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:41.357 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.358 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.361 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.362 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.363 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:41.364 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 10 librados: watch_flush enter 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 10 librados: watch_flush exit 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 20 client.185132.objecter shutdown clearing up homeless session... 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 10 client.185132.objecter successfully canceled tick 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.82:6812/2549104415 conn(0x7fcb440412d0 legacy=0x7fcb44043790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 10 monclient: shutdown 2024-08-19T10:09:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 >> v1:172.21.15.1:6790/0 conn(0x7fcb580694b0 legacy=0x7fcb581ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb5ea59640 1 -- 172.21.15.1:0/2220476834 reap_dead start 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb5ea59640 1 -- 172.21.15.1:0/2220476834 reap_dead start 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 shutdown_connections 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 >> 172.21.15.1:0/2220476834 conn(0x7fcb580a9020 msgr2=0x7fcb580a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 shutdown_connections 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 -- 172.21.15.1:0/2220476834 wait complete. 2024-08-19T10:09:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.330+0000 7fcb604bc640 1 librados: shutdown 2024-08-19T10:09:41.367 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:41.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T10:09:41.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.466+0000 7fc9e8108640 10 monclient: get_monmap_and_config 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.466+0000 7fc9e8108640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.466+0000 7fc9e8108640 10 monclient: monmap: 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:41.467848+0000 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:41.467848+0000 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.469 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.467+0000 7fc9e8108640 1 Processor -- start 2024-08-19T10:09:41.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.467+0000 7fc9e8108640 1 -- start start 2024-08-19T10:09:41.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.467+0000 7fc9e8108640 10 monclient: init 2024-08-19T10:09:41.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient: _add_conns ranks=[7,4,0,6,2,8,5,1,3] 2024-08-19T10:09:41.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient(hunting): picked mon.noname-i con 0x7fc9e00a67c0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:41.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e00a6ba0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:41.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e013f5c0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:41.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e00a3b40 con 0x7fc9e013f5c0 2024-08-19T10:09:41.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e00a8130 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e006ede0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e8108640 10 monclient(hunting): authenticate will time out at 1837.544309s 2024-08-19T10:09:41.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e66a5640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc9e013f5c0 0x7fc9e01419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60558/0 (socket says 172.21.15.1:60558) 2024-08-19T10:09:41.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e56a3640 1 --1- >> v1:172.21.15.1:6791/0 conn(0x7fc9e00a6ba0 0x7fc9e00a6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:58150/0 (socket says 172.21.15.1:58150) 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.468+0000 7fc9e66a5640 1 -- 172.21.15.1:0/394204535 learned_addr learned my addr 172.21.15.1:0/394204535 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e56a3640 1 -- 172.21.15.1:0/394204535 >> v1:172.21.15.1:6791/0 conn(0x7fc9e00a6ba0 legacy=0x7fc9e00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e66a5640 1 -- 172.21.15.1:0/394204535 >> v1:172.21.15.1:6789/0 conn(0x7fc9e013f5c0 legacy=0x7fc9e01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808667119 0 0) 0x7fc9e00a8130 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): my global_id is 194971 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e5ea4640 1 -- 172.21.15.1:0/394204535 >> v1:172.21.15.82:6792/0 conn(0x7fc9e00a67c0 legacy=0x7fc9e00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9cc003520 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600725332 0 0) 0x7fc9e00a3b40 con 0x7fc9e013f5c0 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 10 monclient(hunting): my global_id is 204732 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e00a8130 con 0x7fc9e013f5c0 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.470+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 <== mon.4 v1:172.21.15.1:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2136102207 0 0) 0x7fc9cc003520 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 >> v1:172.21.15.82:6792/0 conn(0x7fc9e00a67c0 legacy=0x7fc9e00a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 >> v1:172.21.15.1:6789/0 conn(0x7fc9e013f5c0 legacy=0x7fc9e01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 monclient: found mon.noname-c 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.1:6791/0 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 --> v1:172.21.15.1:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e00a92a0 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 <== mon.4 v1:172.21.15.1:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d0001ae0 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 5 monclient: authenticate success, global_id 194971 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: dump: 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 <== mon.4 v1:172.21.15.1:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9d00047e0 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 1 -- 172.21.15.1:0/394204535 <== mon.4 v1:172.21.15.1:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d0004d50 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e4ea2640 10 monclient: dump: 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.480 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 4 set_mon_vals no callback set 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 10 monclient: shutdown 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 1 -- 172.21.15.1:0/394204535 >> v1:172.21.15.1:6791/0 conn(0x7fc9e00a6ba0 legacy=0x7fc9e00a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 1 -- 172.21.15.1:0/394204535 shutdown_connections 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 1 -- 172.21.15.1:0/394204535 >> 172.21.15.1:0/394204535 conn(0x7fc9e00ac4b0 msgr2=0x7fc9e00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 1 -- 172.21.15.1:0/394204535 shutdown_connections 2024-08-19T10:09:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.471+0000 7fc9e8108640 1 -- 172.21.15.1:0/394204535 wait complete. 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 10 monclient: monmap: 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 1 librados: starting msgr at 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 1 librados: starting objecter 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 1 Processor -- start 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 1 -- start start 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 1 librados: setting wanted keys 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 1 librados: calling monclient init 2024-08-19T10:09:41.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.472+0000 7fc9e8108640 10 monclient: init 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient: _add_conns ranks=[8,5,6,4,3,0,2,7,1] 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient(hunting): picked mon.noname-e con 0x7fc9e00a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient(hunting): picked mon.noname-h con 0x7fc9e00a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient(hunting): picked mon.noname-d con 0x7fc9e013f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e01cad90 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e013a730 con 0x7fc9e013f5c0 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e013f3a0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e8108640 10 monclient(hunting): authenticate will time out at 1837.549303s 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e5ea4640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7fc9e00a67c0 0x7fc9e013ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:57216/0 (socket says 172.21.15.1:57216) 2024-08-19T10:09:41.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.473+0000 7fc9e5ea4640 1 -- 172.21.15.1:0/216775041 learned_addr learned my addr 172.21.15.1:0/216775041 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9e5ea4640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.1:6793/0 conn(0x7fc9e00a67c0 legacy=0x7fc9e013ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 client.?.objecter ms_handle_connect 0x7fc9e00a67c0 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9e66a5640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.1:6792/0 conn(0x7fc9e013f5c0 legacy=0x7fc9e01d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 client.?.objecter ms_handle_connect 0x7fc9e013f5c0 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432729229 0 0) 0x7fc9e013f3a0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): my global_id is 204776 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c4004eb0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 622868175 0 0) 0x7fc9e013a730 con 0x7fc9e013f5c0 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): my global_id is 204603 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9e56a3640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6791/0 conn(0x7fc9e00a6ba0 legacy=0x7fc9e01d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e013f3a0 con 0x7fc9e013f5c0 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 client.?.objecter ms_handle_connect 0x7fc9e00a6ba0 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2349768087 0 0) 0x7fc9c4004eb0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e013a730 con 0x7fc9e00a67c0 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9dc003610 con 0x7fc9e00a67c0 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): dump: 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.492 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.475+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1178720487 0 0) 0x7fc9e013f3a0 con 0x7fc9e013f5c0 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c4004eb0 con 0x7fc9e013f5c0 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244766327 0 0) 0x7fc9e00a92a0 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): my global_id is 214574 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e013f3a0 con 0x7fc9e00a6ba0 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d8001f40 con 0x7fc9e013f5c0 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient(hunting): dump: 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.494 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3915146791 0 0) 0x7fc9e013a730 con 0x7fc9e00a67c0 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.1:6792/0 conn(0x7fc9e013f5c0 legacy=0x7fc9e01d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6791/0 conn(0x7fc9e00a6ba0 legacy=0x7fc9e01d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 monclient: found mon.e 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e01429d0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 5 monclient: authenticate success, global_id 204776 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 10 monclient: _renew_subs 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e00a1610 con 0x7fc9e00a67c0 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 10 client.204776.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 10 monclient: _renew_subs 2024-08-19T10:09:41.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.1:6793/0 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.1:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9e0139b40 con 0x7fc9e00a67c0 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 1 librados: init done 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9dc003b80 con 0x7fc9e00a67c0 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9dc004070 con 0x7fc9e00a67c0 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d77fe640 4 set_mon_vals no callback set 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9d67fc640 10 monclient: dump: 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.497 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.476+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc9c80065c0 con 0x7fc9c8002860 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7fc9dc004210 con 0x7fc9e00a67c0 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.8 v1:172.21.15.1:6793/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7fc9dc04dfe0 con 0x7fc9e00a67c0 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9d67fc640 10 client.204776.objecter ms_dispatch 0x7fc9e0000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9d67fc640 3 client.204776.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9d67fc640 3 client.204776.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9d67fc640 20 client.204776.objecter dump_active .. 0 homeless 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.477+0000 7fc9e66a5640 10 monclient: get_auth_request con 0x7fc9c4041750 auth_method 0 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9e56a3640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6792/0 conn(0x7fc9c8002860 legacy=0x7fc9c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 client.204776.objecter ms_handle_connect 0x7fc9c8002860 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 client.204776.objecter resend_mon_ops 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477676690 0 0) 0x7fc9c80065c0 con 0x7fc9c8002860 2024-08-19T10:09:41.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 monclient(hunting): my global_id is 204739 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e013a730 con 0x7fc9c8002860 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 10 client.204776.objecter ms_handle_connect 0x7fc9c4041750 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.479+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3768819371 0 0) 0x7fc9e013a730 con 0x7fc9c8002860 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c80065c0 con 0x7fc9c8002860 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d0001d30 con 0x7fc9c8002860 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 10 monclient: dump: 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.500 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4276411212 0 0) 0x7fc9c80065c0 con 0x7fc9c8002860 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9c800a160 con 0x7fc9c8002860 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc9d0004a90 con 0x7fc9c8002860 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6792/0 conn(0x7fc9c8002860 legacy=0x7fc9c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.480+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6792/0 conn(0x7fc9c8002860 legacy=0x7fc9c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.486+0000 7fc9e8108640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T10:09:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.486+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc9c8009e70 con 0x7fc9c8009130 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9e56a3640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6792/0 conn(0x7fc9c8009130 legacy=0x7fc9c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 client.204776.objecter ms_handle_connect 0x7fc9c8009130 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 client.204776.objecter resend_mon_ops 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 178294004 0 0) 0x7fc9c8009e70 con 0x7fc9c8009130 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient(hunting): my global_id is 204748 2024-08-19T10:09:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c80065c0 con 0x7fc9c8009130 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3861959607 0 0) 0x7fc9c80065c0 con 0x7fc9c8009130 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c8009e70 con 0x7fc9c8009130 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7fc9d00053d0 con 0x7fc9c8009130 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 10 monclient: dump: 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.489+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2152787362 0 0) 0x7fc9c8009e70 con 0x7fc9c8009130 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 --> v1:172.21.15.82:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc9c800c3f0 con 0x7fc9c8009130 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 <== mon.7 v1:172.21.15.82:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7685 (unknown 0 0 3218480524) 0x7fc9d0008540 con 0x7fc9c8009130 2024-08-19T10:09:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9d67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9d67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6792/0 conn(0x7fc9c8009130 legacy=0x7fc9c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9d67fc640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6792/0 conn(0x7fc9c8009130 legacy=0x7fc9c80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 152, 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 247360, 2024-08-19T10:09:41.507 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-08-19T10:09:41.508 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.509 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "0e9f827b-9129-41b7-89ad-24f744d49b77", 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-08-19T09:49:28.615648Z", 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-08-19T10:09:41.511 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-08-19T10:09:41.512 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.513 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.514 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "f", 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789", 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6789/0", 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.516 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790", 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6790/0", 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 3, 2024-08-19T10:09:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "g", 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790", 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6790/0", 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.518 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 4, 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791", 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6791/0", 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 5, 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "h", 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791", 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6791/0", 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 6, 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "d", 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792", 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6792/0", 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 7, 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "i", 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792", 2024-08-19T10:09:41.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.82:6792/0", 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 8, 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "e", 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-08-19T10:09:41.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793", 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6793/0", 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-08-19T10:09:41.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.526 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "squid", 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-08-19T10:09:41.527 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-08-19T10:09:41.528 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 10 librados: watch_flush enter 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 10 librados: watch_flush exit 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 20 client.204776.objecter shutdown clearing up homeless session... 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 10 client.204776.objecter successfully canceled tick 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.82:6812/2549104415 conn(0x7fc9c4041750 legacy=0x7fc9c4043c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 10 monclient: shutdown 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 >> v1:172.21.15.1:6793/0 conn(0x7fc9e00a67c0 legacy=0x7fc9e013ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e66a5640 1 -- 172.21.15.1:0/216775041 reap_dead start 2024-08-19T10:09:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e66a5640 1 -- 172.21.15.1:0/216775041 reap_dead start 2024-08-19T10:09:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 shutdown_connections 2024-08-19T10:09:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 >> 172.21.15.1:0/216775041 conn(0x7fc9e00ac4b0 msgr2=0x7fc9e00aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 shutdown_connections 2024-08-19T10:09:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 -- 172.21.15.1:0/216775041 wait complete. 2024-08-19T10:09:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.490+0000 7fc9e8108640 1 librados: shutdown 2024-08-19T10:09:41.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T10:09:41.531 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T10:09:41.532 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T10:09:41.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.629+0000 7f606a18a640 10 monclient: get_monmap_and_config 2024-08-19T10:09:41.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.629+0000 7f606a18a640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.629+0000 7f606a18a640 10 monclient: monmap: 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:41.630390+0000 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:41.630390+0000 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.632 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 1 Processor -- start 2024-08-19T10:09:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 1 -- start start 2024-08-19T10:09:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 10 monclient: init 2024-08-19T10:09:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 10 monclient: _add_conns ranks=[8,5,6,2,3,4,7,1,0] 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 10 monclient(hunting): picked mon.noname-e con 0x7f60640a67c0 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 10 monclient(hunting): picked mon.noname-h con 0x7f60640a6ba0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 10 monclient(hunting): picked mon.noname-d con 0x7f606413f5c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60640a3b40 con 0x7f60640a6ba0 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60640a8130 con 0x7f606413f5c0 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.630+0000 7f606a18a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606406ede0 con 0x7f60640a67c0 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.631+0000 7f606a18a640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.631+0000 7f606a18a640 10 monclient(hunting): authenticate will time out at 1837.707017s 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.631+0000 7f60637fe640 1 --1- >> v1:172.21.15.1:6793/0 conn(0x7f60640a67c0 0x7f60640a3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:57226/0 (socket says 172.21.15.1:57226) 2024-08-19T10:09:41.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.631+0000 7f60637fe640 1 -- 172.21.15.1:0/3324486305 learned_addr learned my addr 172.21.15.1:0/3324486305 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f60637fe640 1 -- 172.21.15.1:0/3324486305 >> v1:172.21.15.1:6793/0 conn(0x7f60640a67c0 legacy=0x7f60640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6063fff640 1 -- 172.21.15.1:0/3324486305 >> v1:172.21.15.1:6792/0 conn(0x7f606413f5c0 legacy=0x7f60641419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f605bfff640 1 -- 172.21.15.1:0/3324486305 >> v1:172.21.15.82:6791/0 conn(0x7f60640a6ba0 legacy=0x7f60640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194918929 0 0) 0x7f606406ede0 con 0x7f60640a67c0 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): my global_id is 204785 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6050003520 con 0x7f60640a67c0 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952659769 0 0) 0x7f60640a8130 con 0x7f606413f5c0 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient(hunting): my global_id is 204612 2024-08-19T10:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f606406ede0 con 0x7f606413f5c0 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1403710943 0 0) 0x7f606406ede0 con 0x7f606413f5c0 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 >> v1:172.21.15.1:6793/0 conn(0x7f60640a67c0 legacy=0x7f60640a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 >> v1:172.21.15.82:6791/0 conn(0x7f60640a6ba0 legacy=0x7f60640a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 monclient: found mon.noname-d 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60640a92a0 con 0x7f606413f5c0 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f605c001d40 con 0x7f606413f5c0 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f606a18a640 5 monclient: authenticate success, global_id 204612 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f606a18a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: dump: 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f606a18a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f605c002010 con 0x7f606413f5c0 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f606a18a640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 1 -- 172.21.15.1:0/3324486305 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f605c003cd0 con 0x7f606413f5c0 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f606a18a640 4 set_mon_vals no callback set 2024-08-19T10:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.633+0000 7f6062ffd640 10 monclient: dump: 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 10 monclient: shutdown 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 1 -- 172.21.15.1:0/3324486305 >> v1:172.21.15.1:6792/0 conn(0x7f606413f5c0 legacy=0x7f60641419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 1 -- 172.21.15.1:0/3324486305 shutdown_connections 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 1 -- 172.21.15.1:0/3324486305 >> 172.21.15.1:0/3324486305 conn(0x7f60640ac4b0 msgr2=0x7f60640a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 1 -- 172.21.15.1:0/3324486305 shutdown_connections 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 1 -- 172.21.15.1:0/3324486305 wait complete. 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 10 monclient: build_initial_monmap 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.634+0000 7f606a18a640 10 monclient: monmap: 2024-08-19T10:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:41.649 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 librados: starting msgr at 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 librados: starting objecter 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 Processor -- start 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 -- start start 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 librados: setting wanted keys 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 librados: calling monclient init 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient: init 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient: _add_conns ranks=[6,2,1,0,8,4,3,5,7] 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient(hunting): picked mon.noname-d con 0x7f60640a67c0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient(hunting): picked mon.noname-b con 0x7f60640a6ba0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient(hunting): picked mon.noname-f con 0x7f606413f5c0 addr v1:172.21.15.82:6789/0 2024-08-19T10:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606413f3a0 con 0x7f606413f5c0 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606413c870 con 0x7f60640a6ba0 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606413a7b0 con 0x7f60640a67c0 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f606a18a640 10 monclient(hunting): authenticate will time out at 1837.711841s 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f60637fe640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f60640a67c0 0x7f6064139990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:34574/0 (socket says 172.21.15.1:34574) 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.635+0000 7f60637fe640 1 -- 172.21.15.1:0/667528036 learned_addr learned my addr 172.21.15.1:0/667528036 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f60637fe640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.1:6792/0 conn(0x7f60640a67c0 legacy=0x7f6064139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 client.?.objecter ms_handle_connect 0x7f60640a67c0 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 client.?.objecter ms_handle_connect 0x7f60640a6ba0 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f605bfff640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.1:6790/0 conn(0x7f60640a6ba0 legacy=0x7f606413a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1199253451 0 0) 0x7f606413a7b0 con 0x7f60640a67c0 2024-08-19T10:09:41.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): my global_id is 204621 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6044004ee0 con 0x7f60640a67c0 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 client.?.objecter ms_handle_connect 0x7f606413f5c0 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6063fff640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.82:6789/0 conn(0x7f606413f5c0 legacy=0x7f60641d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747670117 0 0) 0x7f606413c870 con 0x7f60640a6ba0 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): my global_id is 185141 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f606413a7b0 con 0x7f60640a6ba0 2024-08-19T10:09:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700948783 0 0) 0x7f606413f3a0 con 0x7f606413f5c0 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): my global_id is 224173 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f606413c870 con 0x7f606413f5c0 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2887336490 0 0) 0x7f6044004ee0 con 0x7f60640a67c0 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f606413f3a0 con 0x7f60640a67c0 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f6054001a50 con 0x7f60640a67c0 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): dump: 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.653 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1375000281 0 0) 0x7f606413a7b0 con 0x7f60640a6ba0 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6044004ee0 con 0x7f60640a6ba0 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f604c003610 con 0x7f60640a6ba0 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): dump: 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1067614684 0 0) 0x7f606413c870 con 0x7f606413f5c0 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f606413a7b0 con 0x7f606413f5c0 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f605c003cd0 con 0x7f606413f5c0 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient(hunting): dump: 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.657 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2219086945 0 0) 0x7f606413f3a0 con 0x7f60640a67c0 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.1:6790/0 conn(0x7f60640a6ba0 legacy=0x7f606413a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.82:6789/0 conn(0x7f606413f5c0 legacy=0x7f60641d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 monclient: found mon.d 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606413cb50 con 0x7f60640a67c0 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: _finish_auth 0 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:09:41.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 5 monclient: authenticate success, global_id 204621 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 10 monclient: _renew_subs 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60640a9d30 con 0x7f60640a67c0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 10 client.204621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 10 monclient: _renew_subs 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f606413c590 con 0x7f60640a67c0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f606a18a640 1 librados: init done 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6054004dd0 con 0x7f60640a67c0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f60540052c0 con 0x7f60640a67c0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f6060ff9640 10 monclient: dump: 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.638+0000 7f60627fc640 4 set_mon_vals no callback set 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.639+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f6054011720 con 0x7f60640a67c0 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.639+0000 7f606a18a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.639+0000 7f606a18a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.639+0000 7f606a18a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.639+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6048005080 con 0x7f60640a67c0 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.640+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f605404e070 con 0x7f60640a67c0 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.640+0000 7f6060ff9640 10 client.204621.objecter ms_dispatch 0x7f6064000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.640+0000 7f6060ff9640 3 client.204621.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.640+0000 7f6060ff9640 3 client.204621.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.640+0000 7f6060ff9640 20 client.204621.objecter dump_active .. 0 homeless 2024-08-19T10:09:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.640+0000 7f605bfff640 10 monclient: get_auth_request con 0x7f6044041880 auth_method 0 2024-08-19T10:09:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.642+0000 7f6060ff9640 10 client.204621.objecter ms_handle_connect 0x7f6044041880 2024-08-19T10:09:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.644+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f60540176d0 con 0x7f60640a67c0 2024-08-19T10:09:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.644+0000 7f6060ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.644+0000 7f6060ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.901+0000 7f606a18a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T10:09:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.901+0000 7f606a18a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T10:09:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.901+0000 7f606a18a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:41.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.901+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6048005c40 con 0x7f60640a67c0 2024-08-19T10:09:41.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f6060ff9640 1 -- 172.21.15.1:0/667528036 <== mon.6 v1:172.21.15.1:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f6054016fd0 con 0x7f60640a67c0 2024-08-19T10:09:41.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f6060ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T10:09:41.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f6060ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:41.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.906 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=2287616311,config=2586702516,health=4246462203,kv=1455257465,logm=3758533525}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=11,logm=49} crc {auth=3672105033,config=2586702516,health=1317928900,kv=1455257465,logm=53068766}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=25,config=2,health=10,kv=12,logm=51} crc {auth=1346317860,config=2586702516,health=197336704,kv=1455257465,logm=3003435141}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=1685254770,config=2586702516,health=4246462204,kv=3949047626,logm=3752388156}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=27,config=1,health=12,kv=11,logm=49} crc {auth=3672105033,config=1228700967,health=4246462203,kv=2345797195,logm=1189236091}) 2024-08-19T10:09:41.907 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=4067243287,config=2586702516,health=4246462203,kv=1455257465,logm=604956810}) 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=3672105033,config=2586702516,health=4246462203,kv=1455257465,logm=3823821013}) 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2834068128,config=2586702516,health=1847981124,kv=46702902,logm=1104176307}) 2024-08-19T10:09:41.908 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.904+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=2834068128,config=2586702516,health=4246462203,kv=1455257465,logm=2728699891}) 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.906+0000 7f606a18a640 10 librados: watch_flush enter 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.906+0000 7f606a18a640 10 librados: watch_flush exit 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.906+0000 7f606a18a640 20 client.204621.objecter shutdown clearing up homeless session... 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.906+0000 7f606a18a640 10 client.204621.objecter successfully canceled tick 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.906+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.82:6812/2549104415 conn(0x7f6044041880 legacy=0x7f6044043d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.906+0000 7f606a18a640 10 monclient: shutdown 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.907+0000 7f606a18a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:41.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.907+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 >> v1:172.21.15.1:6792/0 conn(0x7f60640a67c0 legacy=0x7f6064139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:41.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.908+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 shutdown_connections 2024-08-19T10:09:41.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.908+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 >> 172.21.15.1:0/667528036 conn(0x7f60640ac4b0 msgr2=0x7f60640a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:41.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.908+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 shutdown_connections 2024-08-19T10:09:41.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f606a18a640 1 -- 172.21.15.1:0/667528036 wait complete. 2024-08-19T10:09:41.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f606a18a640 1 librados: shutdown 2024-08-19T10:09:41.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482726315}) 2024-08-19T10:09:41.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.912 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631064793}) 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2971909198}) 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2590861714}) 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.913 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=533547734}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1456960711}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2430469097}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624358949}) 2024-08-19T10:09:41.914 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=104580796}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675996355}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211425996}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=483417919}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3471483955}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3313181472}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3498880207}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986928352}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456915964}) 2024-08-19T10:09:41.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.915+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=892459183}) 2024-08-19T10:09:41.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.920+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.920+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2789718141}) 2024-08-19T10:09:41.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828890936}) 2024-08-19T10:09:41.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1221164778}) 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3668486310}) 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=253252742}) 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2152656153}) 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1502933384}) 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114315208}) 2024-08-19T10:09:41.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:41.921+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3903595034}) 2024-08-19T10:09:41.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869837687}) 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3996961666}) 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.132 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=952260736}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1302713901}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3258146499}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1727245353}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4063927854}) 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.133 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442894534}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2199157841}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3647779429}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962460492}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3638380214}) 2024-08-19T10:09:42.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201893203}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3071877053}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3991093534}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4167222536}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722240362}) 2024-08-19T10:09:42.135 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3523675122}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585998425}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447445688}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1291034720}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=922721802}) 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.136 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2205592794}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3993847204}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3146916934}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590617062}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.016+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4137422515}) 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.137 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002124847}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3754625008}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4177159774}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4279723724}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.138 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1765671343}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=584394870}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=908836440}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428319654}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.017+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2303976901}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.139 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238722856}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633195079}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1899203053}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=523739700}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2785012145}) 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.140 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=109434510}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2501044958}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824368598}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.018+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1568244732}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043675705}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.141 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969905915}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.019+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095049804}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1850392948}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1570020865}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.142 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3447127609}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4002370072}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30343418}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.020+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3378829834}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188979054}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765366960}) 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.143 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3954231850}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3371489725}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=221060263}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2487835029}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.144 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1747198126}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678039548}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.021+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2386939827}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937019749}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=231412291}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1640322823}) 2024-08-19T10:09:42.145 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2939901746}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1312243314}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2171290359}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2420124870}) 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.146 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004429608}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.022+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2124723522}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2830613779}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176073079}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3314424419}) 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.147 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3781845337}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1281578490}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2208436779}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1068036548}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557429798}) 2024-08-19T10:09:42.148 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.023+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900165295}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1180693191}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231379918}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3701885558}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=157167048}) 2024-08-19T10:09:42.149 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3910675964}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1709818212}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2345454449}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287672656}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.024+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1239064002}) 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.025+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.150 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.025+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.025+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3611166673}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.025+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.025+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725186608}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2103824115}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1960043745}) 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.151 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1784671094}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2495490367}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980379857}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836863645}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.026+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1243031586}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2177034183}) 2024-08-19T10:09:42.152 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2846051291}) 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4083052323}) 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2147682413}) 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.153 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2817216734}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3106322207}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3280139713}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335312765}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.027+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1992100204}) 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.154 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4215233661}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009098406}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988850382}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1712730040}) 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.155 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1068677618}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=222426348}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=601752518}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867425927}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.028+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=730078592}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.156 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429553101}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118128596}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2212253282}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2423553662}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=665889087}) 2024-08-19T10:09:42.157 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3488323064}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2084639415}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728430311}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.029+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2326054225}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.030+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.030+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.030+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3419715781}) 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.158 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842212247}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2481874659}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2660977782}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3693123754}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.159 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2218942018}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2611570717}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554727424}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.031+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2992093340}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786221227}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266327731}) 2024-08-19T10:09:42.160 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=798188924}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3601736234}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3844728586}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4163545542}) 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.161 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=45719572}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896301882}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.032+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=766516904}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.033+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.033+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.033+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=934939485}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942333040}) 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.162 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=697995902}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=583880019}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=298713601}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1051665294}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.163 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2240014686}) 2024-08-19T10:09:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153351972}) 2024-08-19T10:09:42.164 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.034+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=308581176}) 2024-08-19T10:09:42.165 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.165 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.165 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758400199}) 2024-08-19T10:09:42.165 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.165 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.165 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935334007}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3557599314}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2550820732}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1236876844}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.166 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1139856681}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3520216113}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352998832}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.035+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=533233196}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808710301}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45590671}) 2024-08-19T10:09:42.167 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=403897592}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3211549807}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1670835966}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2836185429}) 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.168 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4159931935}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350475545}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.036+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1221415003}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987401583}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964184510}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.169 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1898534448}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1882548302}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1926450174}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.037+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1490110233}) 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.170 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=665846903}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125017336}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.038+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3435537578}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=601358280}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465045625}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3321551264}) 2024-08-19T10:09:42.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1015110109}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4273341533}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3091854969}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1622094295}) 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675872721}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.039+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745001863}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571221179}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716693490}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225768390}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1692030300}) 2024-08-19T10:09:42.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2151335279}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1266819399}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1171654649}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972782472}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.040+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2854175984}) 2024-08-19T10:09:42.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396742499}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412097868}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1560216418}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=729461020}) 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=600092608}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2918229429}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3753617215}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285221006}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.041+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2866365007}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2067092702}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466590269}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2007153832}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4008622744}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1360038422}) 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3113567121}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2575741719}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460685254}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.043+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2433077196}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905123143}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955458220}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2390728274}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=655342830}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4174324449}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2935519613}) 2024-08-19T10:09:42.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1839113604}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690500705}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.044+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=696983768}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3145260393}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=412295328}) 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1382054005}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1637093060}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2427733674}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3919180902}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1043033290}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656228694}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.045+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=485867607}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.046+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.046+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.046+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481712917}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.046+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227045885}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2001939176}) 2024-08-19T10:09:42.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2612992846}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4086021851}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2103731781}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2343479893}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047316143}) 2024-08-19T10:09:42.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.047+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2714997367}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319748626}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173622928}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3092896675}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.184 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=192165893}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4203773562}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=394828511}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2114972024}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077561341}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.048+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=985100625}) 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.185 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1255261825}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146635849}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=879918191}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2401129742}) 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.186 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2558111240}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=15834710}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=133236746}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085633654}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.049+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2738736618}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.187 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394104438}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760510138}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=331787935}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3247285041}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2222339715}) 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.188 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2973735852}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=837985118}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.061+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.062+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562293786}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.062+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2077688531}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799216293}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.189 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264924450}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=430914702}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1787907150}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2317359205}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1132882860}) 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.190 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3948974239}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362755211}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.063+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=765203960}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388535693}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237952945}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3889732705}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3000368666}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1942510208}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3855732561}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3072574964}) 2024-08-19T10:09:42.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643359208}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2625547438}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032652636}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896751965}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4282194039}) 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135823041}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2999144206}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=399250004}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4244726559}) 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727270035}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.065+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2120091745}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3683407702}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4120456704}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2574320251}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3579046360}) 2024-08-19T10:09:42.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1305956195}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=676735019}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=785649269}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056913362}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4128776585}) 2024-08-19T10:09:42.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626181361}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505840819}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=347375264}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3637804366}) 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2023438669}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3260889775}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3253356065}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159701709}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1108611675}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466707321}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316199791}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1073289656}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1346414462}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1080258715}) 2024-08-19T10:09:42.199 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=926175320}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3721692079}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704930794}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3087526982}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692768714}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.200 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917082635}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4005297135}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=379830573}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1727410393}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1081679748}) 2024-08-19T10:09:42.201 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1262657939}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90279417}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=997168164}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099773765}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3413012366}) 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.202 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3303730204}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=95620384}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2161527849}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2022578163}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.203 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=240678041}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667765100}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3706015133}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3237999516}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482773209}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3081006412}) 2024-08-19T10:09:42.204 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4126597360}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3920813472}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.072+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2346901106}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3243102930}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289209070}) 2024-08-19T10:09:42.205 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.073+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1759324710}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060895338}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671425230}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1958943638}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.206 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=530279527}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4287440996}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3604891592}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1989262235}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949877055}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.074+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1207392029}) 2024-08-19T10:09:42.207 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=438302655}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048681457}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2210252531}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2827984501}) 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=867037687}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=101130487}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1796673870}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948623173}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.075+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3243552480}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.209 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904651913}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111073011}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1009680558}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=200637814}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.210 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=269059969}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2416613671}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.076+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.077+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3123113965}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.077+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.077+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645050482}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.077+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=697601460}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673040798}) 2024-08-19T10:09:42.211 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2124769999}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=657195132}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=348396372}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1313973711}) 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.212 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1855225908}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1979629825}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352301762}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.078+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=728801469}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945585427}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258068521}) 2024-08-19T10:09:42.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2174936438}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757460757}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1319506780}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2682652668}) 2024-08-19T10:09:42.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3932392637}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725514545}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.079+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1872886671}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2111480420}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712530422}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=449274028}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=193539865}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4277871552}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=419745845}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2219375379}) 2024-08-19T10:09:42.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843482497}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.080+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=889019349}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2210327341}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3716727754}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=462708666}) 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.217 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3892774907}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=980507320}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3194915981}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1429316816}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014910072}) 2024-08-19T10:09:42.218 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.081+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2185948579}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2612298260}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322826030}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.082+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2471326026}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2939645529}) 2024-08-19T10:09:42.219 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2963792347}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=489570236}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=36474305}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988843490}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.083+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2921673547}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.220 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418455347}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3263033005}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1025097642}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3020604011}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2371322411}) 2024-08-19T10:09:42.221 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3136458024}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=640271692}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861853240}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.084+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1932232192}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341643391}) 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.222 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2314352812}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1681455573}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1278527731}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1797962298}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.223 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3544113236}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2041641240}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559564300}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.085+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2463814402}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810813603}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143210140}) 2024-08-19T10:09:42.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=849445640}) 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1547395232}) 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1915622975}) 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2696161349}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3546804985}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119738443}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.086+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=122051871}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263485058}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881040290}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1140481139}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.087+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=308251062}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3776108019}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1182573193}) 2024-08-19T10:09:42.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3008093395}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469005271}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.088+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2466150892}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71626072}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3789027333}) 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4025141374}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3436329507}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=73681825}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=557893649}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1734874311}) 2024-08-19T10:09:42.229 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271758032}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.089+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=792858753}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651070681}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916625199}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3790871791}) 2024-08-19T10:09:42.230 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=581883611}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2341094866}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2852353231}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1531463591}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299455458}) 2024-08-19T10:09:42.231 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.090+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23169965}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3670736668}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=637605743}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414247545}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.232 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=282976028}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2710700122}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=81457005}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4115860924}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482196394}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.091+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3231616980}) 2024-08-19T10:09:42.233 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278573246}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939196101}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=284163885}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4061624894}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.234 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4121260347}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1366893210}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3570873200}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759182985}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.092+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3566641013}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285547030}) 2024-08-19T10:09:42.235 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=463098428}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3043944027}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=567235574}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2516515165}) 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.236 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3690853817}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4227183768}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186961850}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.102+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=795032429}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975614583}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825699194}) 2024-08-19T10:09:42.237 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=450081640}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1553649214}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2964964526}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=262589957}) 2024-08-19T10:09:42.238 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1979660378}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256149834}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.103+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=927433855}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.104+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.104+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.104+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919070975}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527387457}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2291939021}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=129409614}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2611324466}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3454740285}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2603417245}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919309514}) 2024-08-19T10:09:42.240 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.105+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=74595677}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178496602}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2001377118}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4211595883}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1642133725}) 2024-08-19T10:09:42.241 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1300796340}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1029518371}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3027823524}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436982837}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.106+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2163479436}) 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.242 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=334163996}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464067239}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1448455943}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1180926880}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.243 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3872472288}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=21672948}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1617786727}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272996998}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.107+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1723866683}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874313980}) 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.244 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802314170}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1570930454}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4200922878}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1813883847}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.108+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=464588252}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.109+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.109+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.109+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1820064779}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.109+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.109+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176110305}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.109+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=77879793}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3537831786}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3127327802}) 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2173712804}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2130920897}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2947677701}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=932874308}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1387904554}) 2024-08-19T10:09:42.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433851644}) 2024-08-19T10:09:42.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.110+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=608643626}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934736765}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1313204125}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4012048836}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4176610921}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=85858536}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3480087243}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2018701814}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.218+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829353650}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.219+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=738259993}) 2024-08-19T10:09:42.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476519622}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3877673171}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3915550632}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4175050321}) 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4174209014}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2986301676}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2782819193}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184705101}) 2024-08-19T10:09:42.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4048013747}) 2024-08-19T10:09:42.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=292818776}) 2024-08-19T10:09:42.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686277824}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3186318286}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2855807741}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2836228952}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3457287533}) 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.916 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.901+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.902+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=251229482}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.902+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.902+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677279758}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.902+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2245991391}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1732365827}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.917 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2655899041}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2222640439}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2379514403}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3642358667}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.918 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4245004387}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1738648380}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892368109}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.903+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788439389}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=847638693}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=721688608}) 2024-08-19T10:09:42.919 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1749731104}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2826649918}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3772500579}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1701959187}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.920 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1682022549}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543164141}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.905+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2016417649}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1462562316}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216811743}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3462480863}) 2024-08-19T10:09:42.921 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3182266218}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2364460384}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=27488826}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3673717856}) 2024-08-19T10:09:42.922 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419829296}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.907+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=29974446}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.908+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.908+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.908+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472201242}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536685637}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1469460818}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=712554}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4156465680}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2242459606}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3352667539}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340829735}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.909+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=390218907}) 2024-08-19T10:09:42.924 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230093213}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222601998}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3705633376}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2674307708}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.925 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3542924346}) 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.910+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2002580143}) 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=519810867}) 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332089208}) 2024-08-19T10:09:42.926 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.911+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508193614}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.924+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518753437}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176145852}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=931230176}) 2024-08-19T10:09:42.929 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3295150959}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=344862102}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3945402450}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4165652370}) 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.930 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364310111}) 2024-08-19T10:09:42.931 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.925+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=731864759}) 2024-08-19T10:09:42.931 ERROR:tasks.mon_thrash.mon_thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 272, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 321, in _do_thrash while not self.stopping.is_set(): AttributeError: 'bool' object has no attribute 'is_set' 2024-08-19T10:09:42.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747437372}) 2024-08-19T10:09:42.934 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846468642}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3906277190}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1696843217}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4119812537}) 2024-08-19T10:09:42.935 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3304296430}) 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3447796457}) 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.931+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779491690}) 2024-08-19T10:09:42.936 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.932+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2449818863}) 2024-08-19T10:09:43.000 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963636513}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116985717}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3963676002}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715769365}) 2024-08-19T10:09:43.001 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2213231633}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3136983568}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2779870733}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559397648}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3484468105}) 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186439913}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437470863}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1997768603}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=933258373}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3155070748}) 2024-08-19T10:09:43.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3414211882}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1742294666}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673175391}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.987+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2617155591}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1841659277}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2012884986}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1252409205}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=824820670}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=539206333}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3865670172}) 2024-08-19T10:09:43.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715054657}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208929977}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.988+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3100804839}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1324520901}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2510526707}) 2024-08-19T10:09:43.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1586911219}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=60406107}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1021869741}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3728666264}) 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3761344022}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599533011}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.989+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2429182133}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1347813722}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610191205}) 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.008 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3103164362}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=973712843}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2683632625}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=9009568}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.009 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=801487370}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450564019}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3721695208}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068907782}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136869343}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4058917181}) 2024-08-19T10:09:43.010 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3756040918}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=700809757}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=617328248}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3231232854}) 2024-08-19T10:09:43.011 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850745284}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.992+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1081571056}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1527918886}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428133792}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764685372}) 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3251718319}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4274984736}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1306869213}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=24722447}) 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.013 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193772274}) 2024-08-19T10:09:43.014 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:42.994+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3148456618}) 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989059389}) 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421110616}) 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.292 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1223989479}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2295026840}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1297364594}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=981971665}) 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3547338197}) 2024-08-19T10:09:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097826628}) 2024-08-19T10:09:43.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2616830002}) 2024-08-19T10:09:43.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245117154}) 2024-08-19T10:09:43.938 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231905513}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1998366304}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1662101688}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.939 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2396042700}) 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1825131365}) 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=811019599}) 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.940 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165690556}) 2024-08-19T10:09:43.941 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.935+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1743834729}) 2024-08-19T10:09:43.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862244273}) 2024-08-19T10:09:43.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.944 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943669612}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3974538029}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2171044304}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3542957821}) 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.945 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4063750189}) 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3335815609}) 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009779122}) 2024-08-19T10:09:43.946 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.942+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=423583302}) 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673017705}) 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199174821}) 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.961 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2900251430}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.958+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2542836428}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1558851678}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.962 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=656420821}) 2024-08-19T10:09:43.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2450606813}) 2024-08-19T10:09:43.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:43.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160586091}) 2024-08-19T10:09:43.963 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:43.959+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1130135272}) 2024-08-19T10:09:44.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.290+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603346931}) 2024-08-19T10:09:44.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410992277}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2058477638}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1204917938}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3017037042}) 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3381843290}) 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1391582004}) 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684573093}) 2024-08-19T10:09:44.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.291+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1110214050}) 2024-08-19T10:09:44.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.364 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1693527210}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164542092}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=980067889}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=410684809}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=177345400}) 2024-08-19T10:09:44.365 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2843678628}) 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2903794179}) 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544195772}) 2024-08-19T10:09:44.366 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.361+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=323222095}) 2024-08-19T10:09:44.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1701504901}) 2024-08-19T10:09:44.387 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1219912239}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=147512885}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2841899258}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1805705862}) 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.388 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=435460183}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2340447986}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876028063}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.380+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2627060178}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.381+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.381+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.381+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465104595}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.381+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.381+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.389 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222524407}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=57084958}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=420829994}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1757829048}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3609082742}) 2024-08-19T10:09:44.390 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=848000837}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036474397}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.382+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2046935171}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176278078}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3569002666}) 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.391 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1287071748}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3679840887}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=166761164}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1498239603}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1822778902}) 2024-08-19T10:09:44.392 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899981787}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.383+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1042106345}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366723710}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=148403476}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3983134500}) 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.393 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4244102842}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2355099012}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1352932690}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3741802230}) 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.394 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164519918}) 2024-08-19T10:09:44.395 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.385+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1707208030}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860621363}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988261210}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1270293454}) 2024-08-19T10:09:44.399 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=93329122}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2115916152}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=818436106}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591062644}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669871794}) 2024-08-19T10:09:44.400 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=511149277}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=174587689}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=35576373}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=767551720}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3089210264}) 2024-08-19T10:09:44.401 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=17312803}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4131203139}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=60274705}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435791816}) 2024-08-19T10:09:44.402 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.389+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1889150989}) 2024-08-19T10:09:44.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731252111}) 2024-08-19T10:09:44.951 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146651761}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.948+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3736509664}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2458096974}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=291217569}) 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.952 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=350627192}) 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=219096215}) 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883440350}) 2024-08-19T10:09:44.953 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.949+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=985216638}) 2024-08-19T10:09:44.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1066822281}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035049807}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1698652192}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1850239394}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2335189006}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2491573218}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3676497625}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206339319}) 2024-08-19T10:09:44.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:44.963+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1118326606}) 2024-08-19T10:09:45.074 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.074 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.074 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968824977}) 2024-08-19T10:09:45.074 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.074 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.074 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897663575}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1217783325}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2003161235}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1652447746}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1932795607}) 2024-08-19T10:09:45.075 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2158150468}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018697585}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.064+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756358896}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388784403}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3343830036}) 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.076 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1891295254}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1877703193}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3572132696}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2105132445}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.077 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=680406128}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987876883}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.066+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=979848062}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.067+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.067+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.067+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=74109027}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.067+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.067+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009206664}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=364928954}) 2024-08-19T10:09:45.078 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2988185035}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=482799261}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1532816638}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=897571365}) 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.079 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106345093}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.068+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3133941908}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2370565670}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965005657}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4039539238}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.080 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757380399}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.069+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1524378699}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4247704361}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1081669900}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280090295}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.070+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4194944179}) 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.081 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2210202822}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748497955}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1852376304}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=867393044}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.082 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2566108212}) 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3306615107}) 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=846449035}) 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944512367}) 2024-08-19T10:09:45.083 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.071+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=503423611}) 2024-08-19T10:09:45.171 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119913295}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403282233}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3611700012}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2010837762}) 2024-08-19T10:09:45.172 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1500342831}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2787745054}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4111289647}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519853403}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.160+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3891821590}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.173 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112694580}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755102699}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414643936}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1228887775}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.174 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2628712382}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3804639561}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=348403623}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586176260}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.161+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3568384041}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244512417}) 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.175 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2295007350}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2469066398}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.162+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4226304785}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=423109070}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3423765957}) 2024-08-19T10:09:45.176 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1272333166}) 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909073727}) 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.163+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1195598248}) 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556743737}) 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888575287}) 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.178 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1425310424}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1836058141}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2018151895}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1011369455}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.164+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2761206245}) 2024-08-19T10:09:45.179 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628990085}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.165+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2924911112}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866948898}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434381702}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2525232208}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.180 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380415946}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3559493255}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3137899416}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1926504764}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.166+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192993576}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.167+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3706731893}) 2024-08-19T10:09:45.181 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518946839}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439459592}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3656297072}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.182 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2599668137}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1860890083}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2791410207}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3137819430}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409008583}) 2024-08-19T10:09:45.183 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.169+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3438479542}) 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3266448763}) 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069578896}) 2024-08-19T10:09:45.191 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1098744514}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2086594496}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3567834883}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2929016665}) 2024-08-19T10:09:45.192 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3515642013}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077298929}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.182+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2590226242}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703199016}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967749053}) 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.193 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3818928404}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=285347991}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2171767002}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3353173817}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.194 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2626091823}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087770833}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.184+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1893832720}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.185+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.185+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.185+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808674276}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.185+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241086478}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.195 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4097032543}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2956427185}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1937685512}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2095767973}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.196 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1283507164}) 2024-08-19T10:09:45.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597124618}) 2024-08-19T10:09:45.197 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.186+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1950291580}) 2024-08-19T10:09:45.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106986735}) 2024-08-19T10:09:45.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.412 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.386+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232672285}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4284612622}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1109411828}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1617999605}) 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.413 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3212267513}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1886164196}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867010610}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.387+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4277821039}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2091117330}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.414 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3535519239}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036341913}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320256403}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=281674624}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1381532371}) 2024-08-19T10:09:45.415 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3241644862}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308910832}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.388+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3462775985}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629580901}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781219126}) 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.416 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143515062}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=59837621}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2997328115}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=578703609}) 2024-08-19T10:09:45.417 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=147986840}) 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762598076}) 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.390+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3309227303}) 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101198749}) 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.418 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813002108}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2949365840}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4156046245}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2132990108}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.419 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3024299182}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3444011191}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903151828}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.392+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3701340116}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.393+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.393+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3595128500}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152003861}) 2024-08-19T10:09:45.420 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3693654017}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2017838096}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3007205776}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2758354750}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.421 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1394930549}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185296940}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.394+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=636892524}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310336186}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135180133}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.422 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1878340200}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1869265177}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=37991556}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3090961554}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446764858}) 2024-08-19T10:09:45.423 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752267209}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.395+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3907144587}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561611189}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186924892}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2991606528}) 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.424 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3186230270}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1142006339}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1678049998}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2155003767}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439284016}) 2024-08-19T10:09:45.425 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.397+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3463971824}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=526295543}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1357751313}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=638406834}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.426 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=358288061}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2340007198}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2539838556}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.398+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.399+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.399+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=355244547}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.399+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.399+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749087937}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.399+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=864472488}) 2024-08-19T10:09:45.427 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666650625}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810536419}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3107315211}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=136725484}) 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.428 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=676311235}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2221151641}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2325560474}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623547087}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.400+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2981826886}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.429 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860191045}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16944686}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=876272998}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.401+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3339402481}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2956247096}) 2024-08-19T10:09:45.430 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1699618415}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3222085041}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894624514}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.402+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=208892247}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227615361}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4083766855}) 2024-08-19T10:09:45.431 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3235254431}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2961723680}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=62245238}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2647334579}) 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.432 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=523817195}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834354054}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.403+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943809785}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598764469}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=89236700}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.433 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=585534137}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2743820883}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=832799307}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2181524666}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.434 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=285175148}) 2024-08-19T10:09:45.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144612039}) 2024-08-19T10:09:45.435 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.405+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796208889}) 2024-08-19T10:09:45.590 INFO:tasks.daemonwatchdog.daemon_watchdog:MonitorThrasher failed 2024-08-19T10:09:45.590 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/daemonwatchdog.py", line 42, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/daemonwatchdog.py", line 120, in watch thrasher.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T10:09:45.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2793954862}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390209691}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1057281447}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2597233226}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.957 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2704671777}) 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=765349058}) 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2966600853}) 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215310724}) 2024-08-19T10:09:45.958 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.953+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2196953098}) 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612731391}) 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177504781}) 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.965 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2582045233}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2168016817}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2491410453}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3609058909}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.966 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1935490055}) 2024-08-19T10:09:45.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38289231}) 2024-08-19T10:09:45.967 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.962+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3436091570}) 2024-08-19T10:09:45.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.993 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1391061597}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2597066625}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=545745255}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1478206091}) 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.990+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1050225775}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=638226828}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1487659461}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929001995}) 2024-08-19T10:09:45.995 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:45.991+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1688795994}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3329133633}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780895896}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2035822946}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.213 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4254458969}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1723968392}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1813960635}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2053810943}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965433835}) 2024-08-19T10:09:46.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4096956527}) 2024-08-19T10:09:46.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208989765}) 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492124451}) 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3209298716}) 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=304606960}) 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1699949984}) 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.226 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=246094476}) 2024-08-19T10:09:46.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.227 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1098867559}) 2024-08-19T10:09:46.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657383930}) 2024-08-19T10:09:46.228 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1450369715}) 2024-08-19T10:09:46.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.245 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780718238}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4244227960}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4266105344}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3754633856}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.246 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4205720986}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1385582299}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3289757280}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828915026}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2207860810}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=624661090}) 2024-08-19T10:09:46.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658794914}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2599199634}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=21023589}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4271845333}) 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2818944582}) 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1005378103}) 2024-08-19T10:09:46.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728089230}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2072178502}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885923632}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3113343034}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093276463}) 2024-08-19T10:09:46.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3041734475}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3490263948}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=433421559}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984710750}) 2024-08-19T10:09:46.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850365239}) 2024-08-19T10:09:46.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=872343897}) 2024-08-19T10:09:46.263 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.263 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.264 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117282964}) 2024-08-19T10:09:46.265 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.265 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.265 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098278329}) 2024-08-19T10:09:46.265 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.265 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4072057637}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1902353524}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2461256272}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.252+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=32040059}) 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.266 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=130305688}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415847531}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.253+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3022434768}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557376145}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923998392}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.267 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1686044924}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=100106306}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3951235629}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4264352208}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2915061715}) 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.254+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.268 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.255+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285953396}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.255+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=25641197}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820454025}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125119686}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2854707251}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.269 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1905615557}) 2024-08-19T10:09:46.270 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.270 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.270 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1796438760}) 2024-08-19T10:09:46.270 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1901261039}) 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3983720304}) 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099891211}) 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.256+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3083573339}) 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.271 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201050183}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641150644}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3464183436}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=856806162}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1535854812}) 2024-08-19T10:09:46.272 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=93585022}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.257+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=865513449}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191687804}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1484441109}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833172511}) 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.273 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947269142}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3262001463}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.258+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1878082463}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4060323128}) 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.274 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1242525370}) 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1204150116}) 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809538513}) 2024-08-19T10:09:46.275 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.259+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1262291769}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2805706318}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1168445894}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=294176987}) 2024-08-19T10:09:46.293 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2607957920}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2056135682}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3387227591}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3354318946}) 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.294 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357948012}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.271+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4243107203}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=874283276}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461640090}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=439731012}) 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.295 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=548052170}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3527979664}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=894664313}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2327626669}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52178269}) 2024-08-19T10:09:46.296 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.272+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3272458401}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671634119}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746347894}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2884858961}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2344189285}) 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.297 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=12141352}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1681583164}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3351270464}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013237127}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.273+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3311562389}) 2024-08-19T10:09:46.298 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4158335565}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3065841954}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806317747}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1527404390}) 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.299 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2523660078}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3987470689}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1145683520}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360445924}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.274+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2719748051}) 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.300 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182433981}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340297961}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=228506574}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3091941319}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.301 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4175440665}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3714578495}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2459150956}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345863699}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.275+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844200150}) 2024-08-19T10:09:46.302 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196335201}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376196530}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3258910466}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2929246532}) 2024-08-19T10:09:46.303 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2333710420}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2577029124}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2816196981}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546198087}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.276+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=509279854}) 2024-08-19T10:09:46.304 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335955882}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=993962815}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=942586474}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3472245145}) 2024-08-19T10:09:46.305 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2584931992}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4047066478}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4291629164}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870997437}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.277+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3111947528}) 2024-08-19T10:09:46.306 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3250876972}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926964604}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3467256059}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.307 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1204655484}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2481931609}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2355344184}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1761853387}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5331428}) 2024-08-19T10:09:46.308 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.287+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329595152}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155553468}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2832081686}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=331888492}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.309 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=565209677}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4096902088}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=408537152}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3691886400}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203533657}) 2024-08-19T10:09:46.310 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.288+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1967300801}) 2024-08-19T10:09:46.314 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343434935}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293676251}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1190183322}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.315 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=332352785}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=808674061}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3738310051}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2615819668}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443630475}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.300+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=600174146}) 2024-08-19T10:09:46.316 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2039939369}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1346079919}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3629653590}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2296550873}) 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.317 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1402172479}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3794907394}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1427177815}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269044024}) 2024-08-19T10:09:46.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.301+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2316718592}) 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494322218}) 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3135604111}) 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.863 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2084013133}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3866999702}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3111282868}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2999427269}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2436838625}) 2024-08-19T10:09:46.864 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954514148}) 2024-08-19T10:09:46.865 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.860+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2215259704}) 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816874596}) 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3906194316}) 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.977 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3588068582}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=210762600}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1577240031}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=415692057}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.978 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.974+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2983773020}) 2024-08-19T10:09:46.979 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.975+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:46.979 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.975+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034085384}) 2024-08-19T10:09:46.979 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:46.975+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4207781878}) 2024-08-19T10:09:47.002 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.000+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.000+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2455487667}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260669857}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1533559054}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2617898945}) 2024-08-19T10:09:47.003 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3753762609}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1420095344}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=877508020}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714239786}) 2024-08-19T10:09:47.004 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.001+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2827080318}) 2024-08-19T10:09:47.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674411657}) 2024-08-19T10:09:47.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.019 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887744972}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2720257147}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3561837117}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2441540656}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1003923928}) 2024-08-19T10:09:47.020 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2520193215}) 2024-08-19T10:09:47.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175575167}) 2024-08-19T10:09:47.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.013+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1677834002}) 2024-08-19T10:09:47.021 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762255232}) 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342761176}) 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.022 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1264555211}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=42700827}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3941975712}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2920865434}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.023 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3988214312}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255048863}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.014+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=757347006}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=694630038}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492457048}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4157995283}) 2024-08-19T10:09:47.024 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2735696076}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3409231990}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3903050316}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736376480}) 2024-08-19T10:09:47.025 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593829169}) 2024-08-19T10:09:47.026 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.015+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3771912827}) 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563510825}) 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182602604}) 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.214 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3892704072}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1615323965}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1149627423}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=542468696}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1396234216}) 2024-08-19T10:09:47.215 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305165284}) 2024-08-19T10:09:47.216 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.211+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2646757405}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700517519}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=23545300}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3913895958}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.224 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=796221287}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=760305511}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2421331185}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=929030976}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711482476}) 2024-08-19T10:09:47.225 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.221+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3158195814}) 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.322+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.322+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825472594}) 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3422371538}) 2024-08-19T10:09:47.339 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=992081116}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1918642992}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1482702936}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.340 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2461239706}) 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2648319318}) 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355515842}) 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.323+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1337727382}) 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.341 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747099513}) 2024-08-19T10:09:47.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223351206}) 2024-08-19T10:09:47.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.342 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=963045239}) 2024-08-19T10:09:47.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1728784436}) 2024-08-19T10:09:47.343 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1029208071}) 2024-08-19T10:09:47.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3847624739}) 2024-08-19T10:09:47.344 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1617962042}) 2024-08-19T10:09:47.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484066564}) 2024-08-19T10:09:47.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.324+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=748945065}) 2024-08-19T10:09:47.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278032821}) 2024-08-19T10:09:47.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2167982164}) 2024-08-19T10:09:47.346 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3391207509}) 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2986995496}) 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.347 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2368840463}) 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4016995567}) 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.348 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3367539075}) 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084961596}) 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.325+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=658207519}) 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793329184}) 2024-08-19T10:09:47.349 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2718072690}) 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2833162238}) 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.350 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=230709846}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1116535799}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2830760451}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3916647303}) 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.351 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843350028}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.326+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=720680221}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801950676}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2806454058}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2656785597}) 2024-08-19T10:09:47.352 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=378195082}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2380534737}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1246252421}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.353 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3651033240}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833224494}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.327+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1231171288}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372264273}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241767720}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.354 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4161110362}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4169238378}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=255248883}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2964245556}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=354524760}) 2024-08-19T10:09:47.355 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279181347}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.328+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1833134995}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890263100}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909979636}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=958431326}) 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1968551895}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2360572388}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1330816837}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=830266327}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480588868}) 2024-08-19T10:09:47.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.329+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2236426989}) 2024-08-19T10:09:47.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T10:09:47.982 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.979+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.979+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.979+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193778468}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.979+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.979+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342573368}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=965159078}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4229526320}) 2024-08-19T10:09:47.983 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2040810037}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3442486945}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=979638490}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856128544}) 2024-08-19T10:09:47.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.980+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1810150161}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3234771118}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376155987}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1876960848}) 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.988 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.985+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2872232262}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1512571006}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3286986287}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2517205240}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262663355}) 2024-08-19T10:09:47.989 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:47.986+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=85724604}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3250789685}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959182696}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=681081192}) 2024-08-19T10:09:48.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=844917316}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1365063613}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3899640224}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2984626076}) 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.006 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260149562}) 2024-08-19T10:09:48.007 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.003+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2806193359}) 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.053+0000 7f281198a640 10 monclient: get_monmap_and_config 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.053+0000 7f281198a640 10 monclient: build_initial_monmap 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.053+0000 7f281198a640 10 monclient: monmap: 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:48.054888+0000 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:48.054888+0000 2024-08-19T10:09:48.055 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:48.056 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.054+0000 7f281198a640 1 Processor -- start 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.054+0000 7f281198a640 1 -- start start 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.054+0000 7f281198a640 10 monclient: init 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient: _add_conns ranks=[8,6,4,1,2,0,3,7,5] 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient(hunting): picked mon.noname-e con 0x7f280c0a6830 addr v1:172.21.15.1:6793/0 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient(hunting): picked mon.noname-d con 0x7f280c0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient(hunting): picked mon.noname-c con 0x7f280c0a46a0 addr v1:172.21.15.1:6791/0 2024-08-19T10:09:48.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 1 -- --> v1:172.21.15.1:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c0a3c70 con 0x7f280c0a46a0 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c0a33d0 con 0x7f280c0694b0 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 1 -- --> v1:172.21.15.1:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c0a36b0 con 0x7f280c0a6830 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f281198a640 10 monclient(hunting): authenticate will time out at 1844.131138s 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f28027fc640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7f280c0694b0 0x7f280c0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45952/0 (socket says 172.21.15.1:45952) 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f28027fc640 1 -- 172.21.15.1:0/1336317634 learned_addr learned my addr 172.21.15.1:0/1336317634 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f280b7fe640 1 --1- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6791/0 conn(0x7f280c0a46a0 0x7f280c141850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6791/0 says I am v1:172.21.15.1:60478/0 (socket says 172.21.15.1:60478) 2024-08-19T10:09:48.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.055+0000 7f280affd640 1 --1- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6793/0 conn(0x7f280c0a6830 0x7f280c0a6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6793/0 says I am v1:172.21.15.1:35058/0 (socket says 172.21.15.1:35058) 2024-08-19T10:09:48.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f28027fc640 1 -- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6792/0 conn(0x7f280c0694b0 legacy=0x7f280c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280affd640 1 -- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6793/0 conn(0x7f280c0a6830 legacy=0x7f280c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280b7fe640 1 -- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6791/0 conn(0x7f280c0a46a0 legacy=0x7f280c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2665264021 0 0) 0x7f280c0a33d0 con 0x7f280c0694b0 2024-08-19T10:09:48.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): my global_id is 204630 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27f8003520 con 0x7f280c0694b0 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.8 v1:172.21.15.1:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021949809 0 0) 0x7f280c0a36b0 con 0x7f280c0a6830 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): my global_id is 204794 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 --> v1:172.21.15.1:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f280c0a33d0 con 0x7f280c0a6830 2024-08-19T10:09:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.4 v1:172.21.15.1:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104169324 0 0) 0x7f280c0a3c70 con 0x7f280c0a46a0 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient(hunting): my global_id is 194980 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 --> v1:172.21.15.1:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f280c0a36b0 con 0x7f280c0a46a0 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2907234757 0 0) 0x7f27f8003520 con 0x7f280c0694b0 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6793/0 conn(0x7f280c0a6830 legacy=0x7f280c0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6791/0 conn(0x7f280c0a46a0 legacy=0x7f280c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient: _finish_hunting 0 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 monclient: found mon.noname-d 2024-08-19T10:09:48.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c06dbb0 con 0x7f280c0694b0 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient: _finish_auth 0 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient: _check_auth_tickets 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f27f4001ae0 con 0x7f280c0694b0 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.057+0000 7f280a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 10 monclient: dump: 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.068 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 5 monclient: authenticate success, global_id 204630 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27f40048a0 con 0x7f280c0694b0 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 1 -- 172.21.15.1:0/1336317634 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f27f4004e10 con 0x7f280c0694b0 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 4 set_mon_vals no callback set 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f280a7fc640 10 monclient: dump: 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 10 monclient: shutdown 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 1 -- 172.21.15.1:0/1336317634 >> v1:172.21.15.1:6792/0 conn(0x7f280c0694b0 legacy=0x7f280c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 1 -- 172.21.15.1:0/1336317634 shutdown_connections 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 1 -- 172.21.15.1:0/1336317634 >> 172.21.15.1:0/1336317634 conn(0x7f280c0a9020 msgr2=0x7f280c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 1 -- 172.21.15.1:0/1336317634 shutdown_connections 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.058+0000 7f281198a640 1 -- 172.21.15.1:0/1336317634 wait complete. 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 10 monclient: build_initial_monmap 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 10 monclient: monmap: 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 1 librados: starting msgr at 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 1 librados: starting objecter 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 1 Processor -- start 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 1 -- start start 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 1 librados: setting wanted keys 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 1 librados: calling monclient init 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.059+0000 7f281198a640 10 monclient: init 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient: _add_conns ranks=[7,0,5,4,1,2,3,6,8] 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient(hunting): picked mon.noname-i con 0x7f280c0694b0 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient(hunting): picked mon.noname-a con 0x7f280c0a46a0 addr v1:172.21.15.1:6789/0 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient(hunting): picked mon.noname-h con 0x7f280c0a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c138d20 con 0x7f280c0a46a0 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c137eb0 con 0x7f280c0a6830 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c135f70 con 0x7f280c0694b0 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f281198a640 10 monclient(hunting): authenticate will time out at 1844.136399s 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f28027fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f280c0a46a0 0x7f280c135860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35878/0 (socket says 172.21.15.1:35878) 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.060+0000 7f28027fc640 1 -- 172.21.15.1:0/3517297255 learned_addr learned my addr 172.21.15.1:0/3517297255 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.062+0000 7f28027fc640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.1:6789/0 conn(0x7f280c0a46a0 legacy=0x7f280c135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.062+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 DispatchQueue::entry inject delay of 0.002000 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.062+0000 7f280affd640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.82:6792/0 conn(0x7f280c0694b0 legacy=0x7f280c135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.062+0000 7f280b7fe640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.82:6791/0 conn(0x7f280c0a6830 legacy=0x7f280c1d6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 client.?.objecter ms_handle_connect 0x7f280c0a46a0 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 client.?.objecter ms_handle_connect 0x7f280c0694b0 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 client.?.objecter ms_handle_connect 0x7f280c0a6830 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583485929 0 0) 0x7f280c138d20 con 0x7f280c0a46a0 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): my global_id is 204741 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27ec004eb0 con 0x7f280c0a46a0 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2733015856 0 0) 0x7f280c137eb0 con 0x7f280c0a6830 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): my global_id is 214592 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f280c138d20 con 0x7f280c0a6830 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.7 v1:172.21.15.82:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119923546 0 0) 0x7f280c135f70 con 0x7f280c0694b0 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 10 monclient(hunting): my global_id is 204757 2024-08-19T10:09:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.064+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.82:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f280c137eb0 con 0x7f280c0694b0 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 12488771 0 0) 0x7f27ec004eb0 con 0x7f280c0a46a0 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f280c135f70 con 0x7f280c0a46a0 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f27f4004d90 con 0x7f280c0a46a0 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient(hunting): dump: 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3182498731 0 0) 0x7f280c135f70 con 0x7f280c0a46a0 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.82:6792/0 conn(0x7f280c0694b0 legacy=0x7f280c135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.82:6791/0 conn(0x7f280c0a6830 legacy=0x7f280c1d6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: _finish_hunting 0 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 monclient: found mon.a 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c138480 con 0x7f280c0a46a0 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: _finish_auth 0 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: _check_auth_tickets 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 5 monclient: authenticate success, global_id 204741 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 10 monclient: _renew_subs 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f280c136a00 con 0x7f280c0a46a0 2024-08-19T10:09:48.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 10 client.204741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 10 monclient: _renew_subs 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f280c137790 con 0x7f280c0a46a0 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f281198a640 1 librados: init done 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27f4001d30 con 0x7f280c0a46a0 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7f27f40048a0 con 0x7f280c0a46a0 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2809ffb640 4 set_mon_vals no callback set 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.065+0000 7f2808ff9640 10 monclient: dump: 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.083 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.084 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f281198a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f281198a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f281198a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7f27f4011a00 con 0x7f280c0a46a0 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f0005080 con 0x7f280c0a46a0 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7f27f404df70 con 0x7f280c0a46a0 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f2808ff9640 10 client.204741.objecter ms_dispatch 0x7f280c000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f2808ff9640 3 client.204741.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f2808ff9640 3 client.204741.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f2808ff9640 20 client.204741.objecter dump_active .. 0 homeless 2024-08-19T10:09:48.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.066+0000 7f280affd640 10 monclient: get_auth_request con 0x7f27ec041400 auth_method 0 2024-08-19T10:09:48.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.069+0000 7f2808ff9640 10 client.204741.objecter ms_handle_connect 0x7f27ec041400 2024-08-19T10:09:48.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.071+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f27f4017570 con 0x7f280c0a46a0 2024-08-19T10:09:48.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.071+0000 7f2808ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:48.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.071+0000 7f2808ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:48.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447961552}) 2024-08-19T10:09:48.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.247 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946300824}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2746781560}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2084770730}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1428479834}) 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.248 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1725316914}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4124395065}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517818790}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.238+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3064118612}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591734257}) 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.249 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185285540}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711197924}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4165052489}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=455864481}) 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.250 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1709568669}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1611162460}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335327143}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.239+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3465567295}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3482511406}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.251 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722808216}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1609404546}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1127502033}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=733908905}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1209425133}) 2024-08-19T10:09:48.252 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1274029261}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017879630}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.240+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2045242550}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4062029224}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501294596}) 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.253 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3742729320}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2272476317}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3945700251}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=179982405}) 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.254 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=304015733}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170462711}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=41025832}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.241+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3524352433}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651762163}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.255 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=143575604}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3457966656}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1667878880}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3073173803}) 2024-08-19T10:09:48.256 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2388987976}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202515132}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2816833661}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1739163732}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3669604529}) 2024-08-19T10:09:48.257 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=36504003}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2065735666}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2003707986}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.242+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1071610507}) 2024-08-19T10:09:48.258 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1495698968}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413481855}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.243+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3990827323}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=2921430159,mdsmap=61829903,mgr=2098198440}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=88,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1} crc {logm=3884284410,mdsmap=3140514009,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.259 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=3990691648,mdsmap=61829903,mgr=3680630232}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=4097279553,mdsmap=61829903,mgr=157558894}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=3133936393,mdsmap=61829903,mgr=2098198440}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=555153196,mdsmap=61829903,mgr=3591800611}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.260 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=87,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1} crc {logm=3565045756,mdsmap=61829903,mgr=3591800611,mgr_command_descs=3626727126,mgr_metadata=3429325078}) 2024-08-19T10:09:48.261 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.261 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3470886159,mdsmap=61829903,mgr=22105279}) 2024-08-19T10:09:48.261 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.244+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=93,mdsmap=2,mgr=5} crc {logm=1198041900,mdsmap=3690439819,mgr=3680630232}) 2024-08-19T10:09:48.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.257+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 --> v1:172.21.15.82:6812/2549104415 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f27f0000b90 con 0x7f27ec041400 2024-08-19T10:09:48.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.259+0000 7f2808ff9640 1 -- 172.21.15.1:0/3517297255 <== mgr.4116 v1:172.21.15.82:6812/2549104415 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114118 (unknown 2965378022 0 40064931) 0x7f27f0000b90 con 0x7f27ec041400 2024-08-19T10:09:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-08-19T10:09:48.263 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 10 librados: watch_flush enter 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 10 librados: watch_flush exit 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 20 client.204741.objecter shutdown clearing up homeless session... 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 10 client.204741.objecter successfully canceled tick 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.82:6812/2549104415 conn(0x7f27ec041400 legacy=0x7f27ec0438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 10 monclient: shutdown 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 >> v1:172.21.15.1:6789/0 conn(0x7f280c0a46a0 legacy=0x7f280c135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 shutdown_connections 2024-08-19T10:09:48.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 >> 172.21.15.1:0/3517297255 conn(0x7f280c0a9020 msgr2=0x7f280c0a4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:48.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 shutdown_connections 2024-08-19T10:09:48.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 -- 172.21.15.1:0/3517297255 wait complete. 2024-08-19T10:09:48.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.262+0000 7f281198a640 1 librados: shutdown 2024-08-19T10:09:48.274 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1294,"stamp":"2024-08-19T10:09:47.066691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":442608,"kb_used_data":38928,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565788432,"statfs":{"total":579820584960,"available":579367354368,"internally_reserved":0,"allocated":39862272,"data_stored":17623178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":413260450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002816"},"pg_stats":[{"pgid":"1.0","version":"711'74","reported_seq":1507,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:26.689208+0000","last_change":"2024-08-19T10:09:26.689249+0000","last_active":"2024-08-19T10:09:26.689208+0000","last_peered":"2024-08-19T10:09:26.689208+0000","last_clean":"2024-08-19T10:09:26.689208+0000","last_became_active":"2024-08-19T09:50:08.953326+0000","last_became_peered":"2024-08-19T09:50:08.953326+0000","last_unstale":"2024-08-19T10:09:26.689208+0000","last_undegraded":"2024-08-19T10:09:26.689208+0000","last_fullsized":"2024-08-19T10:09:26.689208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'74","last_scrub_stamp":"2024-08-19T10:09:26.689120+0000","last_deep_scrub":"711'74","last_deep_scrub_stamp":"2024-08-19T10:09:26.689120+0000","last_clean_scrub_stamp":"2024-08-19T10:09:26.689120+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:21:01.385704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"711'2","reported_seq":1414,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:31.474929+0000","last_change":"2024-08-19T10:09:31.474973+0000","last_active":"2024-08-19T10:09:31.474929+0000","last_peered":"2024-08-19T10:09:31.474929+0000","last_clean":"2024-08-19T10:09:31.474929+0000","last_became_active":"2024-08-19T09:50:15.770248+0000","last_became_peered":"2024-08-19T09:50:15.770248+0000","last_unstale":"2024-08-19T10:09:31.474929+0000","last_undegraded":"2024-08-19T10:09:31.474929+0000","last_fullsized":"2024-08-19T10:09:31.474929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2","last_scrub_stamp":"2024-08-19T10:09:31.474843+0000","last_deep_scrub":"711'2","last_deep_scrub_stamp":"2024-08-19T10:09:31.474843+0000","last_clean_scrub_stamp":"2024-08-19T10:09:31.474843+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:49.232843+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.001043343,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1414,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:27.657752+0000","last_change":"2024-08-19T10:09:27.657781+0000","last_active":"2024-08-19T10:09:27.657752+0000","last_peered":"2024-08-19T10:09:27.657752+0000","last_clean":"2024-08-19T10:09:27.657752+0000","last_became_active":"2024-08-19T09:50:15.773764+0000","last_became_peered":"2024-08-19T09:50:15.773764+0000","last_unstale":"2024-08-19T10:09:27.657752+0000","last_undegraded":"2024-08-19T10:09:27.657752+0000","last_fullsized":"2024-08-19T10:09:27.657752+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:27.657630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:27.657630+0000","last_clean_scrub_stamp":"2024-08-19T10:09:27.657630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:23:21.391659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054526499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1412,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:28.012545+0000","last_change":"2024-08-19T10:09:28.012567+0000","last_active":"2024-08-19T10:09:28.012545+0000","last_peered":"2024-08-19T10:09:28.012545+0000","last_clean":"2024-08-19T10:09:28.012545+0000","last_became_active":"2024-08-19T09:50:15.765225+0000","last_became_peered":"2024-08-19T09:50:15.765225+0000","last_unstale":"2024-08-19T10:09:28.012545+0000","last_undegraded":"2024-08-19T10:09:28.012545+0000","last_fullsized":"2024-08-19T10:09:28.012545+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:28.012481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:28.012481+0000","last_clean_scrub_stamp":"2024-08-19T10:09:28.012481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:39:24.844178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097040699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"711'3","reported_seq":1415,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:28.395852+0000","last_change":"2024-08-19T10:09:28.395902+0000","last_active":"2024-08-19T10:09:28.395852+0000","last_peered":"2024-08-19T10:09:28.395852+0000","last_clean":"2024-08-19T10:09:28.395852+0000","last_became_active":"2024-08-19T09:50:15.767912+0000","last_became_peered":"2024-08-19T09:50:15.767912+0000","last_unstale":"2024-08-19T10:09:28.395852+0000","last_undegraded":"2024-08-19T10:09:28.395852+0000","last_fullsized":"2024-08-19T10:09:28.395852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'3","last_scrub_stamp":"2024-08-19T10:09:28.395762+0000","last_deep_scrub":"711'3","last_deep_scrub_stamp":"2024-08-19T10:09:28.395762+0000","last_clean_scrub_stamp":"2024-08-19T10:09:28.395762+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:25:26.248551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071252000000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1407,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:25.012786+0000","last_change":"2024-08-19T10:09:25.012809+0000","last_active":"2024-08-19T10:09:25.012786+0000","last_peered":"2024-08-19T10:09:25.012786+0000","last_clean":"2024-08-19T10:09:25.012786+0000","last_became_active":"2024-08-19T09:50:15.765348+0000","last_became_peered":"2024-08-19T09:50:15.765348+0000","last_unstale":"2024-08-19T10:09:25.012786+0000","last_undegraded":"2024-08-19T10:09:25.012786+0000","last_fullsized":"2024-08-19T10:09:25.012786+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:25.012721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:25.012721+0000","last_clean_scrub_stamp":"2024-08-19T10:09:25.012721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:20:07.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099407699999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1414,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:28.622048+0000","last_change":"2024-08-19T10:09:28.622075+0000","last_active":"2024-08-19T10:09:28.622048+0000","last_peered":"2024-08-19T10:09:28.622048+0000","last_clean":"2024-08-19T10:09:28.622048+0000","last_became_active":"2024-08-19T09:50:15.770088+0000","last_became_peered":"2024-08-19T09:50:15.770088+0000","last_unstale":"2024-08-19T10:09:28.622048+0000","last_undegraded":"2024-08-19T10:09:28.622048+0000","last_fullsized":"2024-08-19T10:09:28.622048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:28.621918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:28.621918+0000","last_clean_scrub_stamp":"2024-08-19T10:09:28.621918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:06:52.774750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085938500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1407,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:25.976179+0000","last_change":"2024-08-19T10:09:25.976179+0000","last_active":"2024-08-19T10:09:25.976179+0000","last_peered":"2024-08-19T10:09:25.976179+0000","last_clean":"2024-08-19T10:09:25.976179+0000","last_became_active":"2024-08-19T09:50:15.773474+0000","last_became_peered":"2024-08-19T09:50:15.773474+0000","last_unstale":"2024-08-19T10:09:25.976179+0000","last_undegraded":"2024-08-19T10:09:25.976179+0000","last_fullsized":"2024-08-19T10:09:25.976179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:25.976052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:25.976052+0000","last_clean_scrub_stamp":"2024-08-19T10:09:25.976052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:53:02.356791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000868657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1413,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:30.610469+0000","last_change":"2024-08-19T10:09:30.610495+0000","last_active":"2024-08-19T10:09:30.610469+0000","last_peered":"2024-08-19T10:09:30.610469+0000","last_clean":"2024-08-19T10:09:30.610469+0000","last_became_active":"2024-08-19T09:50:15.773636+0000","last_became_peered":"2024-08-19T09:50:15.773636+0000","last_unstale":"2024-08-19T10:09:30.610469+0000","last_undegraded":"2024-08-19T10:09:30.610469+0000","last_fullsized":"2024-08-19T10:09:30.610469+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:30.610378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:30.610378+0000","last_clean_scrub_stamp":"2024-08-19T10:09:30.610378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:55:38.394037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083964900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1","version":"711'8","reported_seq":1410,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:29.603171+0000","last_change":"2024-08-19T10:09:29.603198+0000","last_active":"2024-08-19T10:09:29.603171+0000","last_peered":"2024-08-19T10:09:29.603171+0000","last_clean":"2024-08-19T10:09:29.603171+0000","last_became_active":"2024-08-19T09:51:22.585789+0000","last_became_peered":"2024-08-19T09:51:22.585789+0000","last_unstale":"2024-08-19T10:09:29.603171+0000","last_undegraded":"2024-08-19T10:09:29.603171+0000","last_fullsized":"2024-08-19T10:09:29.603171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'8","last_scrub_stamp":"2024-08-19T10:09:29.603093+0000","last_deep_scrub":"711'8","last_deep_scrub_stamp":"2024-08-19T10:09:29.603093+0000","last_clean_scrub_stamp":"2024-08-19T10:09:29.603093+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:49:29.252234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1395,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:29.046007+0000","last_change":"2024-08-19T10:09:29.046051+0000","last_active":"2024-08-19T10:09:29.046007+0000","last_peered":"2024-08-19T10:09:29.046007+0000","last_clean":"2024-08-19T10:09:29.046007+0000","last_became_active":"2024-08-19T09:51:22.583401+0000","last_became_peered":"2024-08-19T09:51:22.583401+0000","last_unstale":"2024-08-19T10:09:29.046007+0000","last_undegraded":"2024-08-19T10:09:29.046007+0000","last_fullsized":"2024-08-19T10:09:29.046007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:29.045906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:29.045906+0000","last_clean_scrub_stamp":"2024-08-19T10:09:29.045906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:44:30.535717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:30.011492+0000","last_change":"2024-08-19T10:09:30.011516+0000","last_active":"2024-08-19T10:09:30.011492+0000","last_peered":"2024-08-19T10:09:30.011492+0000","last_clean":"2024-08-19T10:09:30.011492+0000","last_became_active":"2024-08-19T09:51:22.587936+0000","last_became_peered":"2024-08-19T09:51:22.587936+0000","last_unstale":"2024-08-19T10:09:30.011492+0000","last_undegraded":"2024-08-19T10:09:30.011492+0000","last_fullsized":"2024-08-19T10:09:30.011492+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:30.011406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:30.011406+0000","last_clean_scrub_stamp":"2024-08-19T10:09:30.011406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:13:17.805718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:26.969185+0000","last_change":"2024-08-19T10:09:26.969205+0000","last_active":"2024-08-19T10:09:26.969185+0000","last_peered":"2024-08-19T10:09:26.969185+0000","last_clean":"2024-08-19T10:09:26.969185+0000","last_became_active":"2024-08-19T09:51:22.640604+0000","last_became_peered":"2024-08-19T09:51:22.640604+0000","last_unstale":"2024-08-19T10:09:26.969185+0000","last_undegraded":"2024-08-19T10:09:26.969185+0000","last_fullsized":"2024-08-19T10:09:26.969185+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:26.969120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:26.969120+0000","last_clean_scrub_stamp":"2024-08-19T10:09:26.969120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:37:07.992489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1396,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:31.645219+0000","last_change":"2024-08-19T10:09:31.645245+0000","last_active":"2024-08-19T10:09:31.645219+0000","last_peered":"2024-08-19T10:09:31.645219+0000","last_clean":"2024-08-19T10:09:31.645219+0000","last_became_active":"2024-08-19T09:51:22.581246+0000","last_became_peered":"2024-08-19T09:51:22.581246+0000","last_unstale":"2024-08-19T10:09:31.645219+0000","last_undegraded":"2024-08-19T10:09:31.645219+0000","last_fullsized":"2024-08-19T10:09:31.645219+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:31.645134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:31.645134+0000","last_clean_scrub_stamp":"2024-08-19T10:09:31.645134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:38:55.682734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:30.463205+0000","last_change":"2024-08-19T10:09:30.463234+0000","last_active":"2024-08-19T10:09:30.463205+0000","last_peered":"2024-08-19T10:09:30.463205+0000","last_clean":"2024-08-19T10:09:30.463205+0000","last_became_active":"2024-08-19T09:51:22.620042+0000","last_became_peered":"2024-08-19T09:51:22.620042+0000","last_unstale":"2024-08-19T10:09:30.463205+0000","last_undegraded":"2024-08-19T10:09:30.463205+0000","last_fullsized":"2024-08-19T10:09:30.463205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:30.463112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:30.463112+0000","last_clean_scrub_stamp":"2024-08-19T10:09:30.463112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:33:41.080112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1393,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:32.018932+0000","last_change":"2024-08-19T10:09:32.018993+0000","last_active":"2024-08-19T10:09:32.018932+0000","last_peered":"2024-08-19T10:09:32.018932+0000","last_clean":"2024-08-19T10:09:32.018932+0000","last_became_active":"2024-08-19T09:51:22.606054+0000","last_became_peered":"2024-08-19T09:51:22.606054+0000","last_unstale":"2024-08-19T10:09:32.018932+0000","last_undegraded":"2024-08-19T10:09:32.018932+0000","last_fullsized":"2024-08-19T10:09:32.018932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:32.018818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:32.018818+0000","last_clean_scrub_stamp":"2024-08-19T10:09:32.018818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:01:17.027348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:31.003037+0000","last_change":"2024-08-19T10:09:31.003087+0000","last_active":"2024-08-19T10:09:31.003037+0000","last_peered":"2024-08-19T10:09:31.003037+0000","last_clean":"2024-08-19T10:09:31.003037+0000","last_became_active":"2024-08-19T09:51:22.653455+0000","last_became_peered":"2024-08-19T09:51:22.653455+0000","last_unstale":"2024-08-19T10:09:31.003037+0000","last_undegraded":"2024-08-19T10:09:31.003037+0000","last_fullsized":"2024-08-19T10:09:31.003037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:31.002922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:31.002922+0000","last_clean_scrub_stamp":"2024-08-19T10:09:31.002922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:55:32.600212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1387,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:24.840543+0000","last_change":"2024-08-19T10:09:24.840586+0000","last_active":"2024-08-19T10:09:24.840543+0000","last_peered":"2024-08-19T10:09:24.840543+0000","last_clean":"2024-08-19T10:09:24.840543+0000","last_became_active":"2024-08-19T09:51:22.553419+0000","last_became_peered":"2024-08-19T09:51:22.553419+0000","last_unstale":"2024-08-19T10:09:24.840543+0000","last_undegraded":"2024-08-19T10:09:24.840543+0000","last_fullsized":"2024-08-19T10:09:24.840543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:24.840458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:24.840458+0000","last_clean_scrub_stamp":"2024-08-19T10:09:24.840458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:58:08.064445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:35.020654+0000","last_change":"2024-08-19T10:09:35.020683+0000","last_active":"2024-08-19T10:09:35.020654+0000","last_peered":"2024-08-19T10:09:35.020654+0000","last_clean":"2024-08-19T10:09:35.020654+0000","last_became_active":"2024-08-19T09:51:22.554397+0000","last_became_peered":"2024-08-19T09:51:22.554397+0000","last_unstale":"2024-08-19T10:09:35.020654+0000","last_undegraded":"2024-08-19T10:09:35.020654+0000","last_fullsized":"2024-08-19T10:09:35.020654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:35.020560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:35.020560+0000","last_clean_scrub_stamp":"2024-08-19T10:09:35.020560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:28:33.262755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1395,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:27.964032+0000","last_change":"2024-08-19T10:09:27.964053+0000","last_active":"2024-08-19T10:09:27.964032+0000","last_peered":"2024-08-19T10:09:27.964032+0000","last_clean":"2024-08-19T10:09:27.964032+0000","last_became_active":"2024-08-19T09:55:22.703438+0000","last_became_peered":"2024-08-19T09:55:22.703438+0000","last_unstale":"2024-08-19T10:09:27.964032+0000","last_undegraded":"2024-08-19T10:09:27.964032+0000","last_fullsized":"2024-08-19T10:09:27.964032+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:27.963953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:27.963953+0000","last_clean_scrub_stamp":"2024-08-19T10:09:27.963953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:09:21.954853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1389,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:26.972458+0000","last_change":"2024-08-19T10:09:26.972485+0000","last_active":"2024-08-19T10:09:26.972458+0000","last_peered":"2024-08-19T10:09:26.972458+0000","last_clean":"2024-08-19T10:09:26.972458+0000","last_became_active":"2024-08-19T09:51:22.528219+0000","last_became_peered":"2024-08-19T09:51:22.528219+0000","last_unstale":"2024-08-19T10:09:26.972458+0000","last_undegraded":"2024-08-19T10:09:26.972458+0000","last_fullsized":"2024-08-19T10:09:26.972458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:26.972397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:26.972397+0000","last_clean_scrub_stamp":"2024-08-19T10:09:26.972397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:56:27.701977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:29.441468+0000","last_change":"2024-08-19T10:09:29.441489+0000","last_active":"2024-08-19T10:09:29.441468+0000","last_peered":"2024-08-19T10:09:29.441468+0000","last_clean":"2024-08-19T10:09:29.441468+0000","last_became_active":"2024-08-19T09:51:22.614136+0000","last_became_peered":"2024-08-19T09:51:22.614136+0000","last_unstale":"2024-08-19T10:09:29.441468+0000","last_undegraded":"2024-08-19T10:09:29.441468+0000","last_fullsized":"2024-08-19T10:09:29.441468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:29.441399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:29.441399+0000","last_clean_scrub_stamp":"2024-08-19T10:09:29.441399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:55:24.769490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:34.029271+0000","last_change":"2024-08-19T10:09:34.029298+0000","last_active":"2024-08-19T10:09:34.029271+0000","last_peered":"2024-08-19T10:09:34.029271+0000","last_clean":"2024-08-19T10:09:34.029271+0000","last_became_active":"2024-08-19T09:51:22.587149+0000","last_became_peered":"2024-08-19T09:51:22.587149+0000","last_unstale":"2024-08-19T10:09:34.029271+0000","last_undegraded":"2024-08-19T10:09:34.029271+0000","last_fullsized":"2024-08-19T10:09:34.029271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:34.029172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:34.029172+0000","last_clean_scrub_stamp":"2024-08-19T10:09:34.029172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:35:39.480633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1386,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:28.034244+0000","last_change":"2024-08-19T10:09:28.034280+0000","last_active":"2024-08-19T10:09:28.034244+0000","last_peered":"2024-08-19T10:09:28.034244+0000","last_clean":"2024-08-19T10:09:28.034244+0000","last_became_active":"2024-08-19T09:51:22.634477+0000","last_became_peered":"2024-08-19T09:51:22.634477+0000","last_unstale":"2024-08-19T10:09:28.034244+0000","last_undegraded":"2024-08-19T10:09:28.034244+0000","last_fullsized":"2024-08-19T10:09:28.034244+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:28.034151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:28.034151+0000","last_clean_scrub_stamp":"2024-08-19T10:09:28.034151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:16:37.520709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1394,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:32.989490+0000","last_change":"2024-08-19T10:09:32.989538+0000","last_active":"2024-08-19T10:09:32.989490+0000","last_peered":"2024-08-19T10:09:32.989490+0000","last_clean":"2024-08-19T10:09:32.989490+0000","last_became_active":"2024-08-19T09:51:22.590688+0000","last_became_peered":"2024-08-19T09:51:22.590688+0000","last_unstale":"2024-08-19T10:09:32.989490+0000","last_undegraded":"2024-08-19T10:09:32.989490+0000","last_fullsized":"2024-08-19T10:09:32.989490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:32.989386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:32.989386+0000","last_clean_scrub_stamp":"2024-08-19T10:09:32.989386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:09:22.329895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:29.013786+0000","last_change":"2024-08-19T10:09:29.013812+0000","last_active":"2024-08-19T10:09:29.013786+0000","last_peered":"2024-08-19T10:09:29.013786+0000","last_clean":"2024-08-19T10:09:29.013786+0000","last_became_active":"2024-08-19T09:51:22.536584+0000","last_became_peered":"2024-08-19T09:51:22.536584+0000","last_unstale":"2024-08-19T10:09:29.013786+0000","last_undegraded":"2024-08-19T10:09:29.013786+0000","last_fullsized":"2024-08-19T10:09:29.013786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:29.013704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:29.013704+0000","last_clean_scrub_stamp":"2024-08-19T10:09:29.013704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:41:05.325735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1386,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:25.860136+0000","last_change":"2024-08-19T10:09:25.860176+0000","last_active":"2024-08-19T10:09:25.860136+0000","last_peered":"2024-08-19T10:09:25.860136+0000","last_clean":"2024-08-19T10:09:25.860136+0000","last_became_active":"2024-08-19T09:51:22.556182+0000","last_became_peered":"2024-08-19T09:51:22.556182+0000","last_unstale":"2024-08-19T10:09:25.860136+0000","last_undegraded":"2024-08-19T10:09:25.860136+0000","last_fullsized":"2024-08-19T10:09:25.860136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:25.859995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:25.859995+0000","last_clean_scrub_stamp":"2024-08-19T10:09:25.859995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:37:27.472589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:33.023106+0000","last_change":"2024-08-19T10:09:33.023133+0000","last_active":"2024-08-19T10:09:33.023106+0000","last_peered":"2024-08-19T10:09:33.023106+0000","last_clean":"2024-08-19T10:09:33.023106+0000","last_became_active":"2024-08-19T09:51:22.671918+0000","last_became_peered":"2024-08-19T09:51:22.671918+0000","last_unstale":"2024-08-19T10:09:33.023106+0000","last_undegraded":"2024-08-19T10:09:33.023106+0000","last_fullsized":"2024-08-19T10:09:33.023106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:33.023026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:33.023026+0000","last_clean_scrub_stamp":"2024-08-19T10:09:33.023026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:18:43.577900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:32.042911+0000","last_change":"2024-08-19T10:09:32.042940+0000","last_active":"2024-08-19T10:09:32.042911+0000","last_peered":"2024-08-19T10:09:32.042911+0000","last_clean":"2024-08-19T10:09:32.042911+0000","last_became_active":"2024-08-19T09:51:22.605586+0000","last_became_peered":"2024-08-19T09:51:22.605586+0000","last_unstale":"2024-08-19T10:09:32.042911+0000","last_undegraded":"2024-08-19T10:09:32.042911+0000","last_fullsized":"2024-08-19T10:09:32.042911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:32.042813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:32.042813+0000","last_clean_scrub_stamp":"2024-08-19T10:09:32.042813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:23:10.520747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:30.994170+0000","last_change":"2024-08-19T10:09:30.994229+0000","last_active":"2024-08-19T10:09:30.994170+0000","last_peered":"2024-08-19T10:09:30.994170+0000","last_clean":"2024-08-19T10:09:30.994170+0000","last_became_active":"2024-08-19T09:51:22.551119+0000","last_became_peered":"2024-08-19T09:51:22.551119+0000","last_unstale":"2024-08-19T10:09:30.994170+0000","last_undegraded":"2024-08-19T10:09:30.994170+0000","last_fullsized":"2024-08-19T10:09:30.994170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:30.994052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:30.994052+0000","last_clean_scrub_stamp":"2024-08-19T10:09:30.994052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:56:21.822652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1393,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:36.044980+0000","last_change":"2024-08-19T10:09:36.045007+0000","last_active":"2024-08-19T10:09:36.044980+0000","last_peered":"2024-08-19T10:09:36.044980+0000","last_clean":"2024-08-19T10:09:36.044980+0000","last_became_active":"2024-08-19T09:51:22.630612+0000","last_became_peered":"2024-08-19T09:51:22.630612+0000","last_unstale":"2024-08-19T10:09:36.044980+0000","last_undegraded":"2024-08-19T10:09:36.044980+0000","last_fullsized":"2024-08-19T10:09:36.044980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:36.044883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:36.044883+0000","last_clean_scrub_stamp":"2024-08-19T10:09:36.044883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:15:54.150073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1386,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:26.853687+0000","last_change":"2024-08-19T10:09:26.853708+0000","last_active":"2024-08-19T10:09:26.853687+0000","last_peered":"2024-08-19T10:09:26.853687+0000","last_clean":"2024-08-19T10:09:26.853687+0000","last_became_active":"2024-08-19T09:51:22.490018+0000","last_became_peered":"2024-08-19T09:51:22.490018+0000","last_unstale":"2024-08-19T10:09:26.853687+0000","last_undegraded":"2024-08-19T10:09:26.853687+0000","last_fullsized":"2024-08-19T10:09:26.853687+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:26.853617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:26.853617+0000","last_clean_scrub_stamp":"2024-08-19T10:09:26.853617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:48:38.499389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:29.014148+0000","last_change":"2024-08-19T10:09:29.014176+0000","last_active":"2024-08-19T10:09:29.014148+0000","last_peered":"2024-08-19T10:09:29.014148+0000","last_clean":"2024-08-19T10:09:29.014148+0000","last_became_active":"2024-08-19T09:51:22.644785+0000","last_became_peered":"2024-08-19T09:51:22.644785+0000","last_unstale":"2024-08-19T10:09:29.014148+0000","last_undegraded":"2024-08-19T10:09:29.014148+0000","last_fullsized":"2024-08-19T10:09:29.014148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:29.014070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:29.014070+0000","last_clean_scrub_stamp":"2024-08-19T10:09:29.014070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:37:08.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:30.033730+0000","last_change":"2024-08-19T10:09:30.033754+0000","last_active":"2024-08-19T10:09:30.033730+0000","last_peered":"2024-08-19T10:09:30.033730+0000","last_clean":"2024-08-19T10:09:30.033730+0000","last_became_active":"2024-08-19T09:51:22.618248+0000","last_became_peered":"2024-08-19T09:51:22.618248+0000","last_unstale":"2024-08-19T10:09:30.033730+0000","last_undegraded":"2024-08-19T10:09:30.033730+0000","last_fullsized":"2024-08-19T10:09:30.033730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:30.033648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:30.033648+0000","last_clean_scrub_stamp":"2024-08-19T10:09:30.033648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:43:55.961174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:29.988268+0000","last_change":"2024-08-19T10:09:29.988304+0000","last_active":"2024-08-19T10:09:29.988268+0000","last_peered":"2024-08-19T10:09:29.988268+0000","last_clean":"2024-08-19T10:09:29.988268+0000","last_became_active":"2024-08-19T09:51:22.558815+0000","last_became_peered":"2024-08-19T09:51:22.558815+0000","last_unstale":"2024-08-19T10:09:29.988268+0000","last_undegraded":"2024-08-19T10:09:29.988268+0000","last_fullsized":"2024-08-19T10:09:29.988268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:29.988167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:29.988167+0000","last_clean_scrub_stamp":"2024-08-19T10:09:29.988167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:51:52.971487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1388,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:34.056561+0000","last_change":"2024-08-19T10:09:34.056592+0000","last_active":"2024-08-19T10:09:34.056561+0000","last_peered":"2024-08-19T10:09:34.056561+0000","last_clean":"2024-08-19T10:09:34.056561+0000","last_became_active":"2024-08-19T09:51:22.581840+0000","last_became_peered":"2024-08-19T09:51:22.581840+0000","last_unstale":"2024-08-19T10:09:34.056561+0000","last_undegraded":"2024-08-19T10:09:34.056561+0000","last_fullsized":"2024-08-19T10:09:34.056561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:34.056466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:34.056466+0000","last_clean_scrub_stamp":"2024-08-19T10:09:34.056466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:35:31.534612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1392,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:27.881756+0000","last_change":"2024-08-19T10:09:27.881797+0000","last_active":"2024-08-19T10:09:27.881756+0000","last_peered":"2024-08-19T10:09:27.881756+0000","last_clean":"2024-08-19T10:09:27.881756+0000","last_became_active":"2024-08-19T09:55:22.701660+0000","last_became_peered":"2024-08-19T09:55:22.701660+0000","last_unstale":"2024-08-19T10:09:27.881756+0000","last_undegraded":"2024-08-19T10:09:27.881756+0000","last_fullsized":"2024-08-19T10:09:27.881756+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:27.881648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:27.881648+0000","last_clean_scrub_stamp":"2024-08-19T10:09:27.881648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:33:35.050150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1385,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:31.976099+0000","last_change":"2024-08-19T10:09:31.976148+0000","last_active":"2024-08-19T10:09:31.976099+0000","last_peered":"2024-08-19T10:09:31.976099+0000","last_clean":"2024-08-19T10:09:31.976099+0000","last_became_active":"2024-08-19T09:51:22.564919+0000","last_became_peered":"2024-08-19T09:51:22.564919+0000","last_unstale":"2024-08-19T10:09:31.976099+0000","last_undegraded":"2024-08-19T10:09:31.976099+0000","last_fullsized":"2024-08-19T10:09:31.976099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:31.975945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:31.975945+0000","last_clean_scrub_stamp":"2024-08-19T10:09:31.975945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:23:24.674489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1393,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:32.496994+0000","last_change":"2024-08-19T10:09:32.497057+0000","last_active":"2024-08-19T10:09:32.496994+0000","last_peered":"2024-08-19T10:09:32.496994+0000","last_clean":"2024-08-19T10:09:32.496994+0000","last_became_active":"2024-08-19T09:51:22.612552+0000","last_became_peered":"2024-08-19T09:51:22.612552+0000","last_unstale":"2024-08-19T10:09:32.496994+0000","last_undegraded":"2024-08-19T10:09:32.496994+0000","last_fullsized":"2024-08-19T10:09:32.496994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:32.496897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:32.496897+0000","last_clean_scrub_stamp":"2024-08-19T10:09:32.496897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:51.763539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1393,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:37.060329+0000","last_change":"2024-08-19T10:09:37.060352+0000","last_active":"2024-08-19T10:09:37.060329+0000","last_peered":"2024-08-19T10:09:37.060329+0000","last_clean":"2024-08-19T10:09:37.060329+0000","last_became_active":"2024-08-19T09:51:22.609674+0000","last_became_peered":"2024-08-19T09:51:22.609674+0000","last_unstale":"2024-08-19T10:09:37.060329+0000","last_undegraded":"2024-08-19T10:09:37.060329+0000","last_fullsized":"2024-08-19T10:09:37.060329+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:37.060245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:37.060245+0000","last_clean_scrub_stamp":"2024-08-19T10:09:37.060245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:12:43.015108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1387,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-08-19T10:09:31.004516+0000","last_change":"2024-08-19T10:09:31.004550+0000","last_active":"2024-08-19T10:09:31.004516+0000","last_peered":"2024-08-19T10:09:31.004516+0000","last_clean":"2024-08-19T10:09:31.004516+0000","last_became_active":"2024-08-19T09:51:22.652083+0000","last_became_peered":"2024-08-19T09:51:22.652083+0000","last_unstale":"2024-08-19T10:09:31.004516+0000","last_undegraded":"2024-08-19T10:09:31.004516+0000","last_fullsized":"2024-08-19T10:09:31.004516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T10:09:31.004421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T10:09:31.004421+0000","last_clean_scrub_stamp":"2024-08-19T10:09:31.004421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:51:02.736738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575091,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73592,"kb_used_data":6312,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298248,"statfs":{"total":96636764160,"available":96561405952,"internally_reserved":0,"allocated":6463488,"data_stored":2760879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575087,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74116,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297724,"statfs":{"total":96636764160,"available":96560869376,"internally_reserved":0,"allocated":7000064,"data_stored":3289812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607791,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69436,"kb_used_data":6316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302404,"statfs":{"total":96636764160,"available":96565661696,"internally_reserved":0,"allocated":6467584,"data_stored":2760898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607790,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81916,"kb_used_data":6316,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289924,"statfs":{"total":96636764160,"available":96552882176,"internally_reserved":0,"allocated":6467584,"data_stored":2760884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607792,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69952,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301888,"statfs":{"total":96636764160,"available":96565133312,"internally_reserved":0,"allocated":6995968,"data_stored":3289807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73596,"kb_used_data":6316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298244,"statfs":{"total":96636764160,"available":96561401856,"internally_reserved":0,"allocated":6467584,"data_stored":2760898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T10:09:48.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.411+0000 7ffbe0981640 10 monclient: get_monmap_and_config 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.411+0000 7ffbe0981640 10 monclient: build_initial_monmap 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.411+0000 7ffbe0981640 10 monclient: monmap: 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T10:09:48.412843+0000 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T10:09:48.412843+0000 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:48.413 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:48.414 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.412+0000 7ffbe0981640 1 Processor -- start 2024-08-19T10:09:48.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.412+0000 7ffbe0981640 1 -- start start 2024-08-19T10:09:48.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.412+0000 7ffbe0981640 10 monclient: init 2024-08-19T10:09:48.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:48.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient: _add_conns ranks=[7,6,5,0,3,8,1,2,4] 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient(hunting): picked mon.noname-i con 0x7ffbdc0a6830 addr v1:172.21.15.82:6792/0 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient(hunting): picked mon.noname-d con 0x7ffbdc0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient(hunting): picked mon.noname-h con 0x7ffbdc0a46a0 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbdc0a3c70 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbdc0a33d0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 1 -- --> v1:172.21.15.82:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbdc0a36b0 con 0x7ffbdc0a6830 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbe0981640 10 monclient(hunting): authenticate will time out at 1844.489191s 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbd9d9b640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ffbdc0694b0 0x7ffbdc0a7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45964/0 (socket says 172.21.15.1:45964) 2024-08-19T10:09:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.413+0000 7ffbd9d9b640 1 -- 172.21.15.1:0/768275549 learned_addr learned my addr 172.21.15.1:0/768275549 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd9d9b640 1 -- 172.21.15.1:0/768275549 >> v1:172.21.15.1:6792/0 conn(0x7ffbdc0694b0 legacy=0x7ffbdc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269236740 0 0) 0x7ffbdc0a33d0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient(hunting): my global_id is 204639 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbc8003520 con 0x7ffbdc0694b0 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbdad9d640 1 -- 172.21.15.1:0/768275549 >> v1:172.21.15.82:6791/0 conn(0x7ffbdc0a46a0 legacy=0x7ffbdc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbda59c640 1 -- 172.21.15.1:0/768275549 >> v1:172.21.15.82:6792/0 conn(0x7ffbdc0a6830 legacy=0x7ffbdc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1744993796 0 0) 0x7ffbc8003520 con 0x7ffbdc0694b0 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 >> v1:172.21.15.82:6792/0 conn(0x7ffbdc0a6830 legacy=0x7ffbdc0a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 >> v1:172.21.15.82:6791/0 conn(0x7ffbdc0a46a0 legacy=0x7ffbdc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: _finish_hunting 0 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 monclient: found mon.noname-d 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbdc06dbb0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: _finish_auth 0 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: _check_auth_tickets 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc4001ae0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbe0981640 5 monclient: authenticate success, global_id 204639 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbe0981640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 10 monclient: dump: 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbe0981640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.415+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 <== mon.6 v1:172.21.15.1:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffbc40048a0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbd959a640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 10 monclient: get_monmap_and_config success 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbd959a640 1 -- 172.21.15.1:0/768275549 <== mon.6 v1:172.21.15.1:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc4004e50 con 0x7ffbdc0694b0 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 4 set_mon_vals no callback set 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbd959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbd959a640 10 monclient: dump: 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.424 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 10 monclient: shutdown 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 1 -- 172.21.15.1:0/768275549 >> v1:172.21.15.1:6792/0 conn(0x7ffbdc0694b0 legacy=0x7ffbdc0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 1 -- 172.21.15.1:0/768275549 shutdown_connections 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 1 -- 172.21.15.1:0/768275549 >> 172.21.15.1:0/768275549 conn(0x7ffbdc0a9020 msgr2=0x7ffbdc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 1 -- 172.21.15.1:0/768275549 shutdown_connections 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.416+0000 7ffbe0981640 1 -- 172.21.15.1:0/768275549 wait complete. 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 10 monclient: build_initial_monmap 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 10 monclient: monmap: 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-08-19T10:09:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.noname-f 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.noname-g 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.noname-c 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.noname-h 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.noname-d 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.noname-i 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.noname-e 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 1 librados: starting msgr at 2024-08-19T10:09:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 1 librados: starting objecter 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 1 Processor -- start 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 1 -- start start 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 1 librados: setting wanted keys 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 1 librados: calling monclient init 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.417+0000 7ffbe0981640 10 monclient: init 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient: _reopen_session rank -1 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient: _add_conns ranks=[6,2,5,0,4,7,1,8,3] 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient(hunting): picked mon.noname-d con 0x7ffbdc0694b0 addr v1:172.21.15.1:6792/0 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient(hunting): picked mon.noname-b con 0x7ffbdc0a46a0 addr v1:172.21.15.1:6790/0 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient(hunting): picked mon.noname-h con 0x7ffbdc0a6830 addr v1:172.21.15.82:6791/0 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbdc138940 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 1 -- --> v1:172.21.15.82:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbdc1365e0 con 0x7ffbdc0a6830 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 1 -- --> v1:172.21.15.1:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbdc1380a0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient(hunting): _renew_subs 2024-08-19T10:09:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbe0981640 10 monclient(hunting): authenticate will time out at 1844.494113s 2024-08-19T10:09:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbda59c640 1 --1- >> v1:172.21.15.1:6792/0 conn(0x7ffbdc0694b0 0x7ffbdc1cb980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6792/0 says I am v1:172.21.15.1:45974/0 (socket says 172.21.15.1:45974) 2024-08-19T10:09:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.418+0000 7ffbda59c640 1 -- 172.21.15.1:0/2535443367 learned_addr learned my addr 172.21.15.1:0/2535443367 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-08-19T10:09:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd9d9b640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.1:6790/0 conn(0x7ffbdc0a46a0 legacy=0x7ffbdc141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbdc0a46a0 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbdc0694b0 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbda59c640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.1:6792/0 conn(0x7ffbdc0694b0 legacy=0x7ffbdc1cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbdc0a6830 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbdad9d640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.82:6791/0 conn(0x7ffbdc0a6830 legacy=0x7ffbdc1d5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790740211 0 0) 0x7ffbdc1380a0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): my global_id is 204648 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbbc004eb0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970238727 0 0) 0x7ffbdc138940 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): my global_id is 185150 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbdc1380a0 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2049093898 0 0) 0x7ffbbc004eb0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbdc138940 con 0x7ffbdc0694b0 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbcc002080 con 0x7ffbdc0694b0 2024-08-19T10:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.5 v1:172.21.15.82:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527631310 0 0) 0x7ffbdc1365e0 con 0x7ffbdc0a6830 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): my global_id is 214610 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.82:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbbc004eb0 con 0x7ffbdc0a6830 2024-08-19T10:09:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3564623175 0 0) 0x7ffbdc1380a0 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbdc1365e0 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbc4001d30 con 0x7ffbdc0a46a0 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient(hunting): dump: 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2665580077 0 0) 0x7ffbdc138940 con 0x7ffbdc0694b0 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.82:6791/0 conn(0x7ffbdc0a6830 legacy=0x7ffbdc1d5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.1:6790/0 conn(0x7ffbdc0a46a0 legacy=0x7ffbdc141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient: _finish_hunting 0 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 monclient: found mon.d 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T10:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbdc138380 con 0x7ffbdc0694b0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient: _finish_auth 0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 10 monclient: _check_auth_tickets 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.420+0000 7ffbd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 5 monclient: authenticate success, global_id 204648 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 monclient: _renew_subs 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffbdc1cc090 con 0x7ffbdc0694b0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 client.204648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 monclient: _renew_subs 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ffbdc142500 con 0x7ffbdc0694b0 2024-08-19T10:09:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 1 librados: init done 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffbcc003d70 con 0x7ffbdc0694b0 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 4076959990 0 0) 0x7ffbcc004260 con 0x7ffbdc0694b0 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd8d99640 4 set_mon_vals no callback set 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 10 monclient: dump: 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 0e9f827b-9129-41b7-89ad-24f744d49b77 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-08-19T09:49:28.615648+0000 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-08-19T10:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.82:6789/0 mon.f 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.b 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:3: v1:172.21.15.82:6790/0 mon.g 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:4: v1:172.21.15.1:6791/0 mon.c 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:5: v1:172.21.15.82:6791/0 mon.h 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:6: v1:172.21.15.1:6792/0 mon.d 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:7: v1:172.21.15.82:6792/0 mon.i 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:8: v1:172.21.15.1:6793/0 mon.e 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 975036883 0 0) 0x7ffbcc004400 con 0x7ffbdc0694b0 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.421+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc0005080 con 0x7ffbdc0694b0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.422+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 8 ==== osd_map(711..711 src has 299..711) ==== 4747+0+0 (unknown 3578140583 0 0) 0x7ffbcc04e250 con 0x7ffbdc0694b0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.422+0000 7ffbd2ffd640 10 client.204648.objecter ms_dispatch 0x7ffbdc000b90 osd_map(711..711 src has 299..711) 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.422+0000 7ffbd2ffd640 3 client.204648.objecter handle_osd_map got epochs [711,711] > 0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.422+0000 7ffbd2ffd640 3 client.204648.objecter handle_osd_map decoding full epoch 711 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.422+0000 7ffbd2ffd640 20 client.204648.objecter dump_active .. 0 homeless 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.422+0000 7ffbd9d9b640 10 monclient: get_auth_request con 0x7ffbbc0419a0 auth_method 0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.424+0000 7ffbd2ffd640 10 client.204648.objecter ms_handle_connect 0x7ffbbc0419a0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.426+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ffbcc017850 con 0x7ffbdc0694b0 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.426+0000 7ffbd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T10:09:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.426+0000 7ffbd2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T10:09:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.610+0000 7ffbe0981640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-08-19T10:09:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.610+0000 7ffbe0981640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-08-19T10:09:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.610+0000 7ffbe0981640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.1:6792/0 2024-08-19T10:09:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.610+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 --> v1:172.21.15.1:6792/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7ffbc0005370 con 0x7ffbdc0694b0 2024-08-19T10:09:48.615 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.615 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=5,osd_snap=40,osdmap=21} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=1981144331,mon_config_key=3718736655,monmap=3096227638,osd_metadata=84764626,osd_snap=1057805154,osdmap=2184861657}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=17} crc {mgr=1542226056,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=29605565,mon_config_key=1905280744,monmap=3096227638,osd_metadata=675158879,osd_pg_creating=1246591483,osd_snap=3463650982,osdmap=4006990149}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=17} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=29605565,mon_config_key=561037852,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=1246591483,osd_snap=4077208141,osdmap=92163100}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=18} crc {mgr=2062615269,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=4086450874,mon_config_key=3203914822,monmap=3096227638,osd_metadata=84764626,osd_pg_creating=1246591483,osd_snap=4099029728,osdmap=2766079020}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.616 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=17,monmap=3,osd_metadata=6,osd_snap=37,osdmap=19} crc {mgr=2062615269,mgr_command_descs=3626727127,mgr_metadata=3429325078,mgrstat=2377991690,mon_config_key=1905280744,monmap=3096227638,osd_metadata=2120554823,osd_snap=1675540339,osdmap=616432681}) 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=17} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=29605565,mon_config_key=4240075498,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=1246591483,osd_snap=4066293026,osdmap=2966548822}) 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=20} crc {mgr=1049525333,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=3224583058,mon_config_key=1439588724,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=1246591483,osd_snap=3887749556,osdmap=2154189796}) 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=481300245,mon_config_key=4227192348,monmap=3096227638,osd_metadata=2120554824,osd_pg_creating=1246591483,osd_snap=246204431,osdmap=3087195172}) 2024-08-19T10:09:48.617 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.612+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=15} crc {mgr=2606302912,mgr_command_descs=3626727126,mgr_metadata=3429325078,mgrstat=4086450874,mon_config_key=699569359,monmap=3096227638,osd_metadata=2120554823,osd_pg_creating=1246591483,osd_snap=3716701805,osdmap=1854433346}) 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7ffbcc016cc0 con 0x7ffbdc0694b0 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd2ffd640 10 monclient: handle_config config(1 keys) 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd8d99640 4 set_mon_vals no callback set 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd8d99640 10 set_mon_vals mon_health_to_clog = false 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd2ffd640 1 -- 172.21.15.1:0/2535443367 <== mon.6 v1:172.21.15.1:6792/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (unknown 1453024670 0 0) 0x7ffbcc017150 con 0x7ffbdc0694b0 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-08-19T10:09:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.616+0000 7ffbd2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=704951425}) 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2222033310}) 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2771536518}) 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1260863193}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=800108590}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3945975559}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4277083939}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099104059}) 2024-08-19T10:09:48.621 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.618+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1531926969}) 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 10 librados: watch_flush enter 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 10 librados: watch_flush exit 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 20 client.204648.objecter shutdown clearing up homeless session... 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 10 client.204648.objecter successfully canceled tick 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.82:6812/2549104415 conn(0x7ffbbc0419a0 legacy=0x7ffbbc043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 10 monclient: shutdown 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 >> v1:172.21.15.1:6792/0 conn(0x7ffbdc0694b0 legacy=0x7ffbdc1cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 shutdown_connections 2024-08-19T10:09:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 >> 172.21.15.1:0/2535443367 conn(0x7ffbdc0a9020 msgr2=0x7ffbdc0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T10:09:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.619+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 shutdown_connections 2024-08-19T10:09:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.620+0000 7ffbe0981640 1 -- 172.21.15.1:0/2535443367 wait complete. 2024-08-19T10:09:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-08-19T10:09:48.620+0000 7ffbe0981640 1 librados: shutdown 2024-08-19T10:09:48.625 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=681338269}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=13186002}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2428637451}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.626 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2112849340}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1023447336}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1960864663}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3737404083}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108440516}) 2024-08-19T10:09:48.627 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.624+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1546976589}) 2024-08-19T10:09:48.637 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/ceph.py", line 1980, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 466, in task thrash_proc.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T10:09:48.638 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-19T10:09:48.639 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-19T10:09:48.639 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-19T10:09:48.639 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:48.641 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.641 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.641 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3150219525}) 2024-08-19T10:09:48.641 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.641 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1106179056}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1926490809}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1679653412}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=437346482}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.642 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1183394508}) 2024-08-19T10:09:48.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=20099020}) 2024-08-19T10:09:48.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.639+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=979268110}) 2024-08-19T10:09:48.643 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.640+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3152684964}) 2024-08-19T10:09:48.656 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.656 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.656 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=342506839}) 2024-08-19T10:09:48.656 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.656 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2940117892}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3474895527}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3450558222}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4042659954}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.657 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2417169809}) 2024-08-19T10:09:48.658 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.658 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.658 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3540589753}) 2024-08-19T10:09:48.658 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.653+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.658 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.654+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3132985606}) 2024-08-19T10:09:48.658 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.654+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3142180039}) 2024-08-19T10:09:48.661 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.661 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.661 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1237216313}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=340991020}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2467351949}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3273055866}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4141703493}) 2024-08-19T10:09:48.662 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1359301225}) 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3211973453}) 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927403945}) 2024-08-19T10:09:48.663 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.659+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1270447390}) 2024-08-19T10:09:48.665 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.665 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=87} crc {osdmap=840815122}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=90} crc {osdmap=157141321}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=90} crc {osdmap=1450921806}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=91} crc {osdmap=2468787904}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.663+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.666 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=90} crc {osdmap=1576826497}) 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=90} crc {osdmap=2065934910}) 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=87} crc {osdmap=839228689}) 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=2948675958}) 2024-08-19T10:09:48.667 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-08-19T10:09:48.664+0000 7f90c3343640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=91} crc {osdmap=2498341838}) 2024-08-19T10:09:48.685 INFO:tasks.ceph.osd.0:Stopped 2024-08-19T10:09:48.686 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-19T10:09:48.686 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:48.758 INFO:tasks.ceph.osd.1:Stopped 2024-08-19T10:09:48.759 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-19T10:09:48.759 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:48.833 INFO:tasks.ceph.osd.2:Stopped 2024-08-19T10:09:48.834 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-19T10:09:48.834 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:48.874 INFO:tasks.ceph.osd.3:Stopped 2024-08-19T10:09:48.875 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-08-19T10:09:48.875 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:48.942 INFO:tasks.ceph.osd.4:Stopped 2024-08-19T10:09:48.942 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-08-19T10:09:48.942 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.013 INFO:tasks.ceph.osd.5:Stopped 2024-08-19T10:09:49.014 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-19T10:09:49.014 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-19T10:09:49.014 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.080 INFO:tasks.ceph.mgr.x:Stopped 2024-08-19T10:09:49.080 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-19T10:09:49.080 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T10:09:49.081 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.094 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T10:09:49.094 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-19T10:09:49.094 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.149 INFO:tasks.ceph.mon.b:Stopped 2024-08-19T10:09:49.149 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-19T10:09:49.149 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.204 INFO:tasks.ceph.mon.c:Stopped 2024-08-19T10:09:49.205 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T10:09:49.205 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.258 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T10:09:49.258 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T10:09:49.259 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.311 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T10:09:49.311 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T10:09:49.311 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.323 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T10:09:49.323 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T10:09:49.323 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.376 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T10:09:49.376 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-08-19T10:09:49.376 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.428 INFO:tasks.ceph.mon.h:Stopped 2024-08-19T10:09:49.428 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-08-19T10:09:49.428 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T10:09:49.481 INFO:tasks.ceph.mon.i:Stopped 2024-08-19T10:09:49.481 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-19T10:09:49.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-08-19T10:09:49.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T10:09:49.538 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-19T10:09:49.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T10:09:49.903 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-19T10:09:49.991 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T10:09:49.991 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-19T10:09:50.040 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T10:09:50.041 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-19T10:09:50.302 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T10:09:50.303 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-08-19T10:09:50.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T10:09:50.399 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-08-19T10:09:50.466 INFO:tasks.ceph:Archiving mon data... 2024-08-19T10:09:50.467 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.a.tgz 2024-08-19T10:09:50.467 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-08-19T10:09:50.485 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.IQpXnRPKoa 2024-08-19T10:09:50.486 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IQpXnRPKoa 2024-08-19T10:09:50.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.IQpXnRPKoa 2024-08-19T10:09:50.768 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.IQpXnRPKoa is 812KB 2024-08-19T10:09:50.871 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.IQpXnRPKoa 2024-08-19T10:09:50.886 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.b.tgz 2024-08-19T10:09:50.886 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-08-19T10:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.zd5X9aYzVK 2024-08-19T10:09:50.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zd5X9aYzVK 2024-08-19T10:09:51.121 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.zd5X9aYzVK 2024-08-19T10:09:51.204 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.zd5X9aYzVK is 832KB 2024-08-19T10:09:51.306 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.zd5X9aYzVK 2024-08-19T10:09:51.321 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.c.tgz 2024-08-19T10:09:51.321 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-08-19T10:09:51.376 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.YlmNPq9ksb 2024-08-19T10:09:51.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.YlmNPq9ksb 2024-08-19T10:09:51.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.YlmNPq9ksb 2024-08-19T10:09:51.642 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.YlmNPq9ksb is 830KB 2024-08-19T10:09:51.747 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.YlmNPq9ksb 2024-08-19T10:09:51.763 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.d.tgz 2024-08-19T10:09:51.763 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-08-19T10:09:51.818 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.QZPFrtWCKb 2024-08-19T10:09:51.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.QZPFrtWCKb 2024-08-19T10:09:51.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.QZPFrtWCKb 2024-08-19T10:09:52.081 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.QZPFrtWCKb is 826KB 2024-08-19T10:09:52.185 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.QZPFrtWCKb 2024-08-19T10:09:52.202 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.e.tgz 2024-08-19T10:09:52.203 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-08-19T10:09:52.255 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.TviJIJUwff 2024-08-19T10:09:52.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.TviJIJUwff 2024-08-19T10:09:52.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.TviJIJUwff 2024-08-19T10:09:52.522 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.TviJIJUwff is 828KB 2024-08-19T10:09:52.625 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.TviJIJUwff 2024-08-19T10:09:52.640 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.f.tgz 2024-08-19T10:09:52.640 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-08-19T10:09:52.659 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.PlVVg8oCFx 2024-08-19T10:09:52.659 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.PlVVg8oCFx 2024-08-19T10:09:52.926 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.PlVVg8oCFx 2024-08-19T10:09:53.009 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.PlVVg8oCFx is 819KB 2024-08-19T10:09:53.115 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.PlVVg8oCFx 2024-08-19T10:09:53.131 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.g.tgz 2024-08-19T10:09:53.131 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-08-19T10:09:53.185 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.jZJBeZEjwV 2024-08-19T10:09:53.186 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.jZJBeZEjwV 2024-08-19T10:09:53.362 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.jZJBeZEjwV 2024-08-19T10:09:53.450 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.jZJBeZEjwV is 823KB 2024-08-19T10:09:53.558 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.jZJBeZEjwV 2024-08-19T10:09:53.574 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.h.tgz 2024-08-19T10:09:53.574 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-08-19T10:09:53.628 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.CLzkIXpYVC 2024-08-19T10:09:53.629 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.CLzkIXpYVC 2024-08-19T10:09:53.803 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.CLzkIXpYVC 2024-08-19T10:09:53.888 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.CLzkIXpYVC is 823KB 2024-08-19T10:09:53.999 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.CLzkIXpYVC 2024-08-19T10:09:54.014 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/data/mon.i.tgz 2024-08-19T10:09:54.014 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-08-19T10:09:54.069 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.fQxT9qw2A8 2024-08-19T10:09:54.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.fQxT9qw2A8 2024-08-19T10:09:54.247 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.fQxT9qw2A8 2024-08-19T10:09:54.331 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.fQxT9qw2A8 is 825KB 2024-08-19T10:09:54.434 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.fQxT9qw2A8 2024-08-19T10:09:54.450 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-19T10:09:54.450 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-19T10:09:54.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-19T10:09:54.539 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-19T10:09:54.569 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-19T10:09:54.570 INFO:tasks.ceph:Archiving crash dumps... 2024-08-19T10:09:54.572 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/remote/smithi001/crash 2024-08-19T10:09:54.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-19T10:09:54.608 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/remote/smithi082/crash 2024-08-19T10:09:54.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-19T10:09:54.633 INFO:tasks.ceph:Compressing logs... 2024-08-19T10:09:54.634 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T10:09:54.654 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T10:09:54.679 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27270.log 2024-08-19T10:09:54.679 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37220.log 2024-08-19T10:09:54.679 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38242.log 2024-08-19T10:09:54.680 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32672.log 2024-08-19T10:09:54.680 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27270.log.gz 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26185.log 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26689.log 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27091.log 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26689.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26692.log 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25743.log 2024-08-19T10:09:54.744 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.h.log: /var/log/ceph/ceph-client.admin.26692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26134.log 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26332.log 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.26692.log.gz 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26134.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26102.log 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.26689.log.gz 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.i.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26134.log.gz 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27162.log 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27329.log 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27162.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.26185.log.gz 2024-08-19T10:09:54.745 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25842.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27329.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27162.log.gz 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.26332.log.gz 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.27329.log.gz 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26904.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26667.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.27091.log.gz 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27330.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26667.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26904.log.gz 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.26102.log.gz 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr: --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27159.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.3.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.27330.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26582.log 2024-08-19T10:09:54.746 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20799.log 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26582.log: /var/log/ceph/ceph-client.admin.27159.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.25842.log.gz 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.26582.log.gz 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.20799.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.26667.log.gz 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28386.log 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20799.log.gz 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27333.log 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.g.log: /var/log/ceph/ceph-client.admin.28386.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-19T10:09:54.747 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27333.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.27333.log.gz 2024-08-19T10:09:54.748 INFO:teuthology.orchestra.run.smithi082.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28386.log.gz 2024-08-19T10:09:54.748 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26441.log 2024-08-19T10:09:54.748 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20963.log 2024-08-19T10:09:54.748 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26441.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26441.log.gz 2024-08-19T10:09:54.748 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26926.log 2024-08-19T10:09:54.748 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.20963.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.20963.log.gz 2024-08-19T10:09:54.749 INFO:teuthology.orchestra.run.smithi082.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.25743.log.gz 2024-08-19T10:09:54.749 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26691.log 2024-08-19T10:09:54.749 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26497.log 2024-08-19T10:09:54.749 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26691.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.26691.log.gz 2024-08-19T10:09:54.749 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27136.log 2024-08-19T10:09:54.753 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26497.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.26926.log.gz 2024-08-19T10:09:54.753 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25728.log 2024-08-19T10:09:54.758 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27136.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.27159.log.gz 2024-08-19T10:09:54.758 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26976.log 2024-08-19T10:09:54.765 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25728.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27136.log.gz 2024-08-19T10:09:54.766 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26896.log 2024-08-19T10:09:54.775 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26976.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-19T10:09:54.782 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26896.log: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27721.log 2024-08-19T10:09:54.788 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.audit.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.26976.log.gz 2024-08-19T10:09:54.788 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26896.log.gz 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.26497.log.gz 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20919.log 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27721.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27721.log.gz 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26587.log 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.20919.log: 89.9% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.20919.log.gz 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25728.log.gz 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28387.log 2024-08-19T10:09:54.821 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27669.log 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.28387.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.26587.log.gz 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26933.log 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27669.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.28387.log.gz 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25915.log 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26933.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.27669.log.gz 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27722.log 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25915.log: 95.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27722.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.27722.log.gz 2024-08-19T10:09:54.822 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21003.log 2024-08-19T10:09:54.834 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.4.log: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25997.log 2024-08-19T10:09:54.834 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.21003.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.21003.log.gz 2024-08-19T10:09:54.834 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26933.log.gz 2024-08-19T10:09:54.844 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25797.log 2024-08-19T10:09:54.858 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25779.log 2024-08-19T10:09:54.858 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25797.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.25797.log.gz 2024-08-19T10:09:54.871 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26201.log 2024-08-19T10:09:54.883 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26073.log 2024-08-19T10:09:54.887 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38242.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37220.log.gz 2024-08-19T10:09:54.887 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30121.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32630.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38242.log.gz 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29350.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32672.log.gz 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29657.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32630.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30121.log.gz 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34200.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29657.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32630.log.gz 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29350.log.gz 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36884.log 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29657.log.gz 2024-08-19T10:09:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25534.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34200.log.gz 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37262.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25070.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36884.log.gz 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2%/var/log/ceph/ceph-client.admin.37262.log: -- replaced with /var/log/ceph/ceph-client.admin.25534.log.gz 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36798.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37262.log.gz 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28633.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.25070.log.gz 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32924.log 2024-08-19T10:09:54.889 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36798.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28633.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28633.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32336.log 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32924.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26230.log 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35645.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37136.log 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32336.log.gz 2024-08-19T10:09:54.890 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31133.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37990.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26230.log.gz 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27990.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37136.log.gz 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37472.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31133.log.gz 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31767.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37990.log.gz 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29223.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27990.log.gz 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34114.log 2024-08-19T10:09:54.891 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37472.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34030.log 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31767.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26930.log 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29223.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30923.log 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34114.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28761.log 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34030.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32378.log 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26930.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26356.log 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30923.log.gz 2024-08-19T10:09:54.892 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31345.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28761.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28465.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32378.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31049.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26356.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30965.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31345.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37822.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28465.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26735.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31049.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30965.log.gz 2024-08-19T10:09:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26735.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37822.log.gz 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33692.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29055.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26735.log.gz 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25309.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25554.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33692.log.gz 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35517.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29055.log.gz 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29265.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25309.log.gz 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35517.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.28929.log 2024-08-19T10:09:54.894 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25554.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30415.log 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35517.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32756.log 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29265.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25318.log 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28929.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26693.log 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30415.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35349.log 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32756.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25318.log.gz 2024-08-19T10:09:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35097.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26693.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24802.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35349.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35097.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34794.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24802.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32061.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36065.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30625.log 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-08-19T10:09:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28845.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32061.log.gz 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35813.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36065.log.gz 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26972.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30625.log.gz 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35307.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28845.log.gz 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27102.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35813.log.gz 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33946.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.26972.log.gz 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25936.log 2024-08-19T10:09:54.897 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35307.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27102.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25936.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33946.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25603.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25936.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28032.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38789.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34412.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25603.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38789.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28032.log.gz 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29013.log 2024-08-19T10:09:54.898 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38789.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26651.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27526.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34412.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26104.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29013.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26846.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26651.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26104.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27526.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27060.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27354.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26104.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34668.log 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26846.log.gz 2024-08-19T10:09:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25066.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27060.log.gz 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26272.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27354.log.gz 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28803.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34668.log.gz 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.25066.log.gz 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26567.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26272.log: /var/log/ceph/ceph-client.admin.28803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32840.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26272.log.gz 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24844.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28803.log.gz 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38747.log 2024-08-19T10:09:54.900 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9%/var/log/ceph/ceph-client.admin.32840.log: -- replaced with /var/log/ceph/ceph-client.admin.26567.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25319.log 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34326.log 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.24844.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32840.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38747.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24928.log 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25319.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37430.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34326.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30457.log 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37178.log 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-08-19T10:09:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30457.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24928.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38453.log 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28507.log 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37178.log: /var/log/ceph/ceph-client.admin.38453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27780.log 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30457.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28507.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37178.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31091.log 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38453.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28507.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26188.log 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27780.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37556.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31091.log.gz 2024-08-19T10:09:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34538.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28337.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38537.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26188.log.gz 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32252.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28337.log: /var/log/ceph/ceph-client.admin.38537.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34538.log.gz 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27906.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32252.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28337.log.gz 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36378.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38537.log.gz 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36023.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27906.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32252.log.gz 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31893.log 2024-08-19T10:09:54.903 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27906.log.gz 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36233.log 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36378.log.gz 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36023.log.gz 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36756.log 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31893.log.gz 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36672.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36233.log.gz 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37948.log 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32882.log 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24751.log 2024-08-19T10:09:54.904 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36756.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30205.log 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37948.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27144.log 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32882.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29783.log 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24751.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29181.log 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30205.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.29783.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.21379.log 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27144.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27654.log 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29783.log.gz 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.21379.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 88.1% 2024-08-19T10:09:54.905 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29181.log.gz 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37094.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.21379.log.gz 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27654.log.gz -5 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.24670.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38663.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28887.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37094.log.gz 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28719.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24670.log.gz 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33734.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38663.log.gz 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35729.log 2024-08-19T10:09:54.906 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28887.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35561.log 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28719.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27228.log 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33734.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35561.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35729.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35603.log 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.35561.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25852.log 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27228.log: /var/log/ceph/ceph-client.admin.35603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32294.log 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25852.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27228.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33860.log 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35603.log.gz 2024-08-19T10:09:54.907 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38074.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25852.log.gz 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32294.log.gz 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.38074.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31599.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38915.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33860.log.gz 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33260.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38074.log.gz 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31599.log: gzip 87.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38495.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33260.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0%/var/log/ceph/ceph-client.admin.38495.log: -- replaced with /var/log/ceph/ceph-client.admin.31599.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33260.log.gz 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26440.log 2024-08-19T10:09:54.908 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.a.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38495.log.gz 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34584.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37304.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26440.log.gz 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27484.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34584.log.gz 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31387.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6%/var/log/ceph/ceph-client.admin.33218.log: -- replaced with /var/log/ceph/ceph-client.admin.37304.log.gz 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25726.log 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27484.log.gz 2024-08-19T10:09:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29097.log 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25726.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31387.log.gz 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31683.log 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.25726.log.gz 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32420.log 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25894.log 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31683.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29097.log.gz 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25604.log 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31683.log.gz 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29571.log 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32420.log.gz 2024-08-19T10:09:54.910 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25604.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25894.log.gz 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33050.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25604.log.gz 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28971.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29571.log.gz 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29951.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36317.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28971.log: /var/log/ceph/ceph-client.admin.29951.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33050.log.gz 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31725.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28971.log.gz 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36317.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29951.log.gz 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.24562.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26483.log 2024-08-19T10:09:54.911 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36317.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25064.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31725.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26483.log: gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24562.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.25069.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25064.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26483.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35265.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25064.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29139.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33176.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35265.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25069.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36968.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35265.log.gz 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38284.log 2024-08-19T10:09:54.912 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29139.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.36968.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32462.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33176.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29308.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36968.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32462.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24726.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38284.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28381.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32462.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29825.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29308.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28381.log: gzip -5 --verbose -- 86.9% /var/log/ceph/ceph-client.admin.36107.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24726.log.gz 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37010.log 2024-08-19T10:09:54.913 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28381.log.gz 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34072.log 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29825.log.gz 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31557.log 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36840.log 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36107.log.gz/var/log/ceph/ceph-client.admin.34072.log: 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37010.log.gz 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36504.log 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34072.log.gz 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.b.log/var/log/ceph/ceph-client.admin.36840.log: 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31557.log.gz 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30163.log 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36840.log.gz 2024-08-19T10:09:54.914 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph-client.admin.36504.log.gz 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35433.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32798.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32588.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30163.log.gz 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32798.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35433.log.gz 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31977.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32588.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27948.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32798.log.gz 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31977.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32588.log.gz 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.27696.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27948.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31977.log.gz 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31429.log 2024-08-19T10:09:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27696.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27948.log.gz 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26314.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31429.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27696.log.gz 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33008.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.26314.log: -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31429.log.gz 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33008.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26314.log.gz 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33988.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.d.log: gzip 89.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33008.log.gz 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33346.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37780.log: /var/log/ceph/ceph-client.admin.28251.log 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33988.log.gz 2024-08-19T10:09:54.916 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35897.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33346.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28251.log: /var/log/ceph/ceph-client.admin.35897.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28251.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35897.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24886.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25322.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.24886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26525.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25322.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.24886.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25322.log.gz 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27396.log 2024-08-19T10:09:54.917 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31641.log 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27396.log: /var/log/ceph/ceph-client.admin.26525.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26525.log.gz 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27396.log.gz 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34242.log 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25320.log 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34242.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31641.log.gz 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36588.log 96.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25320.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35139.log 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36588.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.25320.log.gz 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27018.log 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31513.log 2024-08-19T10:09:54.918 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35139.log: /var/log/ceph/ceph-client.admin.27018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38116.log 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31513.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35139.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27018.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37640.log 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8%/var/log/ceph/ceph-client.admin.38116.log: -- replaced with /var/log/ceph/ceph-client.admin.31513.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37598.log 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31935.log 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38116.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37640.log: /var/log/ceph/ceph-client.admin.37598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31007.log 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31935.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37640.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37598.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27442.log 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31935.log.gz 2024-08-19T10:09:54.919 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27442.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31007.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36546.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27442.log.gz 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32714.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34752.log: /var/log/ceph/ceph-client.admin.36546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30499.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36546.log.gz 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33902.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30499.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32714.log.gz 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35687.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37052.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30499.log.gz 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33902.log: /var/log/ceph/ceph-client.admin.35687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27738.log 2024-08-19T10:09:54.920 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33902.log.gz 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37052.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35687.log.gzgzip 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37052.log.gz 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38705.log 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27738.log: /var/log/ceph/ceph-client.admin.36462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20955.log 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2%/var/log/ceph/ceph-client.admin.38705.log: -- replaced with /var/log/ceph/ceph-client.admin.27738.log.gz 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30037.log 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0%/var/log/ceph/ceph-client.admin.20955.log: -- replaced with /var/log/ceph/ceph-client.admin.38705.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35475.log 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.20955.log.gz 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34935.log 2024-08-19T10:09:54.921 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35475.log: /var/log/ceph/ceph-client.admin.30037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27312.log 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34935.log: 88.8% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30037.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35475.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33472.log 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34935.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log/var/log/ceph/ceph-client.admin.27312.log: 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28591.log 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27312.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33472.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37388.log 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27568.log 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37388.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28591.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37388.log.gz 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34710.log 2024-08-19T10:09:54.922 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27568.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27568.log.gz 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38621.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26398.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38621.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37346.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26398.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38621.log.gz 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32546.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26398.log.gz/var/log/ceph/ceph-client.admin.37346.log: 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36926.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32546.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37346.log.gz 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21305.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32546.log.gz 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36926.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-19T10:09:54.923 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.21305.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36926.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.21305.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33302.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36275.log: /var/log/ceph/ceph.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38032.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33302.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33302.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25768.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38032.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38032.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25474.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25768.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.25768.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25474.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.25474.log.gz 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27610.log 2024-08-19T10:09:54.924 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38955.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30289.log 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27610.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27610.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26020.log 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30289.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30289.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35391.log 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26020.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.26020.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30247.log 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35391.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35391.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37864.log 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30247.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30247.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32966.log 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37864.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37864.log.gz 2024-08-19T10:09:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34454.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32966.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32966.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25068.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34454.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34454.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38411.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25068.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25068.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29699.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38411.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38411.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38158.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29699.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29699.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26146.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38158.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38158.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37696.log 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26146.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26146.log.gz 2024-08-19T10:09:54.926 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37696.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37696.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26062.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33818.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32504.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26062.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26062.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27186.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32504.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32504.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25062.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27186.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27186.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27822.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25062.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25062.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31851.log 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27822.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27822.log.gz 2024-08-19T10:09:54.927 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29615.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31851.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31851.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34158.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29615.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29615.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34158.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34158.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37906.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30541.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36191.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37906.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37906.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33134.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36191.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36191.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35939.log 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33134.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.33134.log.gz 2024-08-19T10:09:54.928 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27864.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35939.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35939.log.gz 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.27864.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27864.log.gz 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37738.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35855.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37738.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37738.log.gz 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26609.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35855.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35855.log.gz 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33570.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26609.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26609.log.gz 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35181.log 2024-08-19T10:09:54.929 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33570.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33570.log.gz 2024-08-19T10:09:54.934 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26201.log: 91.2% 93.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27332.log 2024-08-19T10:09:54.935 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25992.log 2024-08-19T10:09:54.935 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27332.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.26201.log.gz 2024-08-19T10:09:54.935 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25779.log.gz 2024-08-19T10:09:54.935 INFO:teuthology.orchestra.run.smithi082.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26073.log.gz 2024-08-19T10:09:54.935 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27332.log.gz 2024-08-19T10:09:54.935 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27724.log 2024-08-19T10:09:54.939 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26173.log 2024-08-19T10:09:54.952 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27724.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26120.log 2024-08-19T10:09:54.952 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26173.log: -- replaced with /var/log/ceph/ceph-client.admin.27724.log.gz 2024-08-19T10:09:54.958 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26493.log 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29909.log 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35181.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35181.log.gz 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35981.log 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29909.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29909.log.gz 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30331.log 2024-08-19T10:09:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35981.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35981.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25810.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30331.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30331.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36630.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25810.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25810.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25494.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36630.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36630.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29867.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25494.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25494.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34284.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29867.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29867.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37514.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34284.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34284.log.gz 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30583.log 2024-08-19T10:09:54.963 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37514.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37514.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34368.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30583.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30583.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26888.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34368.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34368.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36149.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26888.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26888.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36714.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36149.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36149.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25978.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36714.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36714.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38326.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25978.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25978.log.gz 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34836.log 2024-08-19T10:09:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38326.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38326.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31177.log 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34836.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34836.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29741.log 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31177.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31177.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28295.log 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29741.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29741.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28423.log 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28295.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28295.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33776.log 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28423.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28423.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.16286.log 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33776.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33776.log.gz 2024-08-19T10:09:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30667.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.16286.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.16286.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30373.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30667.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30667.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33430.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30373.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30373.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33430.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33430.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38200.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38579.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28549.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38200.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38200.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.e.log 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28549.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28549.log.gz 2024-08-19T10:09:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36420.log 2024-08-19T10:09:54.968 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26120.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26173.log.gz 2024-08-19T10:09:54.968 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26575.log 2024-08-19T10:09:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31809.log 2024-08-19T10:09:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36420.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36420.log.gz 2024-08-19T10:09:54.972 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27671.log 2024-08-19T10:09:54.977 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26575.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26493.log.gz 2024-08-19T10:09:54.977 INFO:teuthology.orchestra.run.smithi082.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.26120.log.gz 2024-08-19T10:09:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33388.log 2024-08-19T10:09:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31809.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31809.log.gz 2024-08-19T10:09:54.990 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-08-19T10:09:54.990 INFO:teuthology.orchestra.run.smithi082.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.26575.log.gz 2024-08-19T10:09:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35223.log 2024-08-19T10:09:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33388.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33388.log.gz 2024-08-19T10:09:55.005 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31261.log 2024-08-19T10:09:55.006 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27671.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.27671.log.gz 2024-08-19T10:09:55.006 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27331.log 2024-08-19T10:09:55.017 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27616.log 2024-08-19T10:09:55.019 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35223.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35223.log.gz 2024-08-19T10:09:55.019 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35771.log 2024-08-19T10:09:55.019 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31261.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31261.log.gz 2024-08-19T10:09:55.026 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27331.log: 90.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26346.log 2024-08-19T10:09:55.027 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25321.log 2024-08-19T10:09:55.028 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35771.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35771.log.gz 2024-08-19T10:09:55.033 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27616.log: -- replaced with /var/log/ceph/ceph-client.admin.27331.log.gz 2024-08-19T10:09:55.036 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28675.log 2024-08-19T10:09:55.041 INFO:teuthology.orchestra.run.smithi082.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.27616.log.gz 2024-08-19T10:09:55.046 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27723.log 2024-08-19T10:09:55.047 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38368.log 2024-08-19T10:09:55.048 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28675.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28675.log.gz 2024-08-19T10:09:55.049 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26346.log: 94.5% -- replaced with /var/log/ceph/ceph-client.admin.25992.log.gz 2024-08-19T10:09:55.052 INFO:teuthology.orchestra.run.smithi082.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.26346.log.gz 2024-08-19T10:09:55.056 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30079.log 2024-08-19T10:09:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38368.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38368.log.gz 2024-08-19T10:09:55.061 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.25321.log.gz 2024-08-19T10:09:55.064 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-19T10:09:55.066 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30079.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30079.log.gz 2024-08-19T10:09:55.074 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27723.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.27723.log.gz 2024-08-19T10:09:55.074 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28388.log 2024-08-19T10:09:55.096 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27617.log 2024-08-19T10:09:55.099 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.28388.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28388.log.gz 2024-08-19T10:09:55.110 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26893.log 2024-08-19T10:09:55.112 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27617.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.27617.log.gz 2024-08-19T10:09:55.125 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25765.log 2024-08-19T10:09:55.127 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26893.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.26893.log.gz 2024-08-19T10:09:55.138 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20879.log 2024-08-19T10:09:55.153 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25765.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26996.log 2024-08-19T10:09:55.154 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25765.log.gz 2024-08-19T10:09:55.154 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.20879.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.20879.log.gz 2024-08-19T10:09:55.167 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26402.log 2024-08-19T10:09:55.176 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26996.log: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25819.log 2024-08-19T10:09:55.183 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26402.log: -- replaced with /var/log/ceph/ceph-client.admin.26996.log.gz 2024-08-19T10:09:55.197 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27720.log 2024-08-19T10:09:55.197 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25819.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26402.log.gz 2024-08-19T10:09:55.200 INFO:teuthology.orchestra.run.smithi082.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.25819.log.gz 2024-08-19T10:09:55.210 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26571.log 2024-08-19T10:09:55.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27720.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.27720.log.gz 2024-08-19T10:09:55.223 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25832.log 2024-08-19T10:09:55.224 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.26571.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26571.log.gz 2024-08-19T10:09:55.240 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27618.log 2024-08-19T10:09:55.253 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27670.log 2024-08-19T10:09:55.256 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27618.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.27618.log.gz 2024-08-19T10:09:55.258 INFO:teuthology.orchestra.run.smithi082.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-19T10:09:55.263 INFO:teuthology.orchestra.run.smithi082.stderr: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-19T10:09:55.269 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20839.log 2024-08-19T10:09:55.271 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.27670.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.27670.log.gz 2024-08-19T10:09:55.278 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-08-19T10:09:55.291 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.20839.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.20839.log.gz 2024-08-19T10:09:55.293 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26674.log 2024-08-19T10:09:55.323 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.f.log: /var/log/ceph/ceph-client.admin.26674.log: 90.4% 92.9% -- replaced with /var/log/ceph/ceph-client.admin.26674.log.gz 2024-08-19T10:09:55.330 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25832.log.gz 2024-08-19T10:09:55.479 INFO:teuthology.orchestra.run.smithi082.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.25997.log.gz 2024-08-19T10:09:55.646 INFO:teuthology.orchestra.run.smithi082.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-08-19T10:09:55.808 INFO:teuthology.orchestra.run.smithi082.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-08-19T10:09:55.951 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-08-19T10:09:56.068 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-08-19T10:09:56.087 INFO:teuthology.orchestra.run.smithi082.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-08-19T10:09:56.154 INFO:teuthology.orchestra.run.smithi082.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.25915.log.gz 2024-08-19T10:09:56.396 INFO:teuthology.orchestra.run.smithi001.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-08-19T10:09:56.417 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-08-19T10:09:56.687 INFO:teuthology.orchestra.run.smithi082.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-08-19T10:09:57.098 INFO:teuthology.orchestra.run.smithi001.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-19T10:10:02.058 INFO:teuthology.orchestra.run.smithi082.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-08-19T10:10:02.388 INFO:teuthology.orchestra.run.smithi001.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-19T10:10:02.480 INFO:teuthology.orchestra.run.smithi082.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-08-19T10:10:02.553 INFO:teuthology.orchestra.run.smithi082.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-19T10:10:02.555 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-08-19T10:10:02.555 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m7.865s 2024-08-19T10:10:02.555 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m28.774s 2024-08-19T10:10:02.555 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m1.001s 2024-08-19T10:10:02.676 INFO:teuthology.orchestra.run.smithi001.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-19T10:10:02.688 INFO:teuthology.orchestra.run.smithi001.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-19T10:10:02.690 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-08-19T10:10:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m8.023s 2024-08-19T10:10:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m30.795s 2024-08-19T10:10:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m1.166s 2024-08-19T10:10:02.691 INFO:tasks.ceph:Archiving logs... 2024-08-19T10:10:02.691 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/remote/smithi001/log 2024-08-19T10:10:02.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T10:10:07.358 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/remote/smithi082/log 2024-08-19T10:10:07.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T10:10:11.790 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-19T10:10:11.800 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 466, in task thrash_proc.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T10:10:11.800 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-19T10:10:11.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T10:10:11.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T10:10:11.902 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-19T10:10:11.911 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-19T10:10:11.911 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T10:10:11.914 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T10:10:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-08-19T10:10:11.931 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-08-19T10:10:11.934 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T10:10:11.934 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-08-19T10:10:11.934 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 +423us[ +425us] +/- 36ms 2024-08-19T10:10:11.934 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -929us[ -929us] +/- 66ms 2024-08-19T10:10:11.934 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +192us[ +194us] +/- 74ms 2024-08-19T10:10:11.934 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T10:10:11.937 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T10:10:11.937 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-08-19T10:10:11.937 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 +434us[ +435us] +/- 36ms 2024-08-19T10:10:11.938 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -920us[ -920us] +/- 66ms 2024-08-19T10:10:11.938 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +198us[ +199us] +/- 74ms 2024-08-19T10:10:11.938 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T10:10:11.938 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-19T10:10:11.947 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-19T10:10:11.947 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-19T10:10:11.956 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T10:10:12.036 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T10:10:12.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T10:10:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T10:10:12.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T10:10:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T10:10:12.150 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-19T10:10:12.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-19T10:10:12.168 INFO:teuthology.task.internal:Duration was 1482.472593 seconds 2024-08-19T10:10:12.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-19T10:10:12.177 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-19T10:10:12.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T10:10:12.180 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T10:10:12.220 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T10:10:12.234 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T10:10:12.438 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-19T10:10:12.438 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-08-19T10:10:12.438 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T10:10:12.493 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-08-19T10:10:12.493 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T10:10:12.522 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-19T10:10:12.522 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T10:10:12.535 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T10:10:12.589 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-19T10:10:12.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T10:10:12.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T10:10:12.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-19T10:10:12.706 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-19T10:10:12.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T10:10:12.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T10:10:12.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-19T10:10:12.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T10:10:12.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T10:10:12.795 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-08-19T10:10:12.819 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-08-19T10:10:12.832 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T10:10:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T10:10:12.868 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T10:10:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T10:10:12.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-19T10:10:12.896 INFO:teuthology.task.internal:Transferring archived files... 2024-08-19T10:10:12.897 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/remote/smithi001 2024-08-19T10:10:12.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T10:10:12.940 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859301/remote/smithi082 2024-08-19T10:10:12.940 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T10:10:12.990 INFO:teuthology.task.internal:Removing archive directory... 2024-08-19T10:10:12.990 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T10:10:12.993 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T10:10:13.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-19T10:10:13.035 INFO:teuthology.task.internal:Not uploading archives. 2024-08-19T10:10:13.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-19T10:10:13.044 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-19T10:10:13.044 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T10:10:13.050 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T10:10:13.059 INFO:teuthology.orchestra.run.smithi001.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 10:10 /home/ubuntu/cephtest 2024-08-19T10:10:13.077 INFO:teuthology.orchestra.run.smithi082.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 10:10 /home/ubuntu/cephtest 2024-08-19T10:10:13.078 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-19T10:10:13.088 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-19T10:10:13.099 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: AttributeError: 'bool' object has no attribute 'set' 2024-08-19T10:10:13.101 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests} duration: 1482.4725930690765 failure_reason: '''bool'' object has no attribute ''set''' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-08-19T10:10:13.102 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T10:10:13.199 INFO:teuthology.run:FAIL